public final class ChineseTokenizer
extends org.apache.lucene.analysis.Tokenizer
The difference between ChineseTokenizer and CJKTokenizer is that they have different token parsing logic.
For example, if the Chinese text "C1C2C3C4" is to be indexed:
Therefore the index created by CJKTokenizer is much larger.
The problem is that when searching for C1, C1C2, C1C3, C4C2, C1C2C3 ... the ChineseTokenizer works, but the CJKTokenizer will not work.
Constructor and Description |
---|
ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory,
java.io.Reader in) |
ChineseTokenizer(org.apache.lucene.util.AttributeSource source,
java.io.Reader in) |
ChineseTokenizer(java.io.Reader in) |
Modifier and Type | Method and Description |
---|---|
void |
end() |
boolean |
incrementToken() |
void |
reset() |
void |
reset(java.io.Reader input) |
getOnlyUseNewAPI, next, next, setOnlyUseNewAPI
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, restoreState, toString
public ChineseTokenizer(java.io.Reader in)
public ChineseTokenizer(org.apache.lucene.util.AttributeSource source, java.io.Reader in)
public ChineseTokenizer(org.apache.lucene.util.AttributeSource.AttributeFactory factory, java.io.Reader in)
public boolean incrementToken() throws java.io.IOException
incrementToken
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public final void end()
end
in class org.apache.lucene.analysis.TokenStream
public void reset() throws java.io.IOException
reset
in class org.apache.lucene.analysis.TokenStream
java.io.IOException
public void reset(java.io.Reader input) throws java.io.IOException
reset
in class org.apache.lucene.analysis.Tokenizer
java.io.IOException
Copyright © 2000-2016 Apache Software Foundation. All Rights Reserved.