Class UAX29URLEmailTokenizer
- java.lang.Object
-
- org.apache.lucene.util.AttributeSource
-
- org.apache.lucene.analysis.TokenStream
-
- org.apache.lucene.analysis.Tokenizer
-
- org.apache.lucene.analysis.standard.UAX29URLEmailTokenizer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class UAX29URLEmailTokenizer extends Tokenizer
This class implements Word Break rules from the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29 URLs and email addresses are also tokenized according to the relevant RFCs. Tokens produced are of the following types:- <ALPHANUM>: A sequence of alphabetic and numeric characters
- <NUM>: A number
- <URL>: A URL
- <EMAIL>: An email address
- <SOUTHEAST_ASIAN>: A sequence of characters from South and Southeast Asian languages, including Thai, Lao, Myanmar, and Khmer
- <IDEOGRAPHIC>: A single CJKV ideographic character
- <HIRAGANA>: A single hiragana character
You must specify the required
Version
compatibility when creating UAX29URLEmailTokenizer:- As of 3.4, Hiragana and Han characters are no longer wrongly split from their combining characters. If you use a previous version number, you get the exact broken behavior for backwards compatibility.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource
AttributeSource.AttributeFactory, AttributeSource.State
-
-
Field Summary
Fields Modifier and Type Field Description static int
ALPHANUM
static int
EMAIL
static String
EMAIL_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
HANGUL
static String
HANGUL_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
HIRAGANA
static String
HIRAGANA_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
IDEOGRAPHIC
static String
IDEOGRAPHIC_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
KATAKANA
static String
KATAKANA_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
NUM
static String
NUMERIC_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic String
SOUTH_EAST_ASIAN_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic int
SOUTHEAST_ASIAN
static String[]
TOKEN_TYPES
String token types that correspond to token type int constantsstatic int
URL
static String
URL_TYPE
Deprecated.useTOKEN_TYPES
insteadstatic String
WORD_TYPE
Deprecated.useTOKEN_TYPES
instead
-
Constructor Summary
Constructors Constructor Description UAX29URLEmailTokenizer(InputStream input)
Deprecated.useUAX29URLEmailTokenizer(Version, Reader)
instead.UAX29URLEmailTokenizer(Reader input)
Deprecated.useUAX29URLEmailTokenizer(Version, Reader)
instead.UAX29URLEmailTokenizer(AttributeSource.AttributeFactory factory, Reader input)
Deprecated.UAX29URLEmailTokenizer(AttributeSource source, Reader input)
Deprecated.UAX29URLEmailTokenizer(Version matchVersion, Reader input)
Creates a new instance of the UAX29URLEmailTokenizer.UAX29URLEmailTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader input)
Creates a new UAX29URLEmailTokenizer with a givenAttributeSource.AttributeFactory
UAX29URLEmailTokenizer(Version matchVersion, AttributeSource source, Reader input)
Creates a new UAX29URLEmailTokenizer with a givenAttributeSource
.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description void
end()
This method is called by the consumer after the last token has been consumed, afterTokenStream.incrementToken()
returnedfalse
(using the newTokenStream
API).int
getMaxTokenLength()
boolean
incrementToken()
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token.void
reset(Reader reader)
Expert: Reset the tokenizer to a new reader.void
setMaxTokenLength(int length)
Set the max allowed token length.-
Methods inherited from class org.apache.lucene.analysis.Tokenizer
close, correctOffset
-
Methods inherited from class org.apache.lucene.analysis.TokenStream
reset
-
Methods inherited from class org.apache.lucene.util.AttributeSource
addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, restoreState, toString
-
-
-
-
Field Detail
-
ALPHANUM
public static final int ALPHANUM
- See Also:
- Constant Field Values
-
NUM
public static final int NUM
- See Also:
- Constant Field Values
-
SOUTHEAST_ASIAN
public static final int SOUTHEAST_ASIAN
- See Also:
- Constant Field Values
-
IDEOGRAPHIC
public static final int IDEOGRAPHIC
- See Also:
- Constant Field Values
-
HIRAGANA
public static final int HIRAGANA
- See Also:
- Constant Field Values
-
KATAKANA
public static final int KATAKANA
- See Also:
- Constant Field Values
-
HANGUL
public static final int HANGUL
- See Also:
- Constant Field Values
-
URL
public static final int URL
- See Also:
- Constant Field Values
-
EMAIL
public static final int EMAIL
- See Also:
- Constant Field Values
-
TOKEN_TYPES
public static final String[] TOKEN_TYPES
String token types that correspond to token type int constants
-
WORD_TYPE
@Deprecated public static final String WORD_TYPE
Deprecated.useTOKEN_TYPES
insteadAlphanumeric sequences
-
NUMERIC_TYPE
@Deprecated public static final String NUMERIC_TYPE
Deprecated.useTOKEN_TYPES
insteadNumbers
-
URL_TYPE
@Deprecated public static final String URL_TYPE
Deprecated.useTOKEN_TYPES
insteadURLs with scheme: HTTP(S), FTP, or FILE; no-scheme URLs match HTTP syntax
-
EMAIL_TYPE
@Deprecated public static final String EMAIL_TYPE
Deprecated.useTOKEN_TYPES
insteadE-mail addresses
-
SOUTH_EAST_ASIAN_TYPE
@Deprecated public static final String SOUTH_EAST_ASIAN_TYPE
Deprecated.useTOKEN_TYPES
insteadChars in class \p{Line_Break = Complex_Context} are from South East Asian scripts (Thai, Lao, Myanmar, Khmer, etc.). Sequences of these are kept together as as a single token rather than broken up, because the logic required to break them at word boundaries is too complex for UAX#29.See Unicode Line Breaking Algorithm: http://www.unicode.org/reports/tr14/#SA
-
IDEOGRAPHIC_TYPE
@Deprecated public static final String IDEOGRAPHIC_TYPE
Deprecated.useTOKEN_TYPES
instead
-
HIRAGANA_TYPE
@Deprecated public static final String HIRAGANA_TYPE
Deprecated.useTOKEN_TYPES
instead
-
KATAKANA_TYPE
@Deprecated public static final String KATAKANA_TYPE
Deprecated.useTOKEN_TYPES
instead
-
HANGUL_TYPE
@Deprecated public static final String HANGUL_TYPE
Deprecated.useTOKEN_TYPES
instead
-
-
Constructor Detail
-
UAX29URLEmailTokenizer
@Deprecated public UAX29URLEmailTokenizer(Reader input)
Deprecated.useUAX29URLEmailTokenizer(Version, Reader)
instead.
-
UAX29URLEmailTokenizer
@Deprecated public UAX29URLEmailTokenizer(InputStream input)
Deprecated.useUAX29URLEmailTokenizer(Version, Reader)
instead.
-
UAX29URLEmailTokenizer
@Deprecated public UAX29URLEmailTokenizer(AttributeSource source, Reader input)
Deprecated.
-
UAX29URLEmailTokenizer
@Deprecated public UAX29URLEmailTokenizer(AttributeSource.AttributeFactory factory, Reader input)
Deprecated.
-
UAX29URLEmailTokenizer
public UAX29URLEmailTokenizer(Version matchVersion, Reader input)
Creates a new instance of the UAX29URLEmailTokenizer. Attaches theinput
to the newly created JFlex scanner.- Parameters:
input
- The input reader
-
UAX29URLEmailTokenizer
public UAX29URLEmailTokenizer(Version matchVersion, AttributeSource source, Reader input)
Creates a new UAX29URLEmailTokenizer with a givenAttributeSource
.
-
UAX29URLEmailTokenizer
public UAX29URLEmailTokenizer(Version matchVersion, AttributeSource.AttributeFactory factory, Reader input)
Creates a new UAX29URLEmailTokenizer with a givenAttributeSource.AttributeFactory
-
-
Method Detail
-
setMaxTokenLength
public void setMaxTokenLength(int length)
Set the max allowed token length. Any token longer than this is skipped.
-
getMaxTokenLength
public int getMaxTokenLength()
- See Also:
setMaxTokenLength(int)
-
incrementToken
public final boolean incrementToken() throws IOException
Description copied from class:TokenStream
Consumers (i.e.,IndexWriter
) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriateAttributeImpl
s with the attributes of the next token.The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use
AttributeSource.captureState()
to create a copy of the current attribute state.This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to
AttributeSource.addAttribute(Class)
andAttributeSource.getAttribute(Class)
, references to allAttributeImpl
s that this stream uses should be retrieved during instantiation.To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in
TokenStream.incrementToken()
.- Specified by:
incrementToken
in classTokenStream
- Returns:
- false for end of stream; true otherwise
- Throws:
IOException
-
end
public final void end()
Description copied from class:TokenStream
This method is called by the consumer after the last token has been consumed, afterTokenStream.incrementToken()
returnedfalse
(using the newTokenStream
API). Streams implementing the old API should upgrade to use this feature. This method can be used to perform any end-of-stream operations, such as setting the final offset of a stream. The final offset of a stream might differ from the offset of the last token eg in case one or more whitespaces followed after the last token, but aWhitespaceTokenizer
was used.- Overrides:
end
in classTokenStream
-
reset
public void reset(Reader reader) throws IOException
Description copied from class:Tokenizer
Expert: Reset the tokenizer to a new reader. Typically, an analyzer (in its reusableTokenStream method) will use this to re-use a previously created tokenizer.- Overrides:
reset
in classTokenizer
- Throws:
IOException
-
-