Package org.apache.lucene.analysis.ar
Class ArabicAnalyzer
- java.lang.Object
-
- org.apache.lucene.analysis.Analyzer
-
- org.apache.lucene.analysis.ReusableAnalyzerBase
-
- org.apache.lucene.analysis.StopwordAnalyzerBase
-
- org.apache.lucene.analysis.ar.ArabicAnalyzer
-
- All Implemented Interfaces:
Closeable
,AutoCloseable
public final class ArabicAnalyzer extends StopwordAnalyzerBase
Analyzer
for Arabic.This analyzer implements light-stemming as specified by: Light Stemming for Arabic Information Retrieval http://www.mtholyoke.edu/~lballest/Pubs/arab_stem05.pdf
The analysis package contains three primary components:
ArabicNormalizationFilter
: Arabic orthographic normalization.ArabicStemFilter
: Arabic light stemming- Arabic stop words file: a set of default Arabic stop words.
-
-
Nested Class Summary
-
Nested classes/interfaces inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase
ReusableAnalyzerBase.TokenStreamComponents
-
-
Field Summary
Fields Modifier and Type Field Description static String
DEFAULT_STOPWORD_FILE
File containing default Arabic stopwords.static String
STOPWORDS_COMMENT
Deprecated.useWordlistLoader.getWordSet(Reader, String, Version)
directly-
Fields inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
matchVersion, stopwords
-
-
Constructor Summary
Constructors Constructor Description ArabicAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words:DEFAULT_STOPWORD_FILE
.ArabicAnalyzer(Version matchVersion, File stopwords)
Deprecated.useArabicAnalyzer(Version, Set)
insteadArabicAnalyzer(Version matchVersion, String... stopwords)
Deprecated.useArabicAnalyzer(Version, Set)
insteadArabicAnalyzer(Version matchVersion, Hashtable<?,?> stopwords)
Deprecated.useArabicAnalyzer(Version, Set)
insteadArabicAnalyzer(Version matchVersion, Set<?> stopwords)
Builds an analyzer with the given stop wordsArabicAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionSet)
Builds an analyzer with the given stop word.
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected ReusableAnalyzerBase.TokenStreamComponents
createComponents(String fieldName, Reader reader)
CreatesReusableAnalyzerBase.TokenStreamComponents
used to tokenize all the text in the providedReader
.static Set<?>
getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.-
Methods inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
getStopwordSet, loadStopwordSet, loadStopwordSet, loadStopwordSet
-
Methods inherited from class org.apache.lucene.analysis.ReusableAnalyzerBase
initReader, reusableTokenStream, tokenStream
-
Methods inherited from class org.apache.lucene.analysis.Analyzer
close, getOffsetGap, getPositionIncrementGap, getPreviousTokenStream, setPreviousTokenStream
-
-
-
-
Field Detail
-
DEFAULT_STOPWORD_FILE
public static final String DEFAULT_STOPWORD_FILE
File containing default Arabic stopwords. Default stopword list is from http://members.unine.ch/jacques.savoy/clef/index.html The stopword list is BSD-Licensed.- See Also:
- Constant Field Values
-
STOPWORDS_COMMENT
@Deprecated public static final String STOPWORDS_COMMENT
Deprecated.useWordlistLoader.getWordSet(Reader, String, Version)
directlyThe comment character in the stopwords file. All lines prefixed with this will be ignored- See Also:
- Constant Field Values
-
-
Constructor Detail
-
ArabicAnalyzer
public ArabicAnalyzer(Version matchVersion)
Builds an analyzer with the default stop words:DEFAULT_STOPWORD_FILE
.
-
ArabicAnalyzer
public ArabicAnalyzer(Version matchVersion, Set<?> stopwords)
Builds an analyzer with the given stop words- Parameters:
matchVersion
- lucene compatibility versionstopwords
- a stopword set
-
ArabicAnalyzer
public ArabicAnalyzer(Version matchVersion, Set<?> stopwords, Set<?> stemExclusionSet)
Builds an analyzer with the given stop word. If a none-empty stem exclusion set is provided this analyzer will add aKeywordMarkerFilter
beforeArabicStemFilter
.- Parameters:
matchVersion
- lucene compatibility versionstopwords
- a stopword setstemExclusionSet
- a set of terms not to be stemmed
-
ArabicAnalyzer
@Deprecated public ArabicAnalyzer(Version matchVersion, String... stopwords)
Deprecated.useArabicAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words.
-
ArabicAnalyzer
@Deprecated public ArabicAnalyzer(Version matchVersion, Hashtable<?,?> stopwords)
Deprecated.useArabicAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words.
-
ArabicAnalyzer
@Deprecated public ArabicAnalyzer(Version matchVersion, File stopwords) throws IOException
Deprecated.useArabicAnalyzer(Version, Set)
insteadBuilds an analyzer with the given stop words. Lines can be commented out usingSTOPWORDS_COMMENT
- Throws:
IOException
-
-
Method Detail
-
getDefaultStopSet
public static Set<?> getDefaultStopSet()
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
createComponents
protected ReusableAnalyzerBase.TokenStreamComponents createComponents(String fieldName, Reader reader)
CreatesReusableAnalyzerBase.TokenStreamComponents
used to tokenize all the text in the providedReader
.- Specified by:
createComponents
in classReusableAnalyzerBase
- Parameters:
fieldName
- the name of the fields content passed to theReusableAnalyzerBase.TokenStreamComponents
sink as a readerreader
- the reader passed to theTokenizer
constructor- Returns:
ReusableAnalyzerBase.TokenStreamComponents
built from anStandardTokenizer
filtered withLowerCaseFilter
,StopFilter
,ArabicNormalizationFilter
,KeywordMarkerFilter
if a stem exclusion set is provided andArabicStemFilter
.
-
-