Class LimitTokenCountAnalyzer
java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.AnalyzerWrapper
org.apache.lucene.analysis.miscellaneous.LimitTokenCountAnalyzer
- All Implemented Interfaces:
Closeable
,AutoCloseable
This Analyzer limits the number of tokens while indexing. It is a replacement for the maximum
field length setting inside
IndexWriter
.- Since:
- 3.1
- See Also:
-
Nested Class Summary
Nested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents
-
Field Summary
FieldsFields inherited from class org.apache.lucene.analysis.Analyzer
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY
-
Constructor Summary
ConstructorsConstructorDescriptionLimitTokenCountAnalyzer
(Analyzer delegate, int maxTokenCount) Build an analyzer that limits the maximum number of tokens per field.LimitTokenCountAnalyzer
(Analyzer delegate, int maxTokenCount, boolean consumeAllTokens) Build an analyzer that limits the maximum number of tokens per field. -
Method Summary
Modifier and TypeMethodDescriptionprotected Analyzer
getWrappedAnalyzer
(String fieldName) Retrieves the wrapped Analyzer appropriate for analyzing the field with the given nametoString()
protected Analyzer.TokenStreamComponents
wrapComponents
(String fieldName, Analyzer.TokenStreamComponents components) Wraps / alters the given TokenStreamComponents, taken from the wrapped Analyzer, to form new components.Methods inherited from class org.apache.lucene.analysis.AnalyzerWrapper
attributeFactory, createComponents, getOffsetGap, getPositionIncrementGap, initReader, initReaderForNormalization, normalize, wrapReader, wrapReaderForNormalization, wrapTokenStreamForNormalization
Methods inherited from class org.apache.lucene.analysis.Analyzer
close, getReuseStrategy, normalize, tokenStream, tokenStream
-
Field Details
-
delegate
-
maxTokenCount
private final int maxTokenCount -
consumeAllTokens
private final boolean consumeAllTokens
-
-
Constructor Details
-
LimitTokenCountAnalyzer
Build an analyzer that limits the maximum number of tokens per field. This analyzer will not consume any tokens beyond the maxTokenCount limit- See Also:
-
LimitTokenCountAnalyzer
Build an analyzer that limits the maximum number of tokens per field.- Parameters:
delegate
- the analyzer to wrapmaxTokenCount
- max number of tokens to produceconsumeAllTokens
- whether all tokens from the delegate should be consumed even if maxTokenCount is reached.
-
-
Method Details
-
getWrappedAnalyzer
Description copied from class:AnalyzerWrapper
Retrieves the wrapped Analyzer appropriate for analyzing the field with the given name- Specified by:
getWrappedAnalyzer
in classAnalyzerWrapper
- Parameters:
fieldName
- Name of the field which is to be analyzed- Returns:
- Analyzer for the field with the given name. Assumed to be non-null
-
wrapComponents
protected Analyzer.TokenStreamComponents wrapComponents(String fieldName, Analyzer.TokenStreamComponents components) Description copied from class:AnalyzerWrapper
Wraps / alters the given TokenStreamComponents, taken from the wrapped Analyzer, to form new components. It is through this method that new TokenFilters can be added by AnalyzerWrappers. By default, the given components are returned.- Overrides:
wrapComponents
in classAnalyzerWrapper
- Parameters:
fieldName
- Name of the field which is to be analyzedcomponents
- TokenStreamComponents taken from the wrapped Analyzer- Returns:
- Wrapped / altered TokenStreamComponents.
-
toString
-