This section contains information about tokenizers and filters related to character set conversion or for use with specific languages.<\/p>\n<\/div>\n
For the European languages, tokenization is fairly straightforward. Tokens are delimited by white space and\/or a relatively small set of punctuation characters.<\/p>\n<\/div>\n
In other languages the tokenization rules are often not so simple. Some European languages may also require special tokenization rules, such as rules for decompounding German words.<\/p>\n
KeywordMarkerFilterFactory<\/h2>\n
Protects words from being modified by stemmers. A customized protected word list may be specified with the “protected” attribute in the schema. Any words in the protected word list will not be modified by any stemmer in Solr.<\/p>\n
![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/carbon-98.png\")
<\/p>\n
KeywordRepeatFilterFactory<\/h2>\n\nEmits each token twice, one with the\u00a0KEYWORD<\/code>\u00a0attribute and once without.<\/p>\n<\/div>\n\nIf placed before a stemmer, the result will be that you will get the unstemmed token preserved on the same position as the stemmed one. Queries matching the original exact term will get a better score while still maintaining the recall benefit of stemming. Another advantage of keeping the original token is that wildcard truncation will work as expected.<\/p>\n<\/div>\n
\nTo configure, add the\u00a0KeywordRepeatFilterFactory<\/code>\u00a0early in the analysis chain. It is recommended to also include\u00a0RemoveDuplicatesTokenFilterFactory<\/code>\u00a0to avoid duplicates when tokens are not stemmed.<\/p>\nA sample fieldType configuration could look like this:<\/p>\n
![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/carbon-99.png\")
<\/p>\n
StemmerOverrideFilterFactory<\/h2>\n\nOverrides stemming algorithms by applying a custom mapping, then protecting these terms from being modified by stemmers.<\/p>\n<\/div>\n
\nA customized mapping of words to stems, in a tab-separated file, can be specified to the “dictionary” attribute in the schema. Words in this mapping will be stemmed to the stems from the file, and will not be further changed by any stemmer.<\/p>\n
![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/carbon-100.png\")
<\/p>\n
Dictionary Compound Word Token Filter<\/h2>\n\nThis filter splits, or\u00a0decompounds<\/em>, compound words into individual words using a dictionary of the component words. Each input token is passed through unchanged. If it can also be decompounded into subwords, each subword is also added to the stream at the same logical position.<\/p>\n<\/div>\n\nCompound words are most commonly found in Germanic languages.<\/p>\n
\nFactory class:<\/strong>\u00a0solr.DictionaryCompoundWordTokenFilterFactory<\/code><\/p>\n<\/div>\n\nArguments:<\/strong><\/p>\n<\/div>\n\n\ndictionary<\/code><\/dt>\n- (required) The path of a file that contains a list of simple words, one per line. Blank lines and lines that begin with “#” are ignored. This path may be an absolute path, or path relative to the Solr config directory.<\/dd>\n
minWordSize<\/code><\/dt>\n- (integer, default 5) Any token shorter than this is not decompounded.<\/dd>\n
minSubwordSize<\/code><\/dt>\n- (integer, default 2) Subwords shorter than this are not emitted as tokens.<\/dd>\n
maxSubwordSize<\/code><\/dt>\n- (integer, default 15) Subwords longer than this are not emitted as tokens.<\/dd>\n
onlyLongestMatch<\/code><\/dt>\n- (true\/false) If true (the default), only the longest matching subwords will generate new tokens.<\/dd>\n<\/dl>\n\n
Example:<\/strong><\/p>\n<\/div>\n\nAssume that\u00a0germanwords.txt<\/code>\u00a0contains at least the following words:\u00a0dumm kopf donau dampf schiff<\/code><\/p>\n![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/carbon-2020-04-04T223054.317.png\")
<\/p>\n
\nIn:<\/strong>\u00a0“Donaudampfschiff dummkopf”<\/p>\n<\/div>\n\nTokenizer to Filter:<\/strong>\u00a0“Donaudampfschiff”(1), “dummkopf”(2),<\/p>\n<\/div>\n\nOut:<\/strong>\u00a0“Donaudampfschiff”(1), “Donau”(1), “dampf”(1), “schiff”(1), “dummkopf”(2), “dumm”(2), “kopf”(2)<\/p>\nUnicode Collation<\/h2>\n\n\nUnicode Collation is a language-sensitive method of sorting text that can also be used for advanced search purposes.<\/p>\n<\/div>\n
\nUnicode Collation in Solr is fast, because all the work is done at index time.<\/p>\n<\/div>\n
\nRather than specifying an analyzer within\u00a0<fieldtype \u2026\u200b class=\"solr.TextField\"><\/code>, the\u00a0solr.CollationField<\/code>\u00a0and\u00a0solr.ICUCollationField<\/code>\u00a0field type classes provide this functionality.\u00a0solr.ICUCollationField<\/code>, which is backed by\u00a0the ICU4J library<\/a>, provides more flexible configuration, has more locales, is significantly faster, and requires less memory and less index space, since its keys are smaller than those produced by the JDK implementation that backs\u00a0solr.CollationField<\/code>.<\/p>\n<\/div>\n\nsolr.ICUCollationField<\/code>\u00a0is included in the Solr\u00a0analysis-extras<\/code>\u00a0contrib – see\u00a0solr\/contrib\/analysis-extras\/README.txt<\/code>\u00a0for instructions on which jars you need to add to your\u00a0SOLR_HOME\/lib<\/code>\u00a0in order to use it.<\/p>\n<\/div>\n\nsolr.ICUCollationField<\/code>\u00a0and\u00a0solr.CollationField<\/code>\u00a0fields can be created in two ways:<\/p>\n<\/div>\n\n\n- Based upon a system collator associated with a Locale.<\/li>\n
- Based upon a tailored\u00a0
RuleBasedCollator<\/code>\u00a0ruleset.<\/li>\n<\/ul>\n<\/div>\n\nArguments for\u00a0solr.ICUCollationField<\/code>, specified as attributes within the\u00a0<fieldtype><\/code>\u00a0element:<\/strong><\/p>\n<\/div>\n\nUsing a System collator:<\/p>\n<\/div>\n
\n\nlocale<\/code><\/dt>\n- (required)\u00a0RFC 3066<\/a>\u00a0locale ID. See\u00a0the ICU locale explorer<\/a>\u00a0for a list of supported locales.<\/dd>\n
strength<\/code><\/dt>\n- Valid values are\u00a0
primary<\/code>,\u00a0secondary<\/code>,\u00a0tertiary<\/code>,\u00a0quaternary<\/code>, or\u00a0identical<\/code>. See\u00a0Comparison Levels in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\ndecomposition<\/code><\/dt>\n- Valid values are\u00a0
no<\/code>\u00a0or\u00a0canonical<\/code>. See\u00a0Normalization in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\n<\/dl>\n<\/div>\n\nUsing a Tailored ruleset:<\/p>\n<\/div>\n
\n\ncustom<\/code><\/dt>\n- (required) Path to a UTF-8 text file containing rules supported by the ICU\u00a0
RuleBasedCollator<\/code><\/a><\/dd>\nstrength<\/code><\/dt>\n- Valid values are\u00a0
primary<\/code>,\u00a0secondary<\/code>,\u00a0tertiary<\/code>,\u00a0quaternary<\/code>, or\u00a0identical<\/code>. See\u00a0Comparison Levels in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\ndecomposition<\/code><\/dt>\n- Valid values are\u00a0
no<\/code>\u00a0or\u00a0canonical<\/code>. See\u00a0Normalization in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\n<\/dl>\n<\/div>\n
Emits each token twice, one with the\u00a0 If placed before a stemmer, the result will be that you will get the unstemmed token preserved on the same position as the stemmed one. Queries matching the original exact term will get a better score while still maintaining the recall benefit of stemming. Another advantage of keeping the original token is that wildcard truncation will work as expected.<\/p>\n<\/div>\n To configure, add the\u00a0 A sample fieldType configuration could look like this:<\/p>\n Overrides stemming algorithms by applying a custom mapping, then protecting these terms from being modified by stemmers.<\/p>\n<\/div>\n A customized mapping of words to stems, in a tab-separated file, can be specified to the “dictionary” attribute in the schema. Words in this mapping will be stemmed to the stems from the file, and will not be further changed by any stemmer.<\/p>\n This filter splits, or\u00a0decompounds<\/em>, compound words into individual words using a dictionary of the component words. Each input token is passed through unchanged. If it can also be decompounded into subwords, each subword is also added to the stream at the same logical position.<\/p>\n<\/div>\n Compound words are most commonly found in Germanic languages.<\/p>\n Factory class:<\/strong>\u00a0 Arguments:<\/strong><\/p>\n<\/div>\n Example:<\/strong><\/p>\n<\/div>\n Assume that\u00a0 In:<\/strong>\u00a0“Donaudampfschiff dummkopf”<\/p>\n<\/div>\n Tokenizer to Filter:<\/strong>\u00a0“Donaudampfschiff”(1), “dummkopf”(2),<\/p>\n<\/div>\n Out:<\/strong>\u00a0“Donaudampfschiff”(1), “Donau”(1), “dampf”(1), “schiff”(1), “dummkopf”(2), “dumm”(2), “kopf”(2)<\/p>\n Unicode Collation is a language-sensitive method of sorting text that can also be used for advanced search purposes.<\/p>\n<\/div>\n Unicode Collation in Solr is fast, because all the work is done at index time.<\/p>\n<\/div>\n Rather than specifying an analyzer within\u00a0 Arguments for\u00a0 Using a System collator:<\/p>\n<\/div>\n Using a Tailored ruleset:<\/p>\n<\/div>\nKEYWORD<\/code>\u00a0attribute and once without.<\/p>\n<\/div>\nKeywordRepeatFilterFactory<\/code>\u00a0early in the analysis chain. It is recommended to also include\u00a0RemoveDuplicatesTokenFilterFactory<\/code>\u00a0to avoid duplicates when tokens are not stemmed.<\/p>\n<\/p>\nStemmerOverrideFilterFactory<\/h2>\n
<\/p>\nDictionary Compound Word Token Filter<\/h2>\n
solr.DictionaryCompoundWordTokenFilterFactory<\/code><\/p>\n<\/div>\n\n
dictionary<\/code><\/dt>\nminWordSize<\/code><\/dt>\nminSubwordSize<\/code><\/dt>\nmaxSubwordSize<\/code><\/dt>\nonlyLongestMatch<\/code><\/dt>\ngermanwords.txt<\/code>\u00a0contains at least the following words:\u00a0dumm kopf donau dampf schiff<\/code><\/p>\n<\/p>\nUnicode Collation<\/h2>\n
<fieldtype \u2026\u200b class=\"solr.TextField\"><\/code>, the\u00a0solr.CollationField<\/code>\u00a0and\u00a0solr.ICUCollationField<\/code>\u00a0field type classes provide this functionality.\u00a0solr.ICUCollationField<\/code>, which is backed by\u00a0the ICU4J library<\/a>, provides more flexible configuration, has more locales, is significantly faster, and requires less memory and less index space, since its keys are smaller than those produced by the JDK implementation that backs\u00a0solr.CollationField<\/code>.<\/p>\n<\/div>\nsolr.ICUCollationField<\/code>\u00a0is included in the Solr\u00a0analysis-extras<\/code>\u00a0contrib – see\u00a0solr\/contrib\/analysis-extras\/README.txt<\/code>\u00a0for instructions on which jars you need to add to your\u00a0SOLR_HOME\/lib<\/code>\u00a0in order to use it.<\/p>\n<\/div>\nsolr.ICUCollationField<\/code>\u00a0and\u00a0solr.CollationField<\/code>\u00a0fields can be created in two ways:<\/p>\n<\/div>\n\n
RuleBasedCollator<\/code>\u00a0ruleset.<\/li>\n<\/ul>\n<\/div>\nsolr.ICUCollationField<\/code>, specified as attributes within the\u00a0<fieldtype><\/code>\u00a0element:<\/strong><\/p>\n<\/div>\n\n
locale<\/code><\/dt>\nstrength<\/code><\/dt>\nprimary<\/code>,\u00a0secondary<\/code>,\u00a0tertiary<\/code>,\u00a0quaternary<\/code>, or\u00a0identical<\/code>. See\u00a0Comparison Levels in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\ndecomposition<\/code><\/dt>\nno<\/code>\u00a0or\u00a0canonical<\/code>. See\u00a0Normalization in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\n<\/dl>\n<\/div>\n\n
custom<\/code><\/dt>\nRuleBasedCollator<\/code><\/a><\/dd>\nstrength<\/code><\/dt>\nprimary<\/code>,\u00a0secondary<\/code>,\u00a0tertiary<\/code>,\u00a0quaternary<\/code>, or\u00a0identical<\/code>. See\u00a0Comparison Levels in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\ndecomposition<\/code><\/dt>\nno<\/code>\u00a0or\u00a0canonical<\/code>. See\u00a0Normalization in ICU Collation Concepts<\/a>\u00a0for more information.<\/dd>\n<\/dl>\n<\/div>\n