![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/01\/carbon-6.png\")
<\/p>\nHello everyone! We are back with another post. Today we will discuss in detail another important aspect used in many enterprise-grade applications. All organizations big and small in evidently store data some form of structured data store. Data import handler provides a mechanism for importing content of that datastore and indexing it. In addition to it, there can be HTTP\/RSS based data store along with ATOM feeds, email repositories, structures XML, so on and so forth.<\/p>\n
DIH handler is ideally configured in solrconfig.xml. The handler configuration itself is easy and demands less work. However, when implementing with varied types of data stores, the intrinsic complexity of it becomes pretty evident. For instance, Data Import Handler can be configured as follows :<\/p>\n
<\/p>\n
The config file here defines the information like, how to fetch data, what to fetch, the document structure conceived etc. The below example shows how to extract fields from four tables defining a simple product database. More information about the parameters and options shown here will be described in the sections following :<\/p>\n
![\"\"](\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/01\/carbon-7.png\")
<\/p>\n
dataSource<\/code>, in this case an HSQLDB database. The path to the JDBC driver and the JDBC URL and login credentials are all specified here. Other permissible attributes include whether or not to autocommit to Solr, the batchsize used in the JDBC connection, and a\u00a0readOnly<\/code> flag. The password attribute is optional if there is no password set for the DB.<\/li>\n- Alternately the password can be encrypted as follows. This is the value obtained as a result of the command\u00a0
openssl enc -aes-128-cbc -a -salt -in pwd.txt password=\"U2FsdGVkX18QMjY0yfCqlfBMvAB4d3XkwY96L7gfO2o=\"<\/code>. When the password is encrypted, you must provide an extra attribute\u00a0encryptKeyFile=\"\/location\/of\/encryptionkey\"<\/code>. This file should a text file with a single line containing the encrypt\/decrypt password.<\/li>\n- A\u00a0
document<\/code>\u00a0element follows, containing multiple\u00a0entity<\/code>\u00a0elements. Note that\u00a0entity<\/code>\u00a0elements can be nested, and this allows the entity relationships in the sample database to be mirrored here, so that we can generate a denormalized Solr record which may include multiple features for one item, for instance.<\/li>\n- The possible attributes for the\u00a0
entity<\/code>\u00a0element are described in later sections. Entity elements may contain one or more\u00a0field<\/code>\u00a0elements, which map the data source field names to Solr fields, and optionally specify per-field transformations. This entity is the\u00a0root<\/code>\u00a0entity.<\/li>\n- This entity is nested and reflects the one-to-many relationship between an item and its multiple features. Note the use of variables;\u00a0
${item.ID}<\/code>\u00a0is the value of the column ‘ID’ for the current item (item<\/code> referring to the entity name).<\/li>\n<\/ul>\n\nDatasources can still be specified in\u00a0solrconfig.xml<\/code>. These must be specified in the defaults section of the handler in\u00a0solrconfig.xml<\/code>. However, these are not parsed until the main configuration is loaded.<\/p>\n<\/div>\n\nThe entire configuration itself can be passed as a request parameter using the\u00a0dataConfig<\/code>\u00a0parameter rather than using a file. When configuration errors are encountered, the error message is returned in XML format.<\/p>\n<\/div>\n\nA\u00a0reload-config<\/code>\u00a0command is also supported, which is useful for validating a new configuration file, or if you want to specify a file, load it, and not have it reloaded again on import. If there is an\u00a0xml<\/code>\u00a0mistake in the configuration a user-friendly message is returned in\u00a0xml<\/code>\u00a0format. You can then fix the problem and do a\u00a0reload-config.<\/code><\/p>\n<\/div>\nImportant Commands<\/h3>\n\n\n- abort<\/strong><\/dt>\n
- Aborts an ongoing operation. For example:\u00a0
http:\/\/localhost:8983\/solr\/dih\/dataimport?command=abort<\/code>.<\/dd>\n- delta-import<\/strong><\/dt>\n
- For incremental imports and change detection. Only the\u00a0SqlEntityProcessor<\/a>\u00a0supports delta imports.<\/p>\n\n
For example:\u00a0http:\/\/localhost:8983\/solr\/dih\/dataimport?command=delta-import<\/code>.<\/p>\n<\/div>\n\nThis command supports the same\u00a0clean<\/code>,\u00a0commit<\/code>,\u00a0optimize<\/code>\u00a0and\u00a0debug<\/code>\u00a0parameters as\u00a0full-import<\/code>\u00a0command described below.<\/p>\n<\/div>\n<\/dd>\n- full-import<\/strong><\/dt>\n
- A Full Import operation can be started with a URL such as\u00a0
http:\/\/localhost:8983\/solr\/dih\/dataimport?command=full-import<\/code>. The command returns immediately.<\/p>\n\nThe operation will be started in a new thread and the\u00a0status<\/em>\u00a0attribute in the response should be shown as\u00a0busy<\/em>. The operation may take some time depending on the size of dataset. Queries to Solr are not blocked during full-imports.<\/p>\n<\/div>\n\nWhen a\u00a0full-import<\/code>\u00a0command is executed, it stores the start time of the operation in a file located at\u00a0conf\/dataimport.properties<\/code>. This stored timestamp is used when a\u00a0delta-import<\/code>\u00a0operation is executed.<\/p>\n<\/div>\n\nCommands available to\u00a0full-import<\/code>\u00a0are:<\/p>\n<\/div>\n\n\n- clean<\/strong><\/dt>\n
- Default is true. Tells whether to clean up the index before the indexing is started.<\/dd>\n
- commit<\/strong><\/dt>\n
- Default is true. Tells whether to commit after the operation.<\/dd>\n
- debug<\/strong><\/dt>\n
- Default is false. Runs the command in debug mode and is used by the interactive development mode.<\/p>\n\n
Note that in debug mode, documents are never committed automatically. If you want to run debug mode and commit the results too, add\u00a0commit=true<\/code>\u00a0as a request parameter.<\/p>\n<\/div>\n<\/dd>\n- entity<\/strong><\/dt>\n
- The name of an entity directly under the\u00a0
<document><\/code>\u00a0tag in the configuration file. Use this to execute one or more entities selectively.<\/p>\n\nMultiple “entity” parameters can be passed on to run multiple entities at once. If nothing is passed, all entities are executed.<\/p>\n<\/div>\n<\/dd>\n
- optimize<\/strong><\/dt>\n
- Default is true. Tells Solr whether to optimize after the operation.<\/dd>\n
- synchronous<\/strong><\/dt>\n
- Blocks request until import is completed. Default is false.<\/dd>\n<\/dl>\n<\/div>\n<\/dd>\n
- reload-config<\/strong><\/dt>\n
- If the configuration file has been changed and you wish to reload it without restarting Solr, run the command\u00a0
http:\/\/localhost:8983\/solr\/dih\/dataimport?command=reload-config<\/code><\/dd>\n- status<\/strong><\/dt>\n
- This command returns statistics on the number of documents created, deleted, queries run, rows fetched, status, and so on. For example:\u00a0
http:\/\/localhost:8983\/solr\/dih\/dataimport?command=status<\/code>.<\/dd>\n- show<\/strong>–config<\/strong><\/dt>\n
- This command responds with configuration:\u00a0
http:\/\/localhost:8983\/solr\/dih\/dataimport?command=show-config<\/code>.<\/dd>\n<\/dl>\nSome Important Processors<\/strong><\/h3>\n<\/div>\n\nEntity processors extract data, transform it, and add it to a Solr index. Examples of entities include views or tables in a data store.<\/p>\n<\/div>\n
\nEach processor has its own set of attributes, described in its own section below. In addition, there are several attributes common to all entities which may be specified:<\/p>\n<\/div>\n
\n\n- dataSource<\/strong><\/dt>\n
- The name of a data source. If there are multiple data sources defined, use this attribute with the name of the data source for this entity.<\/dd>\n
- name<\/strong><\/dt>\n
- Required. The unique name used to identify an entity.<\/dd>\n
- pk<\/strong><\/dt>\n
- The primary key for the entity. It is optional, and required only when using delta-imports. It has no relation to the uniqueKey defined in\u00a0
schema.xml<\/code>\u00a0but they can both be the same.<\/p>\n\nThis attribute is mandatory if you do delta-imports and then refer to the column name in\u00a0${dataimporter.delta.<column-name>}<\/code>\u00a0which is used as the primary key.<\/p>\n<\/div>\n<\/dd>\n- processor<\/strong><\/dt>\n
- Default is\u00a0SqlEntityProcessor<\/a>. Required only if the datasource is not RDBMS.<\/dd>\n
- onError<\/strong><\/dt>\n
- Defines what to do if an error is encountered.<\/p>\n\n
Permissible values are:<\/p>\n<\/div>\n
\n\n- abort<\/dt>\n
- Stops the import.<\/dd>\n
- skip<\/dt>\n
- Skips the current document.<\/dd>\n
- continue<\/dt>\n
- Ignores the error and processing continues.<\/dd>\n<\/dl>\n<\/div>\n<\/dd>\n
- preImportDeleteQuery<\/strong><\/dt>\n
- Before a\u00a0
full-import<\/code>\u00a0command, use this query this to cleanup the index instead of using\u00a0*:*<\/code>. This is honored only on an entity that is an immediate sub-child of\u00a0<document><\/code>.<\/dd>\n- postImportDeleteQuery<\/strong><\/dt>\n
- Similar to\u00a0
preImportDeleteQuery<\/code>, but it executes after the import has completed.<\/dd>\n- rootEntity<\/strong><\/dt>\n
- By default the entities immediately under\u00a0
<document><\/code>\u00a0are root entities. If this attribute is set to false, the entity directly falling under that entity will be treated as the root entity (and so on). For every row returned by the root entity, a document is created in Solr.<\/dd>\n- transformer<\/strong><\/dt>\n
- Optional. One or more transformers to be applied on this entity.<\/dd>\n
- cacheImpl<\/strong><\/dt>\n
- Optional. A class (which must implement\u00a0
DIHCache<\/code>) to use for caching this entity when doing lookups from an entity which wraps it. Provided implementation is\u00a0SortedMapBackedCache<\/code>.<\/dd>\n- cacheKey<\/strong><\/dt>\n
- The name of a property of this entity to use as a cache key if\u00a0
cacheImpl<\/code>\u00a0is specified.<\/dd>\n- cacheLookup<\/strong><\/dt>\n
- An entity + property name that will be used to lookup cached instances of this entity if\u00a0
cacheImpl<\/code>\u00a0is specified.<\/dd>\n- where<\/strong><\/dt>\n
- An alternative way to specify\u00a0
cacheKey<\/code>\u00a0and\u00a0cacheLookup<\/code>\u00a0concatenated with ‘=’.<\/p>\n\nFor example,\u00a0where=\"CODE=People.COUNTRY_CODE\"<\/code>\u00a0is equivalent to\u00a0cacheKey=\"CODE\" cacheLookup=\"People.COUNTRY_CODE\"<\/code><\/p>\n<\/div>\n<\/dd>\n- child=”true”<\/strong><\/dt>\n
- Enables indexing document blocks aka\u00a0Nested Child Documents<\/a>\u00a0for searching with\u00a0Block Join Query Parsers<\/a>. It can be only specified on the\u00a0
<entity><\/code>\u00a0element under another root entity. It switches from default behavior (merging field values) to nesting documents as children documents.<\/p>\n\nNote: parent\u00a0<entity><\/code>\u00a0should add a field which is used as a parent filter in query time.<\/p>\n<\/div>\n<\/dd>\n- join=”zipper”<\/strong><\/dt>\n
- Enables merge join, aka “zipper” algorithm, for joining parent and child entities without cache. It should be specified at child (nested)\u00a0
<entity><\/code>. It implies that parent and child queries return results ordered by keys, otherwise it throws an exception. Keys should be specified either with\u00a0where<\/code>\u00a0attribute or with\u00a0cacheKey<\/code>\u00a0and\u00a0cacheLookup<\/code>.<\/dd>\n<\/dl>\n<\/div>\n\n\nThe SQL Entity Processor<\/h3>\n\nThe SqlEntityProcessor is the default processor. The associated\u00a0JdbcDataSource<\/a>\u00a0should be a JDBC URL.<\/p>\n<\/div>\n\nThe entity attributes specific to this processor are shown in the table below. These are in addition to the attributes common to all entity processors described above.<\/p>\n<\/div>\n
\n\n- query<\/strong><\/dt>\n
- Required. The SQL query used to select rows.<\/dd>\n
- deltaQuery<\/strong><\/dt>\n
- SQL query used if the operation is delta-import. This query selects the primary keys of the rows which will be parts of the delta-update. The pks will be available to the deltaImportQuery through the variable\u00a0
${dataimporter.delta.<column-name>}<\/code>.<\/dd>\n- parentDeltaQuery<\/strong><\/dt>\n
- SQL query used if the operation is\u00a0
delta-import<\/code>.<\/dd>\n- deletedPkQuery<\/strong><\/dt>\n
- SQL query used if the operation is\u00a0
delta-import<\/code>.<\/dd>\n- deltaImportQuery<\/strong><\/dt>\n
- SQL query used if the operation is\u00a0
delta-import<\/code>. If this is not present, DIH tries to construct the import query by (after identifying the delta) modifying the ‘query’ (this is error prone).<\/p>\n\nThere is a namespace\u00a0${dataimporter.delta.<column-name>}<\/code>\u00a0which can be used in this query. For example,\u00a0select * from tbl where id=${dataimporter.delta.id}<\/code>.<\/p>\n<\/div>\n<\/dd>\n<\/dl>\n<\/div>\n<\/div>\n\nThe XPathEntityProcessor<\/h3>\n\nThis processor is used when indexing XML formatted data. The data source is typically\u00a0URLDataSource<\/a>\u00a0or\u00a0FileDataSource<\/a>. XPath can also be used with the\u00a0FileListEntityProcessor<\/a>\u00a0described below, to generate a document from each file.<\/p>\n<\/div>\n\nThe entity attributes unique to this processor are shown below. These are in addition to the attributes common to all entity processors described above.<\/p>\n<\/div>\n
\n\n- Processor<\/strong><\/dt>\n
- Required. Must be set to\u00a0
XpathEntityProcessor<\/code>.<\/dd>\n- url<\/strong><\/dt>\n
- Required. The HTTP URL or file location.<\/dd>\n
- stream<\/strong><\/dt>\n
- Optional: Set to true for a large file or download.<\/dd>\n
- forEach<\/strong><\/dt>\n
- Required unless you define\u00a0
useSolrAddSchema<\/code>. The XPath expression which demarcates each record. This will be used to set up the processing loop.<\/dd>\n- xsl<\/strong><\/dt>\n
- Optional: Its value (a URL or filesystem path) is the name of a resource used as a preprocessor for applying the XSL transformation.<\/dd>\n
- useSolrAddSchema<\/strong><\/dt>\n
- Set this to true if the content is in the form of the standard Solr update XML schema.<\/dd>\n<\/dl>\n<\/div>\n\n
Each<\/strong>\u00a0<field><\/code>\u00a0element in the entity can have the following attributes as well as the default ones.<\/p>\n<\/div>\n\n\n- xpath<\/strong><\/dt>\n
- Required. The XPath expression which will extract the content from the record for this field. Only a subset of XPath syntax is supported.<\/dd>\n
- commonField<\/strong><\/dt>\n
- Optional. If true, then when this field is encountered in a record it will be copied to future records when creating a Solr document.<\/dd>\n
- flatten<\/strong><\/dt>\n
- Optional. If set to true, then any children text nodes are collected to form the value of a field.<\/dd>\n<\/dl>\n
The TikaEntityProcessor<\/h3>\n\nThe TikaEntityProcessor uses Apache Tika to process incoming documents. This is similar to\u00a0