{"id":1764,"date":"2021-04-19T08:33:53","date_gmt":"2021-04-19T08:33:53","guid":{"rendered":"https:\/\/www.aeologic.com\/blog\/?p=1764"},"modified":"2021-04-19T09:38:24","modified_gmt":"2021-04-19T09:38:24","slug":"distributed-search-with-index-sharding-ultimate-solr-guide","status":"publish","type":"post","link":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/","title":{"rendered":"Distributed Search – Ultimate Solr Guide"},"content":{"rendered":"
When using traditional index sharding, you will need to consider how to query your documents.It is highly recommended that you use\u00a0SolrCloud<\/a> when needing to scale up or scale-out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency, and other sanity checks that are expected out of a distributed system.Everything on this page is specific to the legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below.<\/p>\n

Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X).\u00a0deleteByQuery<\/strong>\u00a0also handles reorders the same way, to ensure replicas are consistent. All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas.<\/p>\n<\/section>\n

\n

Distributing Documents across Shards<\/h2>\n

When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm. Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode.<\/p>\n

In the legacy distributed mode, Solr does not calculate universal term\/doc frequencies. For most large-scale implementations, it is not likely to matter that Solr calculates TF\/IDF at the shard level. However, if your collection is heavily skewed in its distribution across servers, you may find misleading relevancy results in your searches. In general, it is probably best to randomly distribute documents to your shards.<\/p>\n<\/section>\n

\n

Executing Distributed Searches with the shards Parameter<\/h2>\n

If a query request includes the\u00a0shards<\/code>\u00a0parameter, the Solr server distributes the request across all the shards listed as arguments to the parameter. The\u00a0shards<\/code>\u00a0parameter uses this syntax:<\/p>\n

host:port\/base_url,host:port\/base_url*<\/code><\/p>\n

For example, the\u00a0shards<\/code>\u00a0parameter below causes the search to be distributed across two Solr servers:\u00a0solr1<\/strong>\u00a0and\u00a0solr2<\/strong>, both of which are running on port 8983:<\/p>\n

http:\/\/localhost:8983\/solr\/core1\/select?shards=solr1:8983\/solr\/core1,solr2:8983\/solr\/core1&indent=true&q=ipod+solr<\/code><\/p>\n

Rather than require users to include the shards parameter explicitly, it is usually preferred to configure this parameter as a default in the RequestHandler section of\u00a0solrconfig.xml<\/code>.<\/p>\n

\n\n\n\n
<\/td>\nDo not add the\u00a0shards<\/code> parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the parameter, and pass distributed search requests to that handler.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

With Legacy mode, only query requests are distributed. This includes requests to the SearchHandler (or any handler extending from\u00a0org.apache.solr.handler.component.SearchHandler<\/code>) using standard components that support distributed search.<\/p>\n

As in SolrCloud mode, when\u00a0shards.info=true<\/code>, distributed responses will include information about the shard (where each shard represents a logically different index or physical location)<\/p>\n

The following components support distributed search:<\/p>\n

\n
    \n
  • The\u00a0Query<\/strong>\u00a0component, which returns documents matching a query<\/li>\n
  • The\u00a0Facet<\/strong> component, which processes facet. query and facet.field requests where facets are sorted by count (the default).<\/li>\n
  • The\u00a0Highlighting<\/strong>\u00a0component, which enables Solr to include “highlighted” matches in field values.<\/li>\n
  • The\u00a0Stats<\/strong>\u00a0component, which returns simple statistics for numeric fields within the DocSet.<\/li>\n
  • The\u00a0Debug<\/strong>\u00a0component, which helps with debugging.<\/li>\n<\/ul>\n<\/div>\n
    \n

    Shards Whitelist<\/h3>\n

    The nodes allowed in the\u00a0shards<\/code>\u00a0parameter is configurable through the\u00a0shardsWhitelist<\/code>\u00a0property in\u00a0solr.xml<\/code>. This whitelist is automatically configured for SolrCloud but needs explicit configuration for leader\/follower mode.<\/p>\n

    Limitations to Distributed Search<\/strong><\/p>\n<\/section>\n<\/section>\n

    Distributed searching in Solr has the following limitations:<\/p>\n
    \n
      \n
    • Each document indexed must have a unique key.<\/li>\n
    • If Solr discovers duplicate document IDs, Solr selects the first document and discards subsequent documents.<\/li>\n
    • The index for distributed searching may become momentarily out of sync if a commit happens between the first and second phase of the distributed search. This might cause a situation where a document that once matched a query and was subsequently changed may no longer match the query but will still be retrieved. This situation is expected to be quite rare, however, and is only possible for a single query request.<\/li>\n
    • The number of shards is limited by number of characters allowed for GET method\u2019s URI; most Web servers generally support at least 4000 characters, but many servers limit URI length to reduce their vulnerability to Denial of Service (DoS) attacks.<\/li>\n
    • Shard information can be returned with each document in a distributed search by including\u00a0fl=id, [shard]<\/code>\u00a0in the search request. This returns the shard URL.<\/li>\n
    • In a distributed search, the data directory from the core descriptor overrides any data directory in\u00a0solrconfig.xml.<\/code><\/li>\n
    • Update commands may be sent to any server with distributed indexing configured correctly. Document adds and deletes are forwarded to the appropriate server\/shard based on a hash of the unique document id.\u00a0commit<\/strong>\u00a0commands and\u00a0deleteByQuery<\/strong>\u00a0commands are sent to every server in\u00a0shards<\/code>.<\/li>\n<\/ul>\n<\/div>\n

      Formerly a limitation was that TF\/IDF relevancy computations only used shard-local statistics. This is still the case by default. If your data isn\u2019t randomly distributed, or if you want more exact statistics, then remember to configure the ExactStatsCache.<\/p>\n<\/section>\n

      \n

      Avoiding Distributed Deadlock with Distributed Search<\/h2>\n

      Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock.<\/p>\n<\/section>\n

      \n

      Testing Index Sharding on Two Local Servers<\/h2>\n

      For simple functional testing, it\u2019s easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.)<\/p>\n

      \n
        \n
      1. Make two Solr home directories and copy\u00a0solr.xml<\/code>\u00a0into the new directories:\n
        \n
        <\/code><\/pre>\n
        mkdir example\/nodes\r\nmkdir example\/nodes\/node1\r\n# Copy solr.xml into this solr.home\r\ncp server\/solr\/solr.xml example\/nodes\/node1\/.\r\n# Repeat the above steps for the second node\r\nmkdir example\/nodes\/node2\r\ncp server\/solr\/solr.xml example\/nodes\/node2\/.<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n<\/li>\n
      2. Start the two Solr instances\n
        \n
        <\/code><\/pre>\n
        # Start first node on port 8983\r\nbin\/solr start -s example\/nodes\/node1 -p 8983\r\n\r\n# Start second node on port 8984\r\nbin\/solr start -s example\/nodes\/node2 -p 8984<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n<\/li>\n
      3. Create a core on both the nodes with the sample_techproducts_configs.\n
        \n
        <\/code><\/pre>\n
        bin\/solr create_core -c core1 -p 8983 -d sample_techproducts_configs\r\n# Create a core on the Solr node running on port 8984\r\nbin\/solr create_core -c core1 -p 8984 -d sample_techproducts_configs<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n<\/li>\n
      4. In a third window, index an example document to each of the server:\n
        \n
        <\/code><\/pre>\n
        bin\/post -c core1 example\/exampledocs\/monitor.xml -port 8983\r\n\r\nbin\/post -c core1 example\/exampledocs\/monitor2.xml -port 8984<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n<\/li>\n
      5. Search on the node on port 8983:\n
        \n
        <\/code><\/pre>\n
        curl http:\/\/localhost:8983\/solr\/core1\/select?q=*:*&wt=xml&indent=true<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n
        This should bring back one document.<\/p>\n
        Search on the node on port 8984:<\/p>\n
        \n
        <\/code><\/pre>\n
        curl http:\/\/localhost:8984\/solr\/core1\/select?q=*:*&wt=xml&indent=true<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n
        This should also bring back a single document.<\/p>\n
        Now do a distributed search across both servers with your browser or\u00a0curl.<\/code>\u00a0In the example below, an extra parameter ‘fl’ is passed to restrict the returned fields to id and name.<\/p>\n
        \n
        <\/code><\/pre>\n
        curl http:\/\/localhost:8983\/solr\/core1\/select?q=*:*&indent=true&shards=localhost:8983\/solr\/core1,localhost:8984\/solr\/core1&fl=id,name&wt=xml<\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n
        This should contain both the documents as shown below:<\/p>\n
        \n
        <\/code><\/pre>\n
        <response>\r\n  <lst name=\"responseHeader\">\r\n    <int name=\"status\">0<\/int>\r\n    <int name=\"QTime\">8<\/int>\r\n    <lst name=\"params\">\r\n      <str name=\"q\">*:*<\/str>\r\n      <str name=\"shards\">localhost:8983\/solr\/core1,localhost:8984\/solr\/core1<\/str>\r\n      <str name=\"indent\">true<\/str>\r\n      <str name=\"fl\">id,name<\/str>\r\n      <str name=\"wt\">xml<\/str>\r\n    <\/lst>\r\n  <\/lst>\r\n  <result name=\"response\" numFound=\"2\" start=\"0\" maxScore=\"1.0\">\r\n    <doc>\r\n      <str name=\"id\">3007WFP<\/str>\r\n      <str name=\"name\">Dell Widescreen UltraSharp 3007WFP<\/str>\r\n    <\/doc>\r\n    <doc>\r\n      <str name=\"id\">VA902B<\/str>\r\n      <str name=\"name\">ViewSonic VA902B - flat panel display - TFT - 19\"<\/str>\r\n    <\/doc>\r\n  <\/result>\r\n<\/response><\/code><\/pre>\n
        <\/code><\/pre>\n<\/div>\n<\/li>\n<\/ol>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"
        When using traditional index sharding, you will need to consider how to query your documents.It is highly recommended that you use\u00a0SolrCloud when needing to scale up or scale-out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency, and other sanity checks that are expected out of a distributed system.Everything on this page is specific to the legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below. Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X).\u00a0deleteByQuery\u00a0also handles reorders the same way, to ensure replicas are consistent. All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas. Distributing Documents across Shards When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm. Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode. In the legacy distributed mode, Solr does not calculate universal term\/doc frequencies. For most large-scale implementations, it is not likely to matter that Solr calculates TF\/IDF at the shard level. However, if your collection is heavily skewed in its distribution across servers, you may find misleading relevancy results in your searches. In general, it is probably best to randomly distribute documents to your shards. Executing Distributed Searches with the shards Parameter If a query request includes the\u00a0shards\u00a0parameter, the Solr server distributes the request across all the shards listed as arguments to the parameter. The\u00a0shards\u00a0parameter uses this syntax: host:port\/base_url,host:port\/base_url* For example, the\u00a0shards\u00a0parameter below causes the search to be distributed across two Solr servers:\u00a0solr1\u00a0and\u00a0solr2, both of which are running on port 8983: http:\/\/localhost:8983\/solr\/core1\/select?shards=solr1:8983\/solr\/core1,solr2:8983\/solr\/core1&indent=true&q=ipod+solr Rather than require users to include the shards parameter explicitly, it is usually preferred to configure this parameter as a default in the RequestHandler section of\u00a0solrconfig.xml. Do not add the\u00a0shards parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the parameter, and pass distributed search requests to that handler. With Legacy mode, only query requests are distributed. This includes requests to the SearchHandler (or any handler extending from\u00a0org.apache.solr.handler.component.SearchHandler) using standard components that support distributed search. As in SolrCloud mode, when\u00a0shards.info=true, distributed responses will include information about the shard (where each shard represents a logically different index or physical location) The following components support distributed search: The\u00a0Query\u00a0component, which returns documents matching a query The\u00a0Facet component, which processes facet. query and facet.field requests where facets are sorted by count (the default). The\u00a0Highlighting\u00a0component, which enables Solr to include “highlighted” matches in field values. The\u00a0Stats\u00a0component, which returns simple statistics for numeric fields within the DocSet. The\u00a0Debug\u00a0component, which helps with debugging. Shards Whitelist The nodes allowed in the\u00a0shards\u00a0parameter is configurable through the\u00a0shardsWhitelist\u00a0property in\u00a0solr.xml. This whitelist is automatically configured for SolrCloud but needs explicit configuration for leader\/follower mode. Limitations to Distributed Search Distributed searching in Solr has the following limitations: Each document indexed must have a unique key. If Solr discovers duplicate document IDs, Solr selects the first document and discards subsequent documents. The index for distributed searching may become momentarily out of sync if a commit happens between the first and second phase of the distributed search. This might cause a situation where a document that once matched a query and was subsequently changed may no longer match the query but will still be retrieved. This situation is expected to be quite rare, however, and is only possible for a single query request. The number of shards is limited by number of characters allowed for GET method\u2019s URI; most Web servers generally support at least 4000 characters, but many servers limit URI length to reduce their vulnerability to Denial of Service (DoS) attacks. Shard information can be returned with each document in a distributed search by including\u00a0fl=id, [shard]\u00a0in the search request. This returns the shard URL. In a distributed search, the data directory from the core descriptor overrides any data directory in\u00a0solrconfig.xml. Update commands may be sent to any server with distributed indexing configured correctly. Document adds and deletes are forwarded to the appropriate server\/shard based on a hash of the unique document id.\u00a0commit\u00a0commands and\u00a0deleteByQuery\u00a0commands are sent to every server in\u00a0shards. Formerly a limitation was that TF\/IDF relevancy computations only used shard-local statistics. This is still the case by default. If your data isn\u2019t randomly distributed, or if you want more exact statistics, then remember to configure the ExactStatsCache. Avoiding Distributed Deadlock with Distributed Search Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock. Testing Index Sharding on Two Local Servers For simple functional testing, it\u2019s easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.) Make two Solr home directories and copy\u00a0solr.xml\u00a0into the new directories: mkdir example\/nodes mkdir example\/nodes\/node1 # Copy solr.xml into this solr.home cp server\/solr\/solr.xml example\/nodes\/node1\/. # Repeat the above steps for the second node mkdir example\/nodes\/node2 cp server\/solr\/solr.xml example\/nodes\/node2\/. Start the two Solr instances # Start first node on port 8983 bin\/solr start -s example\/nodes\/node1 -p 8983 # Start second node on port 8984 bin\/solr start -s example\/nodes\/node2 -p 8984 Create a core on both the nodes with the sample_techproducts_configs. bin\/solr create_core -c core1 -p 8983 -d sample_techproducts_configs # Create a core on the Solr node running on port 8984 bin\/solr create_core -c core1 -p 8984 -d sample_techproducts_configs In a third window, index an example document to each of the server: bin\/post -c core1 example\/exampledocs\/monitor.xml -port 8983 bin\/post -c core1 example\/exampledocs\/monitor2.xml -port 8984 Search on the node on port 8983: curl http:\/\/localhost:8983\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should bring back one document. Search on the node on port 8984: curl http:\/\/localhost:8984\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should also bring back a single document. Now do a distributed search across both servers with […]<\/p>\n","protected":false},"author":3,"featured_media":1767,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[15,22,7],"class_list":["post-1764","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-solr","tag-leucene","tag-shard","tag-solr"],"yoast_head":"\nDistributed Search - Ultimate Solr Guide - Aeologic Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Distributed Search - Ultimate Solr Guide - Aeologic Blog\" \/>\n<meta property=\"og:description\" content=\"When using traditional index sharding, you will need to consider how to query your documents.It is highly recommended that you use\u00a0SolrCloud when needing to scale up or scale-out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency, and other sanity checks that are expected out of a distributed system.Everything on this page is specific to the legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below. Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X).\u00a0deleteByQuery\u00a0also handles reorders the same way, to ensure replicas are consistent. All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas. Distributing Documents across Shards When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm. Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode. In the legacy distributed mode, Solr does not calculate universal term\/doc frequencies. For most large-scale implementations, it is not likely to matter that Solr calculates TF\/IDF at the shard level. However, if your collection is heavily skewed in its distribution across servers, you may find misleading relevancy results in your searches. In general, it is probably best to randomly distribute documents to your shards. Executing Distributed Searches with the shards Parameter If a query request includes the\u00a0shards\u00a0parameter, the Solr server distributes the request across all the shards listed as arguments to the parameter. The\u00a0shards\u00a0parameter uses this syntax: host:port\/base_url,host:port\/base_url* For example, the\u00a0shards\u00a0parameter below causes the search to be distributed across two Solr servers:\u00a0solr1\u00a0and\u00a0solr2, both of which are running on port 8983: http:\/\/localhost:8983\/solr\/core1\/select?shards=solr1:8983\/solr\/core1,solr2:8983\/solr\/core1&indent=true&q=ipod+solr Rather than require users to include the shards parameter explicitly, it is usually preferred to configure this parameter as a default in the RequestHandler section of\u00a0solrconfig.xml. Do not add the\u00a0shards parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the parameter, and pass distributed search requests to that handler. With Legacy mode, only query requests are distributed. This includes requests to the SearchHandler (or any handler extending from\u00a0org.apache.solr.handler.component.SearchHandler) using standard components that support distributed search. As in SolrCloud mode, when\u00a0shards.info=true, distributed responses will include information about the shard (where each shard represents a logically different index or physical location) The following components support distributed search: The\u00a0Query\u00a0component, which returns documents matching a query The\u00a0Facet component, which processes facet. query and facet.field requests where facets are sorted by count (the default). The\u00a0Highlighting\u00a0component, which enables Solr to include “highlighted” matches in field values. The\u00a0Stats\u00a0component, which returns simple statistics for numeric fields within the DocSet. The\u00a0Debug\u00a0component, which helps with debugging. Shards Whitelist The nodes allowed in the\u00a0shards\u00a0parameter is configurable through the\u00a0shardsWhitelist\u00a0property in\u00a0solr.xml. This whitelist is automatically configured for SolrCloud but needs explicit configuration for leader\/follower mode. Limitations to Distributed Search Distributed searching in Solr has the following limitations: Each document indexed must have a unique key. If Solr discovers duplicate document IDs, Solr selects the first document and discards subsequent documents. The index for distributed searching may become momentarily out of sync if a commit happens between the first and second phase of the distributed search. This might cause a situation where a document that once matched a query and was subsequently changed may no longer match the query but will still be retrieved. This situation is expected to be quite rare, however, and is only possible for a single query request. The number of shards is limited by number of characters allowed for GET method\u2019s URI; most Web servers generally support at least 4000 characters, but many servers limit URI length to reduce their vulnerability to Denial of Service (DoS) attacks. Shard information can be returned with each document in a distributed search by including\u00a0fl=id, [shard]\u00a0in the search request. This returns the shard URL. In a distributed search, the data directory from the core descriptor overrides any data directory in\u00a0solrconfig.xml. Update commands may be sent to any server with distributed indexing configured correctly. Document adds and deletes are forwarded to the appropriate server\/shard based on a hash of the unique document id.\u00a0commit\u00a0commands and\u00a0deleteByQuery\u00a0commands are sent to every server in\u00a0shards. Formerly a limitation was that TF\/IDF relevancy computations only used shard-local statistics. This is still the case by default. If your data isn\u2019t randomly distributed, or if you want more exact statistics, then remember to configure the ExactStatsCache. Avoiding Distributed Deadlock with Distributed Search Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock. Testing Index Sharding on Two Local Servers For simple functional testing, it\u2019s easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.) Make two Solr home directories and copy\u00a0solr.xml\u00a0into the new directories: mkdir example\/nodes mkdir example\/nodes\/node1 # Copy solr.xml into this solr.home cp server\/solr\/solr.xml example\/nodes\/node1\/. # Repeat the above steps for the second node mkdir example\/nodes\/node2 cp server\/solr\/solr.xml example\/nodes\/node2\/. Start the two Solr instances # Start first node on port 8983 bin\/solr start -s example\/nodes\/node1 -p 8983 # Start second node on port 8984 bin\/solr start -s example\/nodes\/node2 -p 8984 Create a core on both the nodes with the sample_techproducts_configs. bin\/solr create_core -c core1 -p 8983 -d sample_techproducts_configs # Create a core on the Solr node running on port 8984 bin\/solr create_core -c core1 -p 8984 -d sample_techproducts_configs In a third window, index an example document to each of the server: bin\/post -c core1 example\/exampledocs\/monitor.xml -port 8983 bin\/post -c core1 example\/exampledocs\/monitor2.xml -port 8984 Search on the node on port 8983: curl http:\/\/localhost:8983\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should bring back one document. Search on the node on port 8984: curl http:\/\/localhost:8984\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should also bring back a single document. Now do a distributed search across both servers with […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\" \/>\n<meta property=\"og:site_name\" content=\"Aeologic Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AeoLogicTech\/\" \/>\n<meta property=\"article:published_time\" content=\"2021-04-19T08:33:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-04-19T09:38:24+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1800\" \/>\n\t<meta property=\"og:image:height\" content=\"1037\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Manoj Kumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@aeologictech\" \/>\n<meta name=\"twitter:site\" content=\"@aeologictech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Manoj Kumar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\"},\"author\":{\"name\":\"Manoj Kumar\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4\"},\"headline\":\"Distributed Search – Ultimate Solr Guide\",\"datePublished\":\"2021-04-19T08:33:53+00:00\",\"dateModified\":\"2021-04-19T09:38:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\"},\"wordCount\":972,\"publisher\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png\",\"keywords\":[\"leucene\",\"shard\",\"solr\"],\"articleSection\":[\"Solr\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\",\"url\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\",\"name\":\"Distributed Search - Ultimate Solr Guide - Aeologic Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png\",\"datePublished\":\"2021-04-19T08:33:53+00:00\",\"dateModified\":\"2021-04-19T09:38:24+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage\",\"url\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png\",\"contentUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png\",\"width\":1800,\"height\":1037},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.aeologic.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Distributed Search – Ultimate Solr Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#website\",\"url\":\"https:\/\/www.aeologic.com\/blog\/\",\"name\":\"Aeologic Blog\",\"description\":\"Aeologic\",\"publisher\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.aeologic.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\",\"name\":\"AeoLogic Technologies\",\"url\":\"https:\/\/www.aeologic.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg\",\"contentUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg\",\"width\":385,\"height\":162,\"caption\":\"AeoLogic Technologies\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/AeoLogicTech\/\",\"https:\/\/x.com\/aeologictech\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4\",\"name\":\"Manoj Kumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g\",\"caption\":\"Manoj Kumar\"},\"description\":\"Manoj Kumar is a seasoned Digital Marketing Manager and passionate Tech Blogger with deep expertise in SEO, AI trends, and emerging digital technologies. He writes about innovative solutions that drive growth and transformation across industry. Featured on - YOURSTORY | TECHSLING | ELEARNINGINDUSTRY | DATASCIENCECENTRAL | TIMESOFINDIA | MEDIUM | DATAFLOQ\",\"sameAs\":[\"https:\/\/www.aeologic.com\/\",\"https:\/\/www.linkedin.com\/in\/manoj-kumar-rajput\/\"],\"url\":\"https:\/\/www.aeologic.com\/blog\/author\/manoj\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Distributed Search - Ultimate Solr Guide - Aeologic Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/","og_locale":"en_US","og_type":"article","og_title":"Distributed Search - Ultimate Solr Guide - Aeologic Blog","og_description":"When using traditional index sharding, you will need to consider how to query your documents.It is highly recommended that you use\u00a0SolrCloud when needing to scale up or scale-out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency, and other sanity checks that are expected out of a distributed system.Everything on this page is specific to the legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below. Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X).\u00a0deleteByQuery\u00a0also handles reorders the same way, to ensure replicas are consistent. All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas. Distributing Documents across Shards When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm. Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode. In the legacy distributed mode, Solr does not calculate universal term\/doc frequencies. For most large-scale implementations, it is not likely to matter that Solr calculates TF\/IDF at the shard level. However, if your collection is heavily skewed in its distribution across servers, you may find misleading relevancy results in your searches. In general, it is probably best to randomly distribute documents to your shards. Executing Distributed Searches with the shards Parameter If a query request includes the\u00a0shards\u00a0parameter, the Solr server distributes the request across all the shards listed as arguments to the parameter. The\u00a0shards\u00a0parameter uses this syntax: host:port\/base_url,host:port\/base_url* For example, the\u00a0shards\u00a0parameter below causes the search to be distributed across two Solr servers:\u00a0solr1\u00a0and\u00a0solr2, both of which are running on port 8983: http:\/\/localhost:8983\/solr\/core1\/select?shards=solr1:8983\/solr\/core1,solr2:8983\/solr\/core1&indent=true&q=ipod+solr Rather than require users to include the shards parameter explicitly, it is usually preferred to configure this parameter as a default in the RequestHandler section of\u00a0solrconfig.xml. Do not add the\u00a0shards parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the parameter, and pass distributed search requests to that handler. With Legacy mode, only query requests are distributed. This includes requests to the SearchHandler (or any handler extending from\u00a0org.apache.solr.handler.component.SearchHandler) using standard components that support distributed search. As in SolrCloud mode, when\u00a0shards.info=true, distributed responses will include information about the shard (where each shard represents a logically different index or physical location) The following components support distributed search: The\u00a0Query\u00a0component, which returns documents matching a query The\u00a0Facet component, which processes facet. query and facet.field requests where facets are sorted by count (the default). The\u00a0Highlighting\u00a0component, which enables Solr to include “highlighted” matches in field values. The\u00a0Stats\u00a0component, which returns simple statistics for numeric fields within the DocSet. The\u00a0Debug\u00a0component, which helps with debugging. Shards Whitelist The nodes allowed in the\u00a0shards\u00a0parameter is configurable through the\u00a0shardsWhitelist\u00a0property in\u00a0solr.xml. This whitelist is automatically configured for SolrCloud but needs explicit configuration for leader\/follower mode. Limitations to Distributed Search Distributed searching in Solr has the following limitations: Each document indexed must have a unique key. If Solr discovers duplicate document IDs, Solr selects the first document and discards subsequent documents. The index for distributed searching may become momentarily out of sync if a commit happens between the first and second phase of the distributed search. This might cause a situation where a document that once matched a query and was subsequently changed may no longer match the query but will still be retrieved. This situation is expected to be quite rare, however, and is only possible for a single query request. The number of shards is limited by number of characters allowed for GET method\u2019s URI; most Web servers generally support at least 4000 characters, but many servers limit URI length to reduce their vulnerability to Denial of Service (DoS) attacks. Shard information can be returned with each document in a distributed search by including\u00a0fl=id, [shard]\u00a0in the search request. This returns the shard URL. In a distributed search, the data directory from the core descriptor overrides any data directory in\u00a0solrconfig.xml. Update commands may be sent to any server with distributed indexing configured correctly. Document adds and deletes are forwarded to the appropriate server\/shard based on a hash of the unique document id.\u00a0commit\u00a0commands and\u00a0deleteByQuery\u00a0commands are sent to every server in\u00a0shards. Formerly a limitation was that TF\/IDF relevancy computations only used shard-local statistics. This is still the case by default. If your data isn\u2019t randomly distributed, or if you want more exact statistics, then remember to configure the ExactStatsCache. Avoiding Distributed Deadlock with Distributed Search Like in SolrCloud mode, inter-shard requests could lead to a distributed deadlock. Testing Index Sharding on Two Local Servers For simple functional testing, it\u2019s easiest to just set up two local Solr servers on different ports. (In a production environment, of course, these servers would be deployed on separate machines.) Make two Solr home directories and copy\u00a0solr.xml\u00a0into the new directories: mkdir example\/nodes mkdir example\/nodes\/node1 # Copy solr.xml into this solr.home cp server\/solr\/solr.xml example\/nodes\/node1\/. # Repeat the above steps for the second node mkdir example\/nodes\/node2 cp server\/solr\/solr.xml example\/nodes\/node2\/. Start the two Solr instances # Start first node on port 8983 bin\/solr start -s example\/nodes\/node1 -p 8983 # Start second node on port 8984 bin\/solr start -s example\/nodes\/node2 -p 8984 Create a core on both the nodes with the sample_techproducts_configs. bin\/solr create_core -c core1 -p 8983 -d sample_techproducts_configs # Create a core on the Solr node running on port 8984 bin\/solr create_core -c core1 -p 8984 -d sample_techproducts_configs In a third window, index an example document to each of the server: bin\/post -c core1 example\/exampledocs\/monitor.xml -port 8983 bin\/post -c core1 example\/exampledocs\/monitor2.xml -port 8984 Search on the node on port 8983: curl http:\/\/localhost:8983\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should bring back one document. Search on the node on port 8984: curl http:\/\/localhost:8984\/solr\/core1\/select?q=*:*&wt=xml&indent=true This should also bring back a single document. Now do a distributed search across both servers with […]","og_url":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/","og_site_name":"Aeologic Blog","article_publisher":"https:\/\/www.facebook.com\/AeoLogicTech\/","article_published_time":"2021-04-19T08:33:53+00:00","article_modified_time":"2021-04-19T09:38:24+00:00","og_image":[{"width":1800,"height":1037,"url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png","type":"image\/png"}],"author":"Manoj Kumar","twitter_card":"summary_large_image","twitter_creator":"@aeologictech","twitter_site":"@aeologictech","twitter_misc":{"Written by":"Manoj Kumar","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#article","isPartOf":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/"},"author":{"name":"Manoj Kumar","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4"},"headline":"Distributed Search – Ultimate Solr Guide","datePublished":"2021-04-19T08:33:53+00:00","dateModified":"2021-04-19T09:38:24+00:00","mainEntityOfPage":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/"},"wordCount":972,"publisher":{"@id":"https:\/\/www.aeologic.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage"},"thumbnailUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png","keywords":["leucene","shard","solr"],"articleSection":["Solr"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/","url":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/","name":"Distributed Search - Ultimate Solr Guide - Aeologic Blog","isPartOf":{"@id":"https:\/\/www.aeologic.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage"},"thumbnailUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png","datePublished":"2021-04-19T08:33:53+00:00","dateModified":"2021-04-19T09:38:24+00:00","breadcrumb":{"@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#primaryimage","url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png","contentUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2021\/04\/Distributed-Search-with-Index-Sharding.png","width":1800,"height":1037},{"@type":"BreadcrumbList","@id":"https:\/\/www.aeologic.com\/blog\/distributed-search-with-index-sharding-ultimate-solr-guide\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.aeologic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Distributed Search – Ultimate Solr Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.aeologic.com\/blog\/#website","url":"https:\/\/www.aeologic.com\/blog\/","name":"Aeologic Blog","description":"Aeologic","publisher":{"@id":"https:\/\/www.aeologic.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.aeologic.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.aeologic.com\/blog\/#organization","name":"AeoLogic Technologies","url":"https:\/\/www.aeologic.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg","contentUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg","width":385,"height":162,"caption":"AeoLogic Technologies"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/AeoLogicTech\/","https:\/\/x.com\/aeologictech"]},{"@type":"Person","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4","name":"Manoj Kumar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g","caption":"Manoj Kumar"},"description":"Manoj Kumar is a seasoned Digital Marketing Manager and passionate Tech Blogger with deep expertise in SEO, AI trends, and emerging digital technologies. He writes about innovative solutions that drive growth and transformation across industry. Featured on - YOURSTORY | TECHSLING | ELEARNINGINDUSTRY | DATASCIENCECENTRAL | TIMESOFINDIA | MEDIUM | DATAFLOQ","sameAs":["https:\/\/www.aeologic.com\/","https:\/\/www.linkedin.com\/in\/manoj-kumar-rajput\/"],"url":"https:\/\/www.aeologic.com\/blog\/author\/manoj\/"}]}},"_links":{"self":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts\/1764","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/comments?post=1764"}],"version-history":[{"count":0,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts\/1764\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/media\/1767"}],"wp:attachment":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/media?parent=1764"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/categories?post=1764"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/tags?post=1764"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}