{"id":814,"date":"2020-04-18T05:44:18","date_gmt":"2020-04-18T05:44:18","guid":{"rendered":"https:\/\/www.aeologic.com\/blog\/?p=814"},"modified":"2020-04-18T05:44:18","modified_gmt":"2020-04-18T05:44:18","slug":"indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide","status":"publish","type":"post","link":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/","title":{"rendered":"Indexing and Searching Nested Documents in Solr – Ultimate Solr Guide"},"content":{"rendered":"

Hi All, Today I will be writing about another important aspect of Solr that encapsulates the need of a lot of business applications. Suppose we need to index data of an e-commerce company that wants to host a variety of products on their websites along with associated SKU’s. This data needs to be indexed, updated, searched upon in a way that is legible to both the end-users and the merchandisers. So, a natural question arises, how to represent this data? Also, what matrix to follow in order to maintain the data easily and effectively?<\/p>\n

To address this problem, Solr provides us with a mechanism to associate multiple docs with each other and put them in a relationship. Also, it provides both indexing and searching mechanisms to get data in and out. What we are referring to here is called a “Nested document structure”, or more technically, a BLOCK in Solr. Put simply, a block is a set of parent-child relation between multiple documents.<\/p>\n

Let’s understand this in two parts.<\/p>\n

Indexing Nested Documents<\/strong><\/h1>\n

Let’s assume a nested structure as below:<\/p>\n

{
\n“id”:”100″,
\n“scope_ss”:”product”,
\n“category_ss”:”OutdoorSofa”,
\n“productType_ss”:”sofa”,
\n“_childDocuments_<\/strong>“: [
\n{
\n“id”:”101″,
\n“scope_ss”:”sku”,
\n“Fabric_ss”:”Polyester”,
\n“Depth_ss”:”Classic”,
\n“Finish_ss” : “Fine”,
\n“Color_ss”:”Grey”},
\n{
\n“id”:”102″,
\n“scope_ss”:”sku”,
\n“Fabric_ss”:”nylon”,
\n“Depth_ss”:”Petite”,
\n“Finish_ss” : “Grey”,
\n“Color_ss”:”Green”},
\n{
\n“id”:”103″,
\n“scope_ss”:”sku”,
\n“Fabric_ss”:”acrylic”,
\n“Depth_ss”:”Petite”,
\n“Finish_ss” : “Espresso”,
\n“Color_ss”:”Green”},
\n{
\n“id”:”104″,
\n“scope_ss”:”sku”,
\n“Fabric_ss”:”Polyester”,
\n“Depth_ss”: “Classic”,
\n“Finish_ss” : “Grey”,
\n“Color_ss”:”Dove”},<\/p>\n

{“id”:”105″,
\n“scope_ss”:”sku”,
\n“Fabric_ss”:”nylon”,
\n“Depth_ss”: “Classic”,
\n“Finish_ss” : “Grey”,
\n“Color_ss”:”Blue”}]<\/p>\n

}<\/p>\n

_childDocuments_\u00a0<\/strong>tells solr that each object defined within the scope is a child to the doc it belongs to. In essence, we have just created a block of information that has properties for SKU’s which imparts some meaning to the application scenarios. Now let’s see how we can index the doc. Utilizing Solr’s dynamic schema mapping capabilities, we can be certain that all the fields in the document will get indexed and get reflected (Notice _ss <\/strong>in fields, they are stored as String array). Now, one may ask upon careful observation that why the data type used is array when the field is storing String type information. Answer to this question lies in more subtle understanding of inner working of Solr.<\/p>\n

When Solr indexes a nested document, it stores all field values for a field in a set. This effectively means each field has multiple values and hence the type, array of String. If one tries to index data without using _ss or just _s (String type in Solr), solr engine throws an error. Now, assuming we have settled with basics, let’s continue to index the data.<\/p>\n

Fire up UI and index data by navigating to necessary modalities as below:<\/p>\n

\"\"<\/p>\n

Paste the JSON doc in “Documents(s)” section and index the document. If all configuration is proper, then, the index upon querying will appear as below:<\/p>\n

\"\"<\/p>\n

Note:<\/strong> Notice how the parent doc (the product information) is referenced at bottom and child docs are piled upon it. This is not by accident but by design.<\/p>\n

Querying Nested Documents<\/strong><\/h1>\n

Let’s dive into basics of querying nested docs in Solr.<\/p>\n

One of the most needed use-case in context of e-commerce data-set is Faceting. Faceting for nested docs by design is different than traditional docs in Solr.<\/p>\n

    \n
  • To understand this, let’s generate facets for Color_ss and Fabric_ss fields.\n
      \n
    • Query Syntax<\/strong>: &json.facet={Fabrics:{type:terms,field:Fabric_ss<\/strong>,limit:-1},Colors:{type:terms,field:Color_ss<\/strong>,limit:-1}}&rows=0<\/li>\n
    • Response<\/strong>:\"\"<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

      If one observes carefully, one can see clearly how two buckets for each of the fields is formed. First bucket for Fabrics and second bucket for colors. Its worth mentioning here that the name, “Fabrics” and “Colors” are custom headers and can be changed at will.<\/p>\n

        \n
      • Lets try something else. Let’s try nested faceting here.\n
          \n
        • Query<\/strong>: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0<\/li>\n
        • Response<\/strong>:\"\"<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

          Here, we have a bucket called “Fabrics” and nested within it is another bucket called “Colors”. In essence, we are presented with more granular information telling the color of each fabric type.<\/p>\n

          Let’s try another interesting query, one combining nested faceting with independent buckets.<\/p>\n

            \n
          • Query<\/strong>: &json.facet={Depth:{type:terms,field:Depth_ss,limit:-1},Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0<\/li>\n
          • Response:<\/strong>\u00a0\"\"<\/li>\n<\/ul>\n

            Here, as one observes carefully, one bucket stands for independent Depth and other stands for nested facet for Fabric with Color. This enables businesses using Solr to present product information at a granular level for their customers.<\/p>\n

            Some other useful queries include:<\/p>\n

              \n
            • To view all products and SKU information:\n
                \n
              • Query:<\/strong> q=*:*&rows=10<\/li>\n
              • Response:<\/strong>\n
                  \n
                • \"\"<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n
                    \n
                  • To fetch specific products and its associated SKUs:\n
                      \n
                    • Query<\/strong>: q={!child%20of=”scope_ss:product”}id:100&wt=json<\/li>\n
                    • Response:<\/strong><\/li>\n
                    • \"\"<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n

                      {!child%20of=”scope_ss:product”}<\/strong> – This filter fetches information for all parent products by using the unique field-value pair used to identify a parent doc in index.<\/p>\n

                      id:100 – <\/strong>This filter fetches “some” parents based on a specific field-value pair for a parent doc.<\/p>\n

                      Updating a nested document in Solr<\/h1>\n

                      Updating a nested document in Solr involves is a very tricky business. One has to update a complete BLOCK if a single SKU has to be updated or a new SKU has to be added. (THIS IS THE ONLY LIMITATION OF SOLR<\/strong>). This essentially happens because of underlying Lucene handlers. Exact explanation for this, though present, is beyond the scope of this post.<\/p>\n

                      Deleting a SKU\/child document<\/h1>\n

                      The solr document is a little ambiguous about it though based on a simple POC, its clear that one can remove a child doc from solr by using a simple delete by ID API. One can invoke update handler for core\/collection as below:<\/p>\n

                      Query:<\/strong> update?stream.body=<delete><query>id:101<<\/strong>\/query><\/delete>&commit=true<\/p>\n

                      Response:\u00a0<\/strong> This removes the child document with id:101 from index<\/p>\n

                      So, that’s it for today. Will be back with another interesting post on Solr.<\/p>\n","protected":false},"excerpt":{"rendered":"

                      Hi All, Today I will be writing about another important aspect of Solr that encapsulates the need of a lot of business applications. Suppose we need to index data of an e-commerce company that wants to host a variety of products on their websites along with associated SKU’s. This data needs to be indexed, updated, searched upon in a way that is legible to both the end-users and the merchandisers. So, a natural question arises, how to represent this data? Also, what matrix to follow in order to maintain the data easily and effectively? To address this problem, Solr provides us with a mechanism to associate multiple docs with each other and put them in a relationship. Also, it provides both indexing and searching mechanisms to get data in and out. What we are referring to here is called a “Nested document structure”, or more technically, a BLOCK in Solr. Put simply, a block is a set of parent-child relation between multiple documents. Let’s understand this in two parts. Indexing Nested Documents Let’s assume a nested structure as below: { “id”:”100″, “scope_ss”:”product”, “category_ss”:”OutdoorSofa”, “productType_ss”:”sofa”, “_childDocuments_“: [ { “id”:”101″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”:”Classic”, “Finish_ss” : “Fine”, “Color_ss”:”Grey”}, { “id”:”102″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”:”Petite”, “Finish_ss” : “Grey”, “Color_ss”:”Green”}, { “id”:”103″, “scope_ss”:”sku”, “Fabric_ss”:”acrylic”, “Depth_ss”:”Petite”, “Finish_ss” : “Espresso”, “Color_ss”:”Green”}, { “id”:”104″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Dove”}, {“id”:”105″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Blue”}] } _childDocuments_\u00a0tells solr that each object defined within the scope is a child to the doc it belongs to. In essence, we have just created a block of information that has properties for SKU’s which imparts some meaning to the application scenarios. Now let’s see how we can index the doc. Utilizing Solr’s dynamic schema mapping capabilities, we can be certain that all the fields in the document will get indexed and get reflected (Notice _ss in fields, they are stored as String array). Now, one may ask upon careful observation that why the data type used is array when the field is storing String type information. Answer to this question lies in more subtle understanding of inner working of Solr. When Solr indexes a nested document, it stores all field values for a field in a set. This effectively means each field has multiple values and hence the type, array of String. If one tries to index data without using _ss or just _s (String type in Solr), solr engine throws an error. Now, assuming we have settled with basics, let’s continue to index the data. Fire up UI and index data by navigating to necessary modalities as below: Paste the JSON doc in “Documents(s)” section and index the document. If all configuration is proper, then, the index upon querying will appear as below: Note: Notice how the parent doc (the product information) is referenced at bottom and child docs are piled upon it. This is not by accident but by design. Querying Nested Documents Let’s dive into basics of querying nested docs in Solr. One of the most needed use-case in context of e-commerce data-set is Faceting. Faceting for nested docs by design is different than traditional docs in Solr. To understand this, let’s generate facets for Color_ss and Fabric_ss fields. Query Syntax: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1},Colors:{type:terms,field:Color_ss,limit:-1}}&rows=0 Response: If one observes carefully, one can see clearly how two buckets for each of the fields is formed. First bucket for Fabrics and second bucket for colors. Its worth mentioning here that the name, “Fabrics” and “Colors” are custom headers and can be changed at will. Lets try something else. Let’s try nested faceting here. Query: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response: Here, we have a bucket called “Fabrics” and nested within it is another bucket called “Colors”. In essence, we are presented with more granular information telling the color of each fabric type. Let’s try another interesting query, one combining nested faceting with independent buckets. Query: &json.facet={Depth:{type:terms,field:Depth_ss,limit:-1},Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response:\u00a0 Here, as one observes carefully, one bucket stands for independent Depth and other stands for nested facet for Fabric with Color. This enables businesses using Solr to present product information at a granular level for their customers. Some other useful queries include: To view all products and SKU information: Query: q=*:*&rows=10 Response: To fetch specific products and its associated SKUs: Query: q={!child%20of=”scope_ss:product”}id:100&wt=json Response: {!child%20of=”scope_ss:product”} – This filter fetches information for all parent products by using the unique field-value pair used to identify a parent doc in index. id:100 – This filter fetches “some” parents based on a specific field-value pair for a parent doc. Updating a nested document in Solr Updating a nested document in Solr involves is a very tricky business. One has to update a complete BLOCK if a single SKU has to be updated or a new SKU has to be added. (THIS IS THE ONLY LIMITATION OF SOLR). This essentially happens because of underlying Lucene handlers. Exact explanation for this, though present, is beyond the scope of this post. Deleting a SKU\/child document The solr document is a little ambiguous about it though based on a simple POC, its clear that one can remove a child doc from solr by using a simple delete by ID API. One can invoke update handler for core\/collection as below: Query: update?stream.body=<delete><query>id:101<\/query><\/delete>&commit=true Response:\u00a0 This removes the child document with id:101 from index So, that’s it for today. Will be back with another interesting post on Solr.<\/p>\n","protected":false},"author":3,"featured_media":839,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[41],"tags":[105,107,91,106],"class_list":["post-814","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-solr","tag-nested-doc-solr","tag-nested-structure-solr","tag-solr-index","tag-solr-nested-documents"],"yoast_head":"\nIndexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Indexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog\" \/>\n<meta property=\"og:description\" content=\"Hi All, Today I will be writing about another important aspect of Solr that encapsulates the need of a lot of business applications. Suppose we need to index data of an e-commerce company that wants to host a variety of products on their websites along with associated SKU’s. This data needs to be indexed, updated, searched upon in a way that is legible to both the end-users and the merchandisers. So, a natural question arises, how to represent this data? Also, what matrix to follow in order to maintain the data easily and effectively? To address this problem, Solr provides us with a mechanism to associate multiple docs with each other and put them in a relationship. Also, it provides both indexing and searching mechanisms to get data in and out. What we are referring to here is called a “Nested document structure”, or more technically, a BLOCK in Solr. Put simply, a block is a set of parent-child relation between multiple documents. Let’s understand this in two parts. Indexing Nested Documents Let’s assume a nested structure as below: { “id”:”100″, “scope_ss”:”product”, “category_ss”:”OutdoorSofa”, “productType_ss”:”sofa”, “_childDocuments_“: [ { “id”:”101″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”:”Classic”, “Finish_ss” : “Fine”, “Color_ss”:”Grey”}, { “id”:”102″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”:”Petite”, “Finish_ss” : “Grey”, “Color_ss”:”Green”}, { “id”:”103″, “scope_ss”:”sku”, “Fabric_ss”:”acrylic”, “Depth_ss”:”Petite”, “Finish_ss” : “Espresso”, “Color_ss”:”Green”}, { “id”:”104″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Dove”}, {“id”:”105″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Blue”}] } _childDocuments_\u00a0tells solr that each object defined within the scope is a child to the doc it belongs to. In essence, we have just created a block of information that has properties for SKU’s which imparts some meaning to the application scenarios. Now let’s see how we can index the doc. Utilizing Solr’s dynamic schema mapping capabilities, we can be certain that all the fields in the document will get indexed and get reflected (Notice _ss in fields, they are stored as String array). Now, one may ask upon careful observation that why the data type used is array when the field is storing String type information. Answer to this question lies in more subtle understanding of inner working of Solr. When Solr indexes a nested document, it stores all field values for a field in a set. This effectively means each field has multiple values and hence the type, array of String. If one tries to index data without using _ss or just _s (String type in Solr), solr engine throws an error. Now, assuming we have settled with basics, let’s continue to index the data. Fire up UI and index data by navigating to necessary modalities as below: Paste the JSON doc in “Documents(s)” section and index the document. If all configuration is proper, then, the index upon querying will appear as below: Note: Notice how the parent doc (the product information) is referenced at bottom and child docs are piled upon it. This is not by accident but by design. Querying Nested Documents Let’s dive into basics of querying nested docs in Solr. One of the most needed use-case in context of e-commerce data-set is Faceting. Faceting for nested docs by design is different than traditional docs in Solr. To understand this, let’s generate facets for Color_ss and Fabric_ss fields. Query Syntax: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1},Colors:{type:terms,field:Color_ss,limit:-1}}&rows=0 Response: If one observes carefully, one can see clearly how two buckets for each of the fields is formed. First bucket for Fabrics and second bucket for colors. Its worth mentioning here that the name, “Fabrics” and “Colors” are custom headers and can be changed at will. Lets try something else. Let’s try nested faceting here. Query: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response: Here, we have a bucket called “Fabrics” and nested within it is another bucket called “Colors”. In essence, we are presented with more granular information telling the color of each fabric type. Let’s try another interesting query, one combining nested faceting with independent buckets. Query: &json.facet={Depth:{type:terms,field:Depth_ss,limit:-1},Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response:\u00a0 Here, as one observes carefully, one bucket stands for independent Depth and other stands for nested facet for Fabric with Color. This enables businesses using Solr to present product information at a granular level for their customers. Some other useful queries include: To view all products and SKU information: Query: q=*:*&rows=10 Response: To fetch specific products and its associated SKUs: Query: q={!child%20of=”scope_ss:product”}id:100&wt=json Response: {!child%20of=”scope_ss:product”} – This filter fetches information for all parent products by using the unique field-value pair used to identify a parent doc in index. id:100 – This filter fetches “some” parents based on a specific field-value pair for a parent doc. Updating a nested document in Solr Updating a nested document in Solr involves is a very tricky business. One has to update a complete BLOCK if a single SKU has to be updated or a new SKU has to be added. (THIS IS THE ONLY LIMITATION OF SOLR). This essentially happens because of underlying Lucene handlers. Exact explanation for this, though present, is beyond the scope of this post. Deleting a SKU\/child document The solr document is a little ambiguous about it though based on a simple POC, its clear that one can remove a child doc from solr by using a simple delete by ID API. One can invoke update handler for core\/collection as below: Query: update?stream.body=<delete><query>id:101<\/query><\/delete>&commit=true Response:\u00a0 This removes the child document with id:101 from index So, that’s it for today. Will be back with another interesting post on Solr.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\" \/>\n<meta property=\"og:site_name\" content=\"Aeologic Blog\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AeoLogicTech\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-04-18T05:44:18+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1080\" \/>\n\t<meta property=\"og:image:height\" content=\"622\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Manoj Kumar\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@aeologictech\" \/>\n<meta name=\"twitter:site\" content=\"@aeologictech\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Manoj Kumar\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":[\"Article\",\"BlogPosting\"],\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\"},\"author\":{\"name\":\"Manoj Kumar\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4\"},\"headline\":\"Indexing and Searching Nested Documents in Solr – Ultimate Solr Guide\",\"datePublished\":\"2020-04-18T05:44:18+00:00\",\"dateModified\":\"2020-04-18T05:44:18+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\"},\"wordCount\":1059,\"publisher\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png\",\"keywords\":[\"Nested doc solr\",\"Nested structure solr\",\"solr index\",\"Solr nested documents\"],\"articleSection\":[\"Solr\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\",\"url\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\",\"name\":\"Indexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png\",\"datePublished\":\"2020-04-18T05:44:18+00:00\",\"dateModified\":\"2020-04-18T05:44:18+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage\",\"url\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png\",\"contentUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png\",\"width\":1080,\"height\":622},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.aeologic.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Indexing and Searching Nested Documents in Solr – Ultimate Solr Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#website\",\"url\":\"https:\/\/www.aeologic.com\/blog\/\",\"name\":\"Aeologic Blog\",\"description\":\"Aeologic\",\"publisher\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.aeologic.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#organization\",\"name\":\"AeoLogic Technologies\",\"url\":\"https:\/\/www.aeologic.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg\",\"contentUrl\":\"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg\",\"width\":385,\"height\":162,\"caption\":\"AeoLogic Technologies\"},\"image\":{\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/AeoLogicTech\/\",\"https:\/\/x.com\/aeologictech\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4\",\"name\":\"Manoj Kumar\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g\",\"caption\":\"Manoj Kumar\"},\"description\":\"Manoj Kumar is a seasoned Digital Marketing Manager and passionate Tech Blogger with deep expertise in SEO, AI trends, and emerging digital technologies. He writes about innovative solutions that drive growth and transformation across industry. Featured on - YOURSTORY | TECHSLING | ELEARNINGINDUSTRY | DATASCIENCECENTRAL | TIMESOFINDIA | MEDIUM | DATAFLOQ\",\"sameAs\":[\"https:\/\/www.aeologic.com\/\",\"https:\/\/www.linkedin.com\/in\/manoj-kumar-rajput\/\"],\"url\":\"https:\/\/www.aeologic.com\/blog\/author\/manoj\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Indexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/","og_locale":"en_US","og_type":"article","og_title":"Indexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog","og_description":"Hi All, Today I will be writing about another important aspect of Solr that encapsulates the need of a lot of business applications. Suppose we need to index data of an e-commerce company that wants to host a variety of products on their websites along with associated SKU’s. This data needs to be indexed, updated, searched upon in a way that is legible to both the end-users and the merchandisers. So, a natural question arises, how to represent this data? Also, what matrix to follow in order to maintain the data easily and effectively? To address this problem, Solr provides us with a mechanism to associate multiple docs with each other and put them in a relationship. Also, it provides both indexing and searching mechanisms to get data in and out. What we are referring to here is called a “Nested document structure”, or more technically, a BLOCK in Solr. Put simply, a block is a set of parent-child relation between multiple documents. Let’s understand this in two parts. Indexing Nested Documents Let’s assume a nested structure as below: { “id”:”100″, “scope_ss”:”product”, “category_ss”:”OutdoorSofa”, “productType_ss”:”sofa”, “_childDocuments_“: [ { “id”:”101″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”:”Classic”, “Finish_ss” : “Fine”, “Color_ss”:”Grey”}, { “id”:”102″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”:”Petite”, “Finish_ss” : “Grey”, “Color_ss”:”Green”}, { “id”:”103″, “scope_ss”:”sku”, “Fabric_ss”:”acrylic”, “Depth_ss”:”Petite”, “Finish_ss” : “Espresso”, “Color_ss”:”Green”}, { “id”:”104″, “scope_ss”:”sku”, “Fabric_ss”:”Polyester”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Dove”}, {“id”:”105″, “scope_ss”:”sku”, “Fabric_ss”:”nylon”, “Depth_ss”: “Classic”, “Finish_ss” : “Grey”, “Color_ss”:”Blue”}] } _childDocuments_\u00a0tells solr that each object defined within the scope is a child to the doc it belongs to. In essence, we have just created a block of information that has properties for SKU’s which imparts some meaning to the application scenarios. Now let’s see how we can index the doc. Utilizing Solr’s dynamic schema mapping capabilities, we can be certain that all the fields in the document will get indexed and get reflected (Notice _ss in fields, they are stored as String array). Now, one may ask upon careful observation that why the data type used is array when the field is storing String type information. Answer to this question lies in more subtle understanding of inner working of Solr. When Solr indexes a nested document, it stores all field values for a field in a set. This effectively means each field has multiple values and hence the type, array of String. If one tries to index data without using _ss or just _s (String type in Solr), solr engine throws an error. Now, assuming we have settled with basics, let’s continue to index the data. Fire up UI and index data by navigating to necessary modalities as below: Paste the JSON doc in “Documents(s)” section and index the document. If all configuration is proper, then, the index upon querying will appear as below: Note: Notice how the parent doc (the product information) is referenced at bottom and child docs are piled upon it. This is not by accident but by design. Querying Nested Documents Let’s dive into basics of querying nested docs in Solr. One of the most needed use-case in context of e-commerce data-set is Faceting. Faceting for nested docs by design is different than traditional docs in Solr. To understand this, let’s generate facets for Color_ss and Fabric_ss fields. Query Syntax: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1},Colors:{type:terms,field:Color_ss,limit:-1}}&rows=0 Response: If one observes carefully, one can see clearly how two buckets for each of the fields is formed. First bucket for Fabrics and second bucket for colors. Its worth mentioning here that the name, “Fabrics” and “Colors” are custom headers and can be changed at will. Lets try something else. Let’s try nested faceting here. Query: &json.facet={Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response: Here, we have a bucket called “Fabrics” and nested within it is another bucket called “Colors”. In essence, we are presented with more granular information telling the color of each fabric type. Let’s try another interesting query, one combining nested faceting with independent buckets. Query: &json.facet={Depth:{type:terms,field:Depth_ss,limit:-1},Fabrics:{type:terms,field:Fabric_ss,limit:-1,facet:{Colors:{type:terms,field:Color_ss,limit:-1}}}}&rows=0 Response:\u00a0 Here, as one observes carefully, one bucket stands for independent Depth and other stands for nested facet for Fabric with Color. This enables businesses using Solr to present product information at a granular level for their customers. Some other useful queries include: To view all products and SKU information: Query: q=*:*&rows=10 Response: To fetch specific products and its associated SKUs: Query: q={!child%20of=”scope_ss:product”}id:100&wt=json Response: {!child%20of=”scope_ss:product”} – This filter fetches information for all parent products by using the unique field-value pair used to identify a parent doc in index. id:100 – This filter fetches “some” parents based on a specific field-value pair for a parent doc. Updating a nested document in Solr Updating a nested document in Solr involves is a very tricky business. One has to update a complete BLOCK if a single SKU has to be updated or a new SKU has to be added. (THIS IS THE ONLY LIMITATION OF SOLR). This essentially happens because of underlying Lucene handlers. Exact explanation for this, though present, is beyond the scope of this post. Deleting a SKU\/child document The solr document is a little ambiguous about it though based on a simple POC, its clear that one can remove a child doc from solr by using a simple delete by ID API. One can invoke update handler for core\/collection as below: Query: update?stream.body=<delete><query>id:101<\/query><\/delete>&commit=true Response:\u00a0 This removes the child document with id:101 from index So, that’s it for today. Will be back with another interesting post on Solr.","og_url":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/","og_site_name":"Aeologic Blog","article_publisher":"https:\/\/www.facebook.com\/AeoLogicTech\/","article_published_time":"2020-04-18T05:44:18+00:00","og_image":[{"width":1080,"height":622,"url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png","type":"image\/png"}],"author":"Manoj Kumar","twitter_card":"summary_large_image","twitter_creator":"@aeologictech","twitter_site":"@aeologictech","twitter_misc":{"Written by":"Manoj Kumar","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":["Article","BlogPosting"],"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#article","isPartOf":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/"},"author":{"name":"Manoj Kumar","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4"},"headline":"Indexing and Searching Nested Documents in Solr – Ultimate Solr Guide","datePublished":"2020-04-18T05:44:18+00:00","dateModified":"2020-04-18T05:44:18+00:00","mainEntityOfPage":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/"},"wordCount":1059,"publisher":{"@id":"https:\/\/www.aeologic.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage"},"thumbnailUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png","keywords":["Nested doc solr","Nested structure solr","solr index","Solr nested documents"],"articleSection":["Solr"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/","url":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/","name":"Indexing and Searching Nested Documents in Solr - Ultimate Solr Guide - Aeologic Blog","isPartOf":{"@id":"https:\/\/www.aeologic.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage"},"thumbnailUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png","datePublished":"2020-04-18T05:44:18+00:00","dateModified":"2020-04-18T05:44:18+00:00","breadcrumb":{"@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#primaryimage","url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png","contentUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2020\/04\/Indexing-and-Searching-Nested-Documents-in-Solr.png","width":1080,"height":622},{"@type":"BreadcrumbList","@id":"https:\/\/www.aeologic.com\/blog\/indexing-and-searching-nested-documents-in-solr-ultimate-solr-guide\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.aeologic.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Indexing and Searching Nested Documents in Solr – Ultimate Solr Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.aeologic.com\/blog\/#website","url":"https:\/\/www.aeologic.com\/blog\/","name":"Aeologic Blog","description":"Aeologic","publisher":{"@id":"https:\/\/www.aeologic.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.aeologic.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.aeologic.com\/blog\/#organization","name":"AeoLogic Technologies","url":"https:\/\/www.aeologic.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg","contentUrl":"https:\/\/www.aeologic.com\/blog\/wp-content\/uploads\/2022\/05\/new-logo-aeo.jpg","width":385,"height":162,"caption":"AeoLogic Technologies"},"image":{"@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/AeoLogicTech\/","https:\/\/x.com\/aeologictech"]},{"@type":"Person","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/13549984ba8e5f441cc733ed20d7daa4","name":"Manoj Kumar","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.aeologic.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/24ce77602da5eb5715d74a95733f6c7548e2af73f5a493f9bc0bf55f611d025e?s=96&d=mm&r=g","caption":"Manoj Kumar"},"description":"Manoj Kumar is a seasoned Digital Marketing Manager and passionate Tech Blogger with deep expertise in SEO, AI trends, and emerging digital technologies. He writes about innovative solutions that drive growth and transformation across industry. Featured on - YOURSTORY | TECHSLING | ELEARNINGINDUSTRY | DATASCIENCECENTRAL | TIMESOFINDIA | MEDIUM | DATAFLOQ","sameAs":["https:\/\/www.aeologic.com\/","https:\/\/www.linkedin.com\/in\/manoj-kumar-rajput\/"],"url":"https:\/\/www.aeologic.com\/blog\/author\/manoj\/"}]}},"_links":{"self":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts\/814","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/comments?post=814"}],"version-history":[{"count":0,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/posts\/814\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/media\/839"}],"wp:attachment":[{"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/media?parent=814"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/categories?post=814"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aeologic.com\/blog\/wp-json\/wp\/v2\/tags?post=814"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}