Crawler tags

Sitecore Search automatically generates default tags when you create a source. Each tag corresponds to an entity within your domain. You can use default tags to define attribute extraction rules in the document extractor.

Default tags suffice when:

For example, consider a film and television website where visitors can search for actors, TV shows, and movies. You want to create a search experience where visitors see only the attributes of each entity that a result belongs to. To do this, you can simply define extraction rules for the default tags in the document extractor.

Custom tags are any tags you create. Use custom tags when default tags are not enough. You can create as many custom tags as you want.

Usually, you'll need to configure custom tags when:

To continue with the example of the film and TV website, you might want to create a search experience where you show a list of movies an actor has acted in, along with general information like name and age. In this example, movie information is not available in any attribute of the Actor entity. However, the Movie entity has a movie_actor attribute. You can use this relationship to tie actors to a list of movies they acted in.

There are two ways to configure custom tags in Search: Basic tag configuration and Aggregated tag configuration.

A basic tag is a simple setup you use when you want to create more than one set of index documents from an entity. Usually, you define one or more tags and assign them to an entity.

For example, for a website, some blogs are written by internal writers, and some by external freelancers. Both types of blogs have the same attributes but require different attribute extraction logic. You want your visitors to have a unified search experience where they can search among all blogs. Your implementation has one entity, Blog.

Here's one way to handle this requirement: You can configure basic tags called internal blogs and external blogs under the Blog entity. When you extract attributes for the internal blogs tag, use extraction logic that suits the internal content. When you extract attributes for the external blogs tag, use extraction logic that suits the external content.

Here's a sample index document extracted for the internal tag:

 "id": xvsusdua232533ygd,
 "name": "Top 10 Most Anticipated Cars in 2024",
 "type": "blog",
 "description": "some description"

Here's a sample index document extracted for the external tag:

 "id": agahsfeev1nnbsa7d,
 "name": "How to Save 10% on Your New Car",
 "type": "blog",
 "description": "some other description"

As you can see, index documents extracted by both tags look identical. Attributes from both sets of index documents are from the Blog entity. The only difference was in their extraction logic. To a visitor, there is no difference between freelancer content and internal content.

An aggregated tag configuration is an advanced setup where you can create index document that have related attributes from more than one content item. You define mappings to aggregate related attributes. Then, you create a new attribute to combine and hold this aggregated information. Configuring tags through aggregation is a powerful way to enhance search experiences when dealing with relational data.

Use aggregated tags when you want to take advantage of a relationship between attributes in the same entity or different entities. An example of this relationship is when items have a shared attribute value, like name or id.

Depending on whether you want to define a relationship between attributes in the same entities or other entities, use flat aggregation or hierarchical aggregation.

Flat aggregation to combine attributes from different entities

Use flat aggregation when you want to enhance index documents with a new attribute that contains aggregated information from related items across multiple entities.

For example, you have a website that has information about actors, movies, and TV shows. There is a search bar where visitors can search for results. Your implementation has the Actor and Movie entities. The Actor entity has attributes like actor_name, actor_age, and so on. It does not have attributes about movies the actor has been in. The Movie entity has attributes like movie_name, movie_year, and a movie_cast attribute with a list of actors.

You want to create the following search experience: when an actor appears, also show a list of movies they acted in.

To support this experience, you'll need index documents with Actor attributes and an attribute that lists movies they have acted in.

To get these index documents, you can create a tag through flat aggregation under the Actor entity and associate it with the Movie entity. Then, configure this tag to use the actor's name as a baseline to find movies in which they've acted.

Here's a sample index document extracted for this tag:

{
  "actor_id": "A101",
  "actor_name": "Tom Cruise",
  "actor_description": "A critically acclaimed actor...",
  "actor_age": 61,
  "actor_gender": "male",
  "filmography": [
    {
      "movie_name": "Top Gun",
      "movie_year": 1986,
      "movie_genre": "Action"
    },
    {
      "movie_name": "Edge of Tomorrow",
      "movie_year": 2014,
      "movie_genre": "Science Fiction"
    },
    ...
  ]

Hierarchical aggregation to combine attributes of one entity

Use hierarchical aggregation when you want to enhance index documents with a new attribute that contains aggregated information from related items in one entity.

For example, you have a website where visitors can search for employees who work for your company. Your implementation has an entity called Employee, and one of the attributes in this entity is manager.

You want to create the following search experience: when an employee appears in the search results, also show a list of their colleagues.

To support this experience, you'll need index documents that have regular Employee attributes as well as an attribute that lists colleagues.

To get these index documents, you can create a tag under the Employee entity. Then, configure this tag to use the manager attribute as a baseline to find employees with the same manager, that is, to find colleagues.

Here's a sample index document extracted for this tag:

{
  "employee_id": 1,
  "name": "John Doe",
  "position": "Senior developer",
  "department": "R&D",
  "manager": "Mary Smith",
  "colleagues": [ 
    {
      "employee_id": 2,
      "name": "Jane Smith",
      "position": "Lead Developer",
    },
    {
      "employee_id": 3,
      "name": "David Johnson",
      "position": "Senior  Developer",
    }
  ]
}

Consider a film and television website where visitors can search for actors, TV shows, and movies. Your implementation has the Actor, Movie , and TVShow entities.

The Actor entity has the following attributes: actor_awards, actor_age, actor_gender, actor_movies, actor_name, actor_image, actor_tv, actor_bio.

The Movie entity has the following attributes: movie_description, movie_awards, movie_actor, movie_name, movie_rating, movie_studio, movie_director, movie_poster.

The TV Show entity has the following attributes: tv_name, tv_cast, tv_rating, tv_poster, tv_episodes, tv_seasons.

You want to create the following search experiences:

The following image shows a sample source and tag configuration that you can use to get the desired search experiences:

Here's an explanation of this image: