Configure document extractors

Sitecore Search has to be able to extract all mandatory attributes before it creates an index document out of a URL or document. If it cannot extract the value of any mandatory attribute, it does not index that URL or document.

There are two types of mandatory attributes: attributes that are mandatory for all domains and attributes that are mandatory only for your domain.

All domains have the following mandatory attributes:

In addition to these attributes, your domain might have other attributes configured as mandatory. You'll need to configure how to extract them, too.

To ensure that the document extractor only extracts pages or documents whose URL matches a defined pattern, configure URLs to Match.

You can use a regular expression, a Glob expression, or a JavaScript function to define a URL pattern.

A common use case for this setting is to create different extractors for different areas of your source content. For example, you can create a document extractor for your HTML pages and a document extractor for your PDF content.

In case of redirects, configure URLs to Match to include the redirected URLs.

If you want an attribute to have a constant value across all documents indexed by a source, you can specify a fixed value for this attribute. This means that all index documents have the same predefined value for this attribute. The crawler does not extract this attribute value from the page meta data.

This method is useful when you need an attribute to have the same value across all index documents, but the attribute value is either not available in your content meta data, is available on only some pages, or is has different values on different pages.

A common use case is to specify a fixed value for the type attribute. For example, you have two sources that index content from a banking website. One source indexes content from the commercial banking section, and the other source indexes content from the consumer banking section. One way to handle the type attribute is to specify a fixed value of commercial in the first source and consumer in the second source.

To use an HTML meta tag or an OG tag to extract attribute values, all you have to do is give Search the name of the meta tag or OG tag whose content you want to use as the attribute value. Then, Search internally constructs an XPath expression that evaluates to the attribute value.

We recommend that you use this method when attribute values are in your page header, and Search can access them using a simple XPath expression.

When you enter an HTML meta tag or OG tag, you do not see the XPath expression Search creates as it is done internally. However, it is helpful to know the expression that Search creates so that you can evaluate whether this attribute extraction works for you.

The following sections explain how Search constructs XPath expressions from the meta tag or OG tag that you enter.

Example using an HTML meta tag

First, Search constructs an XPath expression that assumes you have entered a meta tag name. If that expression does not yield any result, Search constructs an XPath expression that assumes you have entered a property name.

For example, if you enter author, Search first constructs this XPath expression: //meta[@name='author']/@content. If this expression gives a value, Search stops. If this expression does not give a value, Search then constructs this XPath expression: //meta[@property='author']/@content.

Example using an OG tag

Assume you enter og:site_name. Because this phrase has 'OG' in it, Search assumes that you entered an OG tag and constructs the following XPath expression:

//meta[@property='og:site_name']/@content

Use an XPath document extractor when you want to use an XPath expression that evaluates to an attribute value.

Use the following settings to extract attributes using XPath:

To extract attributes using JavaScript, add a function that defines how the crawler should get values for each attribute. Usually, you use a JavaScript document in scenarios where an XPath expression is not enough to extract an attribute's value.

For example, if you want to replace parts of a URL that you get from the page metadata before storing it as an attribute, you must use a JS extractor. You can also use a JavaScript function to create composite attributes, that is, create a new attribute by combining values from more than one existing attribute.

To help with configuration, Search adds the following sample JavaScript function that gets values for the description, name, type, and URL attributes from meta tags:

function extract(request, response) {
    $ = response.body;

    return [{
        'description': $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || $('p').text(),
        'name': $('meta[name="searchtitle"]').attr('content') || $('title').text(),
        'type': $('meta[property="og:type"]').attr('content') || 'website_content',
        'url': $('meta[property="og:url"]').attr('content')
    }];
}

You can edit this function or paste a new function.

See the topic on how to configure a JavaScript extractor for a more complex JavaScript function.

Use a JSONPath document extractor to extract attribute values from JSON data.

Use the following settings to extract attributes using JSONPath:

..placeholders['headless-main']..fields.Description.value

..placeholders['headless-main']..fields.Text.value

Use a CSS document extractor when you want to use CSS queries to extract attribute values. Unlike HTML tags that you can extract with XPath, CSS tags do not appear in the page source. For example, titles and images are sometimes within CSS tags.

Use the following settings to use a CSS expression to extract attributes: