Walkthrough: Configuring an advanced web crawler

The Sitecore Search advanced web crawler is a powerful crawler that crawls your content and adds it to an index. It can handle complicated use cases like needing authentication to access your source content, creating index documents in multiple languages, using Java Script to extract attribute values, and more.

This walkthrough describes how to:

Create an advanced web crawler source
Configure advanced web crawler settings
Configure a request trigger
Create an XPath document extractor
Schedule scans
Publish the source

Create an advanced web crawler source

Before you configure a source, you must create one.

To create a source:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click Add Source.
In the SOURCE NAME field, enter a name for the source. For example, enter Doc site web crawler.
In the DESCRIPTION field, enter a few lines to describe the source you want to configure.

For example, enter Web crawler to crawl all pages of the doc site.
In the CONNECTOR field, click the type of source you want to create.
Click Save.

If there are no errors, Search creates a new source.

After you create a source, configure the source to access and index content.

In this example, select Web Crawler (Advanced) in step 5.

Configure advanced web crawler settings

Configure crawler settings to define important high-level configurations that define the scope of the API crawler.

To configure the scope of the advanced web crawler:

Go to Sources and then click the source you created.
On the Source Settings page, click Edit ( )next to Advanced Web Crawler Settings.
Optionally, in the ALLOWED DOMAINS field, enter the domains for the crawler to stay within. Do this to ensure that the web crawler only crawls Sitecore documentation domains, not external sites that might be linked.

For example, enter www.doc.sitecore.com. Do this to ensure that the advanced web crawler only crawls Sitecore documentation domains, not external sites that might be linked.
To configure the depth and number of URLs for the crawler to crawl:
- In the MAX DEPTH field, enter the maximum number of levels the crawler should follow for a URL.
  
  For example, enter 5.
- In the MAX URLS field, enter the maximum number of URLs for the crawler to crawl, in total. Enter a large number to ensure that the crawler leaves no URL out.
  
  For example, enter 5000
To exclude certain URL patterns from the crawler's scope, click Add Exclusion Pattern. Then, in the TYPE drop-down many, select Glob expression or Regular Expression. In the VALUE field, enter the expression to match URLs to exclude.

For example, to prevent the crawler from crawling your search page, enter the following Glob expression:
RequestResponseshell
```
**/search/**
```
In the PARALLELISM (WORKERS) field, enter the number of threads, or workers, that concurrently crawl and index content.

For example, enter 2 to have only two workers, or threads, crawl in parallel. This uses less memory than the default 5 workers.
In the TIMEOUT field, enter the time, in milliseconds, that the crawler waits to get a response.

For example, enter 5000. This ensures that the crawler waits for 5000 milliseconds, or 5 seconds, to get a response from every URL it crawls.
Optionally, in the HEADERS section, click Add Header. Then, in the Key field, enter the name of the user agent that your content expects. In the Value field, enter the value of the user agent that your content expects. This security measure ensures that only the Search crawler, and not any other crawler, can crawl the data.

For example, enter user-agent as the Key and sitecorebot as the Value.
Optionally, in the Render Javascript section, turn on Render JavaScript if you want the crawler to wait for and crawl the JavaScript on the page in addition to the page source.
Click Save.

Configure a request trigger

Configure triggers to give the advanced web crawler a starting point to look for content to index.

In this example, you use a sitemap as the trigger.

Note

Depending on your implementation, you can also use a sitemap index, a request, a JavaScript function, or an RSS feed as the trigger.

To configure a request trigger:

On the Source Settings page, click Edit next to Triggers.
Click Add Trigger.
In the Trigger Type drop-down field, select Request.
Optionally, in the Body field, enter the body of the request.

Note

If you use a POST or Patch request, you have to enter a request body.
Optionally, to configure a header, click Add Header and enter values in the Key and Value fields.

For example, enter user-agent as the Key and sitecorebot as the Value.
Optionally, in the Method drop-down menu, click POST, PUT, or PATCH. By default, GET is selected.
In the URL field, paste the URL you want to use as the trigger.

For example, paste https://dev.sitecore.net/

Create an XPath document extractor

Configure a document extractor to specify how to extract attribute values.

Note

For an advanced web crawler, you can also configure a CSS document extractor or a Javascript document extractor.

In this example, you use an XPath document extractor.

To create an XPath document extractor for an advanced web crawler source:

Log in to the Customer Engagement Console (CEC), click Sources , and select an advanced web crawler source.
On the Source Settings page, next to Document Extractors, click Edit ().
To create a document extractor, on the Document Extractors page:
- In the Name field, enter a meaningful name for this extractor.
  
  For example, enter Sitecore dev portal XPath extractor.
- In the Extractor Type drop-down menu, click XPath.
- Optionally, in the URLs To Match field, click Add Matcher, select the TYPE of expression you want to use, and enter the VALUE of that expression.
To define how to extract attributes, in the Taggers section, click Edit next to the first tag. Usually, it is the content tag.

You see a list of some attributes and corresponding XPath extraction expressions. Search provides these sample attributes and expressions to help with configuration.

Note

Within a document extractor, you can create multiple taggers, each linked to a unique tag. Each tagger generates a set of index documents. Taggers have rules, each of which defines the extraction logic of one attribute.

For example, if you have one tagger with five rules, you get one set of index documents where each document has five attributes. However, if you have three taggers with one rule each, you get three sets of index documents where each document has one attribute.
To delete the sample attributes you don't need, click or Delete.
To edit the configuration of a sample attribute, click Edit, and then make changes.
To configure how to extract the value of a new attribute, click Add Rule and enter the following details:
- In the Attribute drop-down menu, select the attribute you want to configure,
  
  For example, click Abstract.
- In the Value type drop-down menu, choose whether you want the attribute value to be a fixed value or an expression.
  
  For example, click Expressions .
- In the EXPRESSION field, enter an XPath expression that results in the attribute value.
  
  For example, if you want to get the value of an attribute called abstract from the text of all p elements contained within class attributes that contain the word abstract, enter the following XPath expression:
  RequestResponseHTML
  //*[contains(@class, "abstract")]//p
Optionally, to configure more than one way to get an attribute value, click Add Selector . Then, in the Expression field of the second selector, enter the XPath expression that results in the attribute value.

For example, as a second option, you want to get the abstract from the text of all p elements that are within the divclass of topic-content. To do this, enter the following XPath expression:
RequestResponseHTML
```
//div[@class="topic-content"]//p
```
When there are multiple selectors, Search runs them in chronological order and stops when it arrives at an expression that gives a result.
In the tag editor, click Save. Then, on the Document Extractors page, click Save.
Optionally, to extract attributes for another tag, click Add Tagger , and in the Tag drop-down menu, click a tag and repeat Steps 7 through 9.
On the Document Extractors page, click Save.

Schedule scans

To create a crawler schedule:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click the source for which you want to want to schedule crawls.
On the Source Settings page, click Edit next to Crawler Scheduler.
To configure when you want Search to start scheduled crawls, click a value in the STARTS drop-down menu.

If you want the schedule to start as soon as possible, select Anytime. If you want the crawl to start on a particular date, click Specific Date, and click a date in the date picker.
To have the crawl run on a regular schedule, in the REPEAT drop-down menu, click Yes.

Tip

To schedule a crawl to happen just once on a future date, click DOES NOT REPEAT in the REPEAT drop-down menu. With this configuration, a crawl will happen on the date you selected in Step 1 and will not repeat.
To define the frequency of crawls, click values next to the Repeats every field.

You can click any value from 1 to 99 as the interval and either days, weeks, or hours as the unit of time.

For example, if you want a crawl to run every four weeks, click 4 and weeks.
To define the time at which you want crawls to start, click a value in the RUN TIME drop-down menu.

For example, if you want the crawl to start at midnight, click 12:00 AM.
To configure when you want the crawler schedule to end, click a value in the END DATE drop-down menu.

If you want the schedule to continue indefinitely, select Never. If you want the crawl to end on a particular date, click Specific Date, and click a date in the date picker.
Click Save.

Publish the source

You must publish the source for Search to start the first scan and index.

To publish a source:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click the source you want to publish and click Publish.
In the Publish Source dialog, if you want Search to start a recrawl for this source, select the Trigger source recrawl after publishing checkbox.
Click Publish.

Do you have some feedback for us?

If you have suggestions for improving this article,