Walkthrough: Configuring a web crawler source

The Sitecore Search web crawler is a source that crawls your content by starting from a point and following hyperlinks. For each hyperlink it comes across, the web crawler creates an index document. The web crawler is straightforward, easy to configure, and requires no coding.

The web crawler begins from a starting point, called a trigger. If your starting point is a sitemap or sitemap index, the crawler crawls all URLs in the sitemap or sitemap index. If your starting point is a single URL, called a request, the crawler, follows hyperlinks from those pages to new pages.

The web crawler can crawl and index HTML pages, PDFs, and all Microsoft Office content.

Note

This walkthrough describes how to:

Create a web crawler source
Configure the scope of the advanced web crawler
Use an HTML tag to extract the value for an attribute
Use XPath to extract the value for an attribute
Use a fixed value for an attribute value
Schedule scans
Publish the source

Create a web crawler source

To create a source:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click Add Source.
In the SOURCE NAME field, enter a name for the source. For example, enter Doc site web crawler.
In the DESCRIPTION field, enter a few lines to describe the source you want to configure.

For example, enter Web crawler to crawl all pages of the doc site.
In the CONNECTOR field, click the type of source you want to create.
Click Save.

If there are no errors, Search creates a new source.

After you create a source, configure the source to access and index content.

In this example, select Web Crawler in step 5.

Configure the trigger and scope of the web crawler

You configure web crawler settings to define the trigger and other important high-level configurations that define the scope of the advanced web crawler.

To configure the trigger and scope of the web crawler:

Go to Sources and then click the source you created.
On the Source Settings page, click Edit ( ) next to Web Crawler Settings.
To configure a trigger for the web crawler, in the TRIGGER TYPE drop-down menu, click the type of trigger you want to use.

For example, click Sitemap.

Note

You can also create a sitemap index trigger or a request trigger.
In the URL field, enter the URL of the trigger.

For example, enter the sitemap URL https://doc.sitecore.com/search/sitemap.xml.

Note

If you select a sitemap or sitemap index trigger, you can add more than one URL.
To configure the depth and number of URLs for the crawler to crawl:
- In the MAX DEPTH field, enter the maximum number of levels the crawler should follow for a URL.
  
  For example, enter 5.
- In the MAX URLS field, enter the maximum number of URLs for the crawler to crawl, in total. Enter a large number to ensure that the crawler leaves no URL out.
  
  For example, enter 5000
To exclude certain URL patterns from the crawler's scope, click Add Exclusion Pattern. Then, in the TYPE drop-down menu, select Glob expression or Regular Expression. In the VALUE field, enter the expression to match URLs to exclude.

For example, to prevent the crawler from crawling your search page, enter the following Glob expression:
RequestResponseshell
```
**/search/**
```
In the TIMEOUT field, enter the time, in milliseconds, that the crawler waits to get a response.

For example, enter 5000. This ensures that the crawler waits for 5000 milliseconds, or 5 seconds, to get a response from every URL it crawls.
Optionally, in the HEADERS section, click Add Header. Then, in the Key field, enter the name of the user agent that your content expects. In the Value field, enter the value of the user agent that your content expects. This security measure ensures that only the Search crawler, and not any other crawler, can crawl the data.

For example, enter user-agent as the Key and sitecorebot as the Value.
Click Save.

Use HTML meta tags to extract the value for an attribute

You can use HTML meta tags to extract an attribute value.

In this example, you use an HTML meta tag.

To use an HTML tag to extract the value for an attribute:

On the Source Settings page, click Edit next to Attribute Extraction.
On the Attribute Extraction page, click Add Attribute .
In the attribute selector, click Add next to the attribute whose value you want to extract, and then click Add at the bottom.

In this example, click Description.
In the Type drop-down menu, click Meta Tag
In the VALUE field, enter the name of the meta tag or property tag you want to use.

For example, enter description. When you do this, Search internally creates the following XPath expressions to get the value of the description attribute from the content of description meta tag:

First, it creates and runs //meta[@name='description']/@content. If it does not get a value from this expression, it creates and runs //meta[@property='description']/@content. In this example, however, the first expression will give a value.
Click Save.

Use an XPath expression to extract the value of an attribute

You can use an XPath expression to extract an attribute value.

To use an XPath expression to extract the value of an attribute:

On the Source Settings page, click Edit next to Attribute Extraction
On the Attribute Extraction page, click Add Attribute .
In the attribute selector, click Add next to the attribute whose value you want to extract, and then click Add at the bottom.

For example, click Name.
In the Type drop-down menu, click Xpath
In the VALUE field, enter the XPath expression you want to use.

For example, enter //div[@class='wy-menu-vertical']/p. This tells Search to get the value of the name attribute from the text within the <p> tag of the page's first <div> tag with a class value of wy-menu-vertical.
Click Save.

Use a fixed value for an attribute value

You can assign a fixed value for an attribute if the value is a constant and does not need to be extracted.

For example, a common use case is to assign a fixed value for the type attribute.

To use a fixed value for an attribute:

On the Source Settings page, click Edit next to Attribute Extraction
On the Attribute Extraction page, click Add Attribute .
In the attribute selector, click Add next to the attribute whose value you want to extract, and then click Add at the bottom.

For example, click Type.
In the Type drop-down menu, click Fixed
In the VALUE field, enter the constant to use for this attribute.

For example, enter Sitecore Search documentation.
Click Save.

Schedule scans

To create a crawler schedule:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click the source that you want to want to schedule crawls for.
On the Source Settings page, click Edit next to Crawler Scheduler.
To configure when you want Search to start scheduled crawls, click a value in the STARTS drop-down menu. If you want the schedule to start as soon as possible, select Anytime. If you want the crawl to start on a particular date, click Specific Date, and click a date in the date picker.
To have the crawl run on a regular schedule, in the REPEAT drop-down menu, click Yes.

Tip

To schedule a crawl to happen just once on a future date, click DOES NOT REPEAT in the REPEAT drop-down menu. With this configuration, a crawl will happen on the date you selected in Step 1 and will not repeat.
To define the frequency of crawls, click values next to the Repeats every field. You can click any value from 1 to 99 as the interval and either days, weeks, or hours as the unit of time. For example, if you want a crawl to run every four weeks, click 4 and weeks.
To define the time at which you want crawls to start, click a value in the RUN TIME drop-down menu. For example, if you want the crawl to start at midnight, click 12:00 AM.
To configure when you want the crawler schedule to end, click a value in the END DATE drop-down menu. If you want the schedule to continue indefinitely, select Never. If you want the crawl to end on a particular date, click Specific Date, and click a date in the date picker.
Click Save.

Publish the source

You must publish the source to trigger the first scan and index.

To publish a source:

Log in to the Customer Engagement Console (CEC) and click Sources.
Click the source you want to publish and click Publish.
In the Publish Source dialog, if you want Search to start a recrawl for this source, select the Trigger source recrawl after publishing checkbox.
Click Publish.

Do you have some feedback for us?

If you have suggestions for improving this article,