--- title: "Configure a localized advanced web crawler" slug: "configure-a-localized-advanced-web-crawler" abstract: "How to configure a Sitecore Search crawler source to crawl and index for localized content." --- # Configure a localized advanced web crawler In this walkthrough, you learn how to configure a source to crawl and index localized content. This includes configuring available locales, configuring locale extractors, and making sure that localized versions of the same content share the same ID. > [!important] > To index localized versions of crawlable items, you can only use the advanced web crawler. This walkthrough describes how to: :::walkthrough 1. [Create the crawler](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#create-the-crawler) 2. [Configure the crawler's scope](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#configure-the-crawlers-scope) 3. [Configure a sitemap trigger](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#configure-a-sitemap-trigger) 4. [Configure available locales](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#configure-available-locales) 5. [Configure a JavaScript locale extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#configure-a-javascript-locale-extractor) 6. [Configure a JavaScript document extractor for localized content](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#configure-a-javascript-document-extractor-for-localized-content) 7. [Create a crawler scheduler](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#create-a-crawler-scheduler) 8. [Publish updates to the source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html#publish-updates-to-the-source) ::: > [!note] > If your original content requires authentication before the crawler can access it, configure [crawler authentication settings](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/crawler-authentication.html "Crawler authentication"). For example, your original content might need a GUI-based user name and password, or an access token or key in the request header. You can do this at any point after source creation. ## Create the crawler To index localized items, you need an advanced web crawler. If you already have a standard web crawler, you can [convert it to an advanced web crawler](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-web-crawler/convert-a-web-crawler-to-an-advanced-web-crawler.html "Convert a web crawler to an advanced web crawler"). To create a new advanced web crawler: 1. On the menu bar, click **Sources** > **Add Source**. 2. In the **SOURCE NAME** field, enter a name for the source. For example, enter *Doc site web crawler*. 3. In the **DESCRIPTION** field, enter a few lines to describe the source you want to configure. For example, enter *Web crawler to crawl all pages of the doc site*. 4. In the **CONNECTOR** field, click *Advanced web crawler* as the type of source you want to create. 5. Click **Save**. If there are no errors, Search creates a new source. After you create a source, [configure](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing.html "Configure indexing") the source to access and index content. ## Configure the crawler's scope You configure [crawler settings](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configure-crawler-settings.html "Configure crawler settings") to define important high-level configurations that define the scope of the web crawler. To configure the scope of the advanced web crawler: 1. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Advanced Web Crawler Settings**. 2. Optionally, in the **ALLOWED DOMAINS** field, enter the domains for the crawler to stay within. Do this to ensure that the web crawler only crawls Sitecore documentation domains, not external sites that might be linked. For example, to ensure that the web crawler only crawls Sitecore documentation domains, not external sites that might be linked, enter www.doc.sitecore.com. 3. To configure the depth and number of URLs for the crawler to crawl: - In the **MAX DEPTH** field, enter the maximum number of levels that you want the crawler to follow for a URL. For example, to limit the crawler to child pages of the main page, enter 1. - In the **MAX URLS** field, enter the maximum number of URLs for the crawler to crawl, in total. Enter a large number to ensure that the crawler leaves no URL out. For example, enter 5000. 4. To exclude certain URL patterns from the crawler's scope, click **Add Exclusion Pattern**. Then, in the **TYPE** drop-down menu, select **Glob expression** or **Regular Expression**. In the **VALUE** field, enter the expression to match URLs to exclude. For example, to prevent the crawler from crawling your search page, enter the following Glob expression: ```text **/search/** ``` 5. To configure the number of workers to crawl in parallel and an optional delay between requests: - Define the number of threads, or workers, that concurrently crawl and index content by clicking a value in the **PARALLELISM (WORKERS)** drop-down menu. For example, enter *2* to have only two workers crawl in parallel. This uses less memory than the default *5* workers. - Optionally, if you have configured only one worker, you can define a time for the crawler to wait before it accesses the next URL to index. To do this, in the **DELAY (MS)** field, enter time in milliseconds. For example, enter *3*. 6. In the **TIMEOUT** field, enter the time, in milliseconds, that the crawler waits to get a response. For example, enter *5000*. This ensures that the crawler waits for 5000 milliseconds, or 5 seconds, to get a response from every URL it crawls. 7. Optionally, to add headers, click **Add Header**. Then, in the **Key** field, enter the name of the user agent that your content expects. In the **Value** field, enter the value of the user agent that your content expects. This security measure ensures that only the Search crawler, and not any other crawler, can crawl the data. For example, enter *user-agent* as the **Key** and *sitecorebot* as the **Value**. 8. To stop the crawler from accepting navigation cookies, turn off **ENABLE NAVIGATION COOKIES DURING CRAWLING**. When you index content in multiple locales, it's important to disable cookie acceptance. This is because cookies can limit the crawler to the first locale it encounters, and the crawler might not index content across all locales. 9. Optionally, if you want the crawler to wait for and crawl the JavaScript on the page in addition to the page source, turn on **Render JavaScript** in the **Additional Settings** section, 10. Click **Save**. ## Configure a sitemap trigger The trigger gives the crawler a starting point to look for content to index. To configure a sitemap or sitemap index trigger: 1. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Triggers**. 2. Click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Trigger**. 3. In the **Trigger Type** drop-down menu: - If you have a sitemap, click **Sitemap**. - If you have a sitemap index, click **Sitemap Index**. 4. In the **TIMEOUT** field, enter the time, in milliseconds, that the crawler waits to get a response. Enter a large number here because sitemaps can take very long to load, and you do not want the crawler to time out when it reaches the maximum number of seconds to wait. For example, enter *10000*. 5. In the **URL** field, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Item** and then enter the URL of the sitemap or sitemap index. For example, enter *https://www.sitecore.com/sitemap.xml*. 6. Click **Save**. ## Configure available locales You configure available locales to define the subset of domain languages you want to use for this source. For example, you want English (US), French (France), and Japanese (Japan) as locales. The crawler adds English (US) as a default locale, but you must add French (France) and Japanese (Japan). > [!important] > To ensure stable ingestion, [do not include more than 40 locales](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/optimizing-search.html "Optimizing Search") per crawler source. To configure locales: 1. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Available Locales**. 2. On the **Available Locales** page, click inside the **LOCALES** field to see a list of locales available for this domain and then click the locales you want for this source. For example, click *fr_fr* and *ja_jp*. 3. Click **Save**. ## Configure a JavaScript locale extractor You configure a locale extractor to define how the crawler must extract the locale from each URL it crawls. > [!important] > In the extractor logic, the locale format must match the locale formatting in **Available Locales**. That is, `${language}_${country}` or `${language}-${country}`. In this example, you use a JavaScript function to define this rule. > [!note] > You can also configure a URL extractor (using a REGEX) or a header extractor. To configure a JavaScript locale extractor: 1. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Locale Extractors**. 2. To create a locale extractor, on the **Locale Extractors** page: - In the **Name** field, enter a meaningful name for the extractor. For example, enter *Sitecore.com JS locale extractor*. - In the **Extractor Type** drop-down menu, click **JS**. 3. In the **JS Source** field, enter a JS function in [Cheerio](https://cheerio.js.org/) syntax to extract locales. Here's a sample JavaScript function to extract locales: ```csharp function extract(request, response) { locales = ['fr-fr','ja-jp']; for (idx in locales) { locale = locales[idx]; if (request.url.indexOf('/' + locale + '/') >= 0) { return locale.toLowerCase().replace('-','_'); } } return "en_us"; } ``` The function uses this logic: - If the page URL has a locale, get the locale from the URL of the page and format it in lower case and with an underscore. This formatting is important because it must match the locale formatting in **Available Locales**, that is, *fr_fr* and *ja_jp*. - If the page URL has no locale, assign the index document the default locale of *en_us*. 4. Click **Save**. ## Configure a JavaScript document extractor for localized content Configure a document extractor to specify which attributes you want the advanced web crawler to extract. When you have localized content, configure the *id* in addition to other attributes that you might require. You do this to ensure that localized versions of the same content share the same ID. In this example, you use JavaScript to extract the *id*, *type*, *title*, *subtitle*, and *product* attributes. To configure a JavaScript document extractor for localized content: 1. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Document Extractors**. 2. To create a document extractor, on the **Document Extractors** page: - In the **Name** field, enter a meaningful name for the extractor. For example*Sitecore with languages* . - In the **Extractor Type** drop-down menu, click **JS**. 3. In the **Taggers** section, click **Add Tagger** . In the tag editor, you see a sample JavaScript function that returns the *description*, *name*, *type*, *image_url* and *url* attributes. Search provides this sample to help with configuration. 4. To enable localized content, turn on the **Localized** switch. 5. In the tag editor, paste a JavaScript function that returns attribute values. Ensure that the value of the *id* attribute is constant across localized versions of the same content. The function must use [Cheerio](https://cheerio.js.org/) syntax and must return an array of objects. For example, paste the following code: ```csharp function extract(request, response) { $ = response.body; url = request.url; locales = ['/fr-fr/', '/ja-jp/']; for (idx in locales) url = url.replace(locales[idx], '/'); id = url.replaceAll('/', '_').replaceAll(':', '_').replaceAll('.', '_'); return [{ 'id': id, 'type': $('meta[name="type"]').attr('content') || 'Others', 'title': $('h1').text(), 'subtitle': $('meta[name="description"]').attr('content') || $('section[data-component-name="Hero Banner"] div[class*="side-content"]>div>p, header div.lead p').text(), 'product': $('meta[name="product"]').attr('content') }]; } ``` The code uses the following logic to get attribute values: - `id` - get this from the URL after ignoring the locale. To generate an ID from a URL, first replace the locale with a slash (`/`). Then, replace all slashes (`/`), dots (`.`), and colons in the URL with an underscore (`_`). - `type` - use the content from the first `` tag. If this tag does not exist, use *other*. - `title` - use the text first `

` HTML tag. - `subtitle` - use the content from the first `meta name="type"` tag. If this tag does not exist, use the text from the `
` of the first `section` with a `` tag. - `product` - use the content from the first `` tag. 6. In the **Edit Taggers** window, click **Save**. Then, on the **Document Extractors** page, click **Save**. ## Create a crawler scheduler To create a crawler schedule: 1. On the menu bar, click **Sources**. 2. Click the source that you want to want to schedule crawls for. 3. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Crawler Scheduler**. 4. To configure when you want Search to start scheduled crawls, click a value in the **STARTS** drop-down menu. If you want the schedule to start as soon as possible, select **Anytime**. If you want the crawl to start on a particular date, click **Specific Date**, and click a date in the date picker. 5. To have the crawl run on a regular schedule, in the **REPEAT** drop-down menu, click **Yes**. > [!tip] > To schedule a crawl to happen just once on a future date, in the **REPEAT** drop-down menu, click **DOES NOT REPEAT**. With this configuration, a crawl will happen on the date you selected in Step 4 and will not repeat. 6. To define the frequency of crawls, click values next to the **Repeats every** field. You can click any value from **1** to **99** as the interval and either **days**, **weeks**, or (for *production* domains only) **hours** as the unit of time. For example, if you want a crawl to run every four weeks, click **4** and **weeks**. 7. To define the time at which you want crawls to start, click a value in the **RUN TIME** drop-down menu. For example, if you want the crawl to start at midnight, click **`12:00` AM**. The time displayed is automatically adjusted to your timezone. 8. To configure when you want the crawler schedule to end, click a value in the **END DATE** drop-down menu. If you want the schedule to continue indefinitely, select **Never**. If you want the crawl to end on a particular date, click **Specific Date**, and click a date in the date picker. 9. Click **Save**. ## Publish updates to the source You must [publish](urn:resource:component:1033035) the source to start the first scan and index. To publish a source: 1. On the menu bar, click **Sources**. 2. Click the source you want to publish and click **Publish**. 3. In the **Publish Source** dialog, if you want Search to start a [recrawl](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/keep-indexes-up-to-date.html "Keep indexes up to date") for this source, select the **Trigger source recrawl after publishing** check box. 4. Click **Publish**.