--- title: "Configure an API crawler" slug: "configure-an-api-crawler" abstract: "How to configure an API crawler in Sitecore Search." --- # Configure an API crawler The Sitecore Search API crawler is a powerful crawler specifically designed to handle JSON content. It supports complicated use cases like needing authentication to access your source content, [creating index documents in multiple languages](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-localized-advanced-web-crawler.html "Configure a localized advanced web crawler"), using JavaScript to extract attribute values, and more. The API crawler works by accessing URLs or API endpoints and indexing the content in each URL or endpoint. > [!important] > Refer to [detailed specifications of crawler types](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/crawler-specifications.html "Crawler specifications") when [choosing an indexing method](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/identify-the-required-source-type.html "Identify the required source type"). > > Follow these [indexing best practices](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/best-practices-for-indexing.html "Best practices for indexing") to successfully [crawl complete websites](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/best-practices-to-crawl-complete-websites.html "Best practices to crawl complete websites") or [crawl frequent new updates](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/best-practices-to-crawl-frequent-new-updates.html "Best practices to crawl frequent new updates"). > > For most cases, we recommend that you first create a web crawler source. Then, if you find that you need more features, [convert the web crawler source to an advanced web crawler source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/convert-a-web-crawler-to-an-advanced-web-crawler.html "Convert a web crawler to an advanced web crawler"). This walkthrough describes how to: :::walkthrough 1. [Create an API crawler source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#create-an-api-crawler-source) 2. [Configure crawler settings](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#configure-crawler-settings) 3. [Configure a trigger](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#configure-a-trigger) 4. [Configure a request extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#configure-a-request-extractor) 5. [Configure a document extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#configure-a-document-extractor) 6. [Schedule scans](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#schedule-scans) 7. [Publish updates to the source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-an-api-crawler.html#publish-updates-to-the-source) ::: > [!note] > If your original content requires authentication before the crawler can access it, configure [crawler authentication settings](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/crawler-authentication.html "Crawler authentication"). For example, your original content might need a GUI-based user name and password, or an access token or key in the request header. You can do this at any point after source creation. ## Create an API crawler source To create a source: 1. On the menu bar, click **Sources**. 2. Click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Source**. 3. In the **SOURCE NAME** field, enter a name for the source. 4. In the **DESCRIPTION** field, enter a few lines to describe the source you want to configure. 5. In the **CONNECTOR** drop-down list, click **API Crawler**. 6. Click **Save**. If there are no errors, Search creates a new source. ## Configure crawler settings Configure [crawler settings](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configure-crawler-settings.html "Configure crawler settings") to define important high-level configurations that define the scope of the API crawler. To configure API crawler settings: 1. Click **Sources** and then click the source you created. 2. On the **Source Settings** page, next to **API Crawler Settings**, click **Edit** ![pencil_purple.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_purple.png). 3. To configure the depth and number of URLs for the crawler to crawl: - In the **MAX DEPTH** field, enter the maximum number of levels that you want the crawler to follow for a URL. For example, enter *5*. - In the **MAX URLS** field, enter the maximum number of URLs for the crawler to crawl, in total. Enter a large number to ensure that the crawler leaves no URL out. For example, enter *5000*. 4. To configure the number of workers to crawl in parallel and an optional delay between requests: - Define the number of threads, or workers, that concurrently crawl and index content by clicking a value in the **PARALLELISM (WORKERS)** drop-down menu. For example, enter *2* to have only two workers crawl in parallel. This uses less memory than the default *5* workers. - Optionally, if you have configured only one worker, you can define a time for the crawler to wait before it accesses the next URL to index. To do this, in the **DELAY (MS)** field, enter time in milliseconds. For example, enter *3*. 5. In the **TIMEOUT** field, enter the time, in milliseconds, that the crawler waits to get a response. The default is 10,000ms. For example, enter 5000. This ensures that the crawler waits for 5000 milliseconds, or 5 seconds, to get a response from every URL it crawls. 6. Optionally, to add headers, click **Add Header**. Then, in the **Key** field, enter the name of the user agent that your content expects. In the **Value** field, enter the value of the user agent that your content expects. This security measure ensures that only the Search crawler, and not any other crawler, can crawl the data. For example, enter *user-agent* as the **Key** and *sitecorebot* as the **Value**. 7. Optionally, if you want to stop the crawler from accepting navigation cookies, turn off **ENABLE NAVIGATION COOKIES DURING CRAWLING** in the **Additional Settings** section. Navigation cookies track the crawler's path and record the URLs it visits. Sometimes, cookies might mislead the crawler into re-indexing previously visited URLs. However, cookies are important for websites that need them for subsequent access, especially websites that need authentication. 8. Click **Save**. ## Configure a trigger Configure [triggers](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configure-triggers.html "Configure triggers") to give the advanced web crawler a starting point to look for content to index. You can use the following types of triggers: - [Request trigger](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configure-triggers/create-a-request-trigger.html "Create a request trigger") - [JavaScript trigger](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configure-triggers/create-a-javascript-trigger-to-get-a-set-of-urls.html "Create a JavaScript trigger to get a set of URLs") To configure a request trigger with a GraphQL API endpoint: 1. On the **Source Settings** page, next to **Triggers**, click **Edit** ![pencil_purple.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_purple.png). 2. Click ![plus_add__purple.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/plus_add__purple.png) **Add Trigger**. 3. In the **Trigger Type** drop-down list, click **Request**. 4. Optionally, in the **Body** field, enter the body of the request. > [!note] > If you use a `POST` or `Patch` request, you have to enter a request body. For example, use: ```text {"query":"query getItem($path: String) {\n item(language: \"en\", path: $path) {id path children {results {name}\n }\n }\n}\n","variables":{"path":"/sitecore/content/mvpsite"}} ``` 5. Optionally, to configure a header, click **Add Header**. Then, in the **Key** field, enter the name of the user agent that your content expects. In the **Value** field, enter the value of the user agent that your content expects. For example, enter *user-agent* as the **Key** and *sitecorebot* as the **Value**. > [!note] > This security measure ensures that only the Search crawler, and not any other crawler, can crawl your data. 6. Optionally, in the **Method** drop-down menu, click the HTTP method you want to use. The default, **GET**, is selected. For example, click **POST**. 7. In the **URL** field, enter the API endpoint you want to use as the trigger. For example, paste *https://edge.sitecorecloud.io/api/graphql/v1* > [!note] > You do not see the trigger response in Sitecore Search. However, it is useful to keep it in mind because you later configure a request extractor that uses the response of the trigger. In this example, the trigger returns the following JSON response: ```json { "data": { "item": { "id": "xxx", "path": "/sitecore/content/MvpSite", "children": { "results": [ { "name": "Home" }, { "name": "MVP Repository" }, { "name": "Shared Content" }, { "name": "Settings" } ] } } } ``` ## Configure a request extractor A [request extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/request-extractors.html "Request extractors") creates additional URLs for the crawler to crawl. Request extractors are very important when you configure an API crawler. For an API crawler, triggers return JSON and not URLs. To handle this, configure a request extractor to use the output of the trigger and return URLs or API endpoints for the API crawler to crawl. In this example, you need to configure a request extractor that uses the JSON objects that the trigger outputs, and generates API endpoints. To create a request extractor that uses JSON objects as the input and returns a list of API endpoints: 1. Click **Sources** and select the source you created. 2. On the **Source Settings** page, next to **Request Extractors**, click **Edit** ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png). 3. To create a [request extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/request-extractors.html "Request extractors"), on the **Document Extractors** page: - In the **Name** field, enter a meaningful name for the extractor. For example, enter *Sitecore video URLs.* - Optionally, in the **URLs To Match** field, select the **TYPE** of expression you want to use and enter the **VALUE** of that expression. For example, to crawl all URLs in this format: */homeloans/*, select **Glob Expression** and enter the **VALUE** as `**/homeloans/**.*` . 4. In the **JS Source** field, paste a JavaScript function that returns a list of URLs. > [!note] > The function must use [Cheerio](https://cheerio.js.org/) syntax and must return an array of objects. For example, paste: ```javascript function extract(request, response) { requests = []; if (response.body && response.body.data && response.body.data.item && response.body.data.item.children) { requests = response.body.data.item.children.results.map((e, i) => { name = e.name; path = JSON.parse(request.body).variables.path + "/" + name; return { url: request.url, method: 'POST', headers: { 'content-type': ['application/json'] }, body: JSON.stringify({ "query": "query getItem($path: String) {item(language: \"en\", path: $path) {id path rendered children {results {name}}}}", "operationName": "getItem", "variables": { "path": path } }) }; }); } return requests; } ``` This function returns the following API endpoints: ```json [ { "url": "https://edge.sitecorecloud.io/api/graphql/v1", "method": "POST", "headers": { "content-type": [ "application/json" ] }, "body": "{"query":"query getItem($path: String) {item(language: \\"en\\", path: $path) {id path rendered children {results {name}}}}","operationName":"getItem","variables":{"path":"/sitecore/content/Sugcon/SugconEuSxa/Home"}}" }, { "url": "https://edge.sitecorecloud.io/api/graphql/v1", "method": "POST", "headers": { "content-type": [ "application/json" ] }, "body": "{"query":"query getItem($path: String) {item(language: \\"en\\", path: $path) {id path rendered children {results {name}}}}","operationName":"getItem","variables":{"path":"/sitecore/content/Sugcon/SugconEuSxa/Media"}}" }, { "url": "https://edge.sitecorecloud.io/api/graphql/v1", "method": "POST", "headers": { "content-type": [ "application/json" ] }, "body": "{"query":"query getItem($path: String) {item(language: \\"en\\", path: $path) {id path rendered children {results {name}}}}","operationName":"getItem","variables":{"path":"/sitecore/content/Sugcon/SugconEuSxa/Data"}}" }, ..... ] ``` 5. Click **Save**. ## Configure a document extractor Configure a [document extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configuring-document-extractors.html "Configuring document extractors") to specify how to extract attribute values from each content item. The document extractor crawls the URLs or API endpoints that the request extractor generates. You can use the following types of document extractors: - [JavaScript](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configuring-document-extractors/create-a-javascript-document-extractor.html "Create a JavaScript document extractor") - [JSONPath](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configuring-document-extractors/create-a-jsonpath-document-extractor.html "Create a JSONPath document extractor") > [!note] > For an API crawler, you can configure a JavaScript extractor or a JSONPath document extractor. > > Keep the following points in mind when you decide which document extractor to use: > > - If the trigger or request extractor output is JSON, you can use a JSONPath document extractor or a JavaScript document extractor > > - If the trigger or request extractor output is XML, use a JavaScript document extractor. To create a JSONPath document extractor matching URLs with a JavaScript function: 1. On the menu bar, click **Sources**. 2. Select the source you created. 3. On the **Source Settings** page, next to **Document Extractors**, click **Edit** ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png). 4. To create a [document extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/configuring-document-extractors.html "Configuring document extractors"), on the **Document Extractors** page: - In the **Name** field, enter a meaningful name for the extractor. For example, enter *Sitecore cloud*. - In the **Extractor Type** drop-down menu, click **JSONPath**. - Optionally, to ensure that this extractor's logic only applies to [URLs that match](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configuring-document-extractors.html#UUID-bf3b8a27-e317-f7e3-9fe5-88d81969a12c_section-idm4597072673881633832915676016 "URLs to match") a certain pattern, configure **URLs to Match**. To do this, in the **URLs To Match** field, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Matcher**, select the **TYPE** of expression you want to use, and enter the **VALUE** of that expression. For example, click **JS** in the **TYPE** drop-down menu and enter the following expression to use JavaScript to ensure that the extractor only extracts attributes from API endpoints whose response has a value for `body.data.item.rendered`. ```javascript function match(request, response) { return response.body.data.item.rendered != null && response.body.data.item.rendered.sitecore.route.placeholders['headless-main'].length > 0; } ``` 5. In the **Taggers** section, click **Add Tagger** . Then, in the tag editor, select a [tag](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/crawler-tags.html "Crawler tags") in the **Tag** drop-down menu. For example, select *content*. > [!note] > In a document extractor, you can create multiple taggers where each is linked to a unique tag. This way each tagger: > > - Generates a set of index documents. > > - Can have multiple rules such that each rule defines the extraction logic of one attribute. > > > For example, one tagger with five rules yields *one* set of documents each with *five* attributes. > > Three taggers with one rule each yields *three* sets of documents each with *one* attribute. 6. In the tag editor, enter the following details to extract an attribute: - In the **Attribute** drop-down menu, click the attribute you want to configure. For example, click **Description**. - In the **Value type** drop-down menu, choose whether you want the attribute value to be a fixed value or an expression. For example, click **Expressions**. - In the **EXPRESSION** field, enter a JSONPath expression that results in the attribute value. For example, to get the value of *description* from the of the `..placeholders['headless-main']..fields.Description` key, enter ```shell ..placeholders['headless-main']..fields.Description.value ``` 7. Optionally, to configure more than one way to get an attribute value, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Selector**. Then, in the **Expression** field of the second selector, enter the JSONPath expression that results in the attribute value. For example, as a second option, you want to get the value of *description* from the `..placeholders['headless-main']..fields.Text` tag. To get this, enter the following JSONPath expression: ```shell ..placeholders['headless-main']..fields.Text.value ``` > [!note] > When there are multiple selectors, Search runs them in chronological order and stops when it arrives at an expression that gives a result. 8. To configure how to extract other attributes, click **Add Rule** ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) and repeat steps 5 and 6. For example, now that you've configured how to extract the *description* attribute, you can configure how to extract the *title*, *subtitle*, and *image* attributes. 9. In the tag editor, click **Save**. 10. (Optional) To extract attributes for another tag, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Tagger**, and in the **Tag** drop-down menu, click a tag and repeat Steps 5 through 8. 11. On the **Document Extractors** page, click **Save**. ## Schedule scans To create a crawler schedule: 1. On the menu bar, click **Sources**. 2. Click the source that you want to want to schedule crawls for. 3. On the **Source Settings** page, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit** next to **Crawler Scheduler**. 4. To configure when you want Search to start scheduled crawls, click a value in the **STARTS** drop-down menu. If you want the schedule to start as soon as possible, select **Anytime**. If you want the crawl to start on a particular date, click **Specific Date**, and click a date in the date picker. 5. To have the crawl run on a regular schedule, in the **REPEAT** drop-down menu, click **Yes**. > [!tip] > To schedule a crawl to happen just once on a future date, in the **REPEAT** drop-down menu, click **DOES NOT REPEAT**. With this configuration, a crawl will happen on the date you selected in Step 4 and will not repeat. 6. To define the frequency of crawls, click values next to the **Repeats every** field. You can click any value from **1** to **99** as the interval and either **days**, **weeks**, or (for *production* domains only) **hours** as the unit of time. For example, if you want a crawl to run every four weeks, click **4** and **weeks**. 7. To define the time at which you want crawls to start, click a value in the **RUN TIME** drop-down menu. For example, if you want the crawl to start at midnight, click **`12:00` AM**. The time displayed is automatically adjusted to your timezone. 8. To configure when you want the crawler schedule to end, click a value in the **END DATE** drop-down menu. If you want the schedule to continue indefinitely, select **Never**. If you want the crawl to end on a particular date, click **Specific Date**, and click a date in the date picker. 9. Click **Save**. ## Publish updates to the source You must publish the source to start the first scan and index. To publish a source: 1. On the menu bar, click **Sources**. 2. Click the source you want to publish and click **Publish**. 3. In the **Publish Source** dialog, if you want Search to start a [recrawl](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/keep-indexes-up-to-date.html "Keep indexes up to date") for this source, select the **Trigger source recrawl after publishing** check box. 4. Click **Publish**.