---
title: "Configure a feed crawler"
slug: "configure-a-feed-crawler"
abstract: "How to configure a feed crawler, including document and object level transformers, in Sitecore Search."
---
# Configure a feed crawler

In Sitecore Search, a feed crawler is used to index data from documents such as:

- Text files containing delimiters.
- CSV files.
- JSON string files.

It systematically extracts data from these files and transforms it into searchable content within Sitecore Search. This process involves creating a feed crawler source, configuring document extractors, and setting up transformers to handle the data.

> [!note]
> If you select the sales feed crawler connector type, then certain [sales feed crawler configurations](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/sales-feed-crawler-configurations.html "Sales feed crawler configurations") must be present.

> [!important]
> CSV files must be correctly formatted before being uploaded. Errors such as invalid quoting, an inconsistent number of field per row, or a malformed header row will cause the crawler to fail.

This walkthrough describes how to:

:::walkthrough
1. [Create a feed crawler source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#create-a-feed-crawler-source)
2. [Choose locales](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#choose-locales)
3. [Define tags](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#define-tags)
4. [Create a document extractor](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#create-a-document-extractor)
5. [Add document and column transformers](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#add-document-and-column-transformers)
6. [Set incremental updates](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#set-incremental-updates)
7. [Publish the source](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/configure-a-feed-crawler.html#publish-the-source)
:::

> [!note]
> **Before you begin**
> Documents can only be uploaded to the feed crawler via Secure File Transfer Protocol (SFTP). To receive your SFTP credentials, contact Sitecore support.

## Create a feed crawler source

The first step is to create a source. This is a connector that retrieves items from a specified website, document, or repository, and indexes them so they are searchable.

To create a source:

1. In the menu bar, click **SOURCES**.
2. Click **Add Source** ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) to add a source.
3. In the **SOURCE NAME** field, enter a name for the source.
4. In the **DESCRIPTION** field, enter a few lines to describe the source you want to configure.
5. In the **CONNECTOR** drop-down list, click **Feed Crawler**.
6. Click **Save**. If there are no errors, Search creates a new source.

## Choose locales

[Locales](https://doc.sitecore.com/search/en/users/search-user-guide/localize-implementation/localize-implementation.html "Localize implementation") specify the geographical and linguistic regions for which the data is relevant. Searchuses locale values to ensure that data indexed by the feed crawler is appropriate for the language and regional settings of the target audience, to direct visitors to locale specific pages, and to create locale specific rules for all types of widgets.

To choose locales for your source:

1. On the menu bar, click **Sources**, then select your feed crawler source.
2. In the **Source Settings** menu, click **Available Locales**.
3. In the **Locales** drop-down list, select a locale that you have configured for the domain.
4. Click **Save**.

## Define tags

[Tags](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/crawler-tags/crawler-tags.html "Crawler tags") are used to create search experiences for specific entities. When you set up a source, the **Tags Definition** window is used to specify which entities will be updated by that source.

To use tags to update specific entities with this source:

1. On the menu bar, click **Sources**, then select your feed crawler source.
2. In the **Source Settings** menu, click **Tags Definition**.
3. In the **Entity** drop-down list, select an entity to be updated by this source.
4. To configure a basic tag, in the **From** drop-down list, click **Tags**.
5. In the **Tags** field, enter a name for the tag.
6. Click **Save**.

## Create a document extractor

Document extractors process and extract data from the files uploaded to the feed crawler, and convert it into a structured format that can be indexed by Search. Document extractors come with default JavaScript logic to perform this conversion, but you can edit this logic to suit your needs if you require further configuration.

To create a document extractor:

1. On the menu bar, click **Sources** and then select your feed crawler source.
2. In the left pane, click **Document Extractors** and then in the **Document Extractors** section, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit**.
3. To create an extractor, on the **Document Extractors** page, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Extractor**, and then on the **Document Extractors** page, do the following:
    - In the **Basepath** field, enter the file's basepath that is appended to */upload/*. For example, enter *files*.
    - In the **File Name** field, enter the filename. For example, enter *filename.json*.
    - In the **File Type** drop-down menu, select *CSV* or *JSON*.
    - If the file is in GZip format, turn on the **isGzip** switch.

    Uploading a document using SFTP will automatically trigger the source.

4. In the **Taggers** section, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Tagger**. Then, in the tag editor, select a [tag](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/crawler-tags/crawler-tags.html "Crawler tags") in the **Tag** drop-down list. For example, select *content*.

    > [!note]
    > In a document extractor, you can create multiple taggers where each is linked to a unique tag. This way each tagger:
    >
    > -   Generates a set of index documents.
    >
    > -   Can have multiple rules such that each rule defines the extraction logic of one attribute.
    >
    >
    > For example, one tagger with five rules yields *one* set of documents each with *five* attributes.
    >
    > Three taggers with one rule each yields *three* sets of documents each with *one* attribute.

5. In the **Extraction Type** field, select either **Base** or **JavaScript** from the extraction logic drop-down list.
    - To use existing column headers from the uploaded document as indexed attribute names, select **Base** from the drop-down list, enter ID column header in the **ID Field** field, and enter the separator used in the CSV file in the **Separator** field.
    - To alter Search 's default extractor logic, select **Javascript** from the drop-down list, and edit the function in the **JS Source** window.

    ```csharp
    // Sample extractor function. Change the function to suit your individual needs
    function extract(headerSegments, lineSegments) {   
        response = {}   
        for (i = 0; i < lineSegments.length; i++) {     
            response[headerSegments[i]] = lineSegments[i]   
        }   
        return [response]
    }
    ```

    > [!note]
    > If the function name in the **Function Name** field and the **JS Source** window do not match, an error will occur when the extractor is initiated.

6. In the tag editor, click **Save**.
7. (Optional) To extract attributes for another tag, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Add Tagger**, and in the **Tag** drop-down list, click a tag and repeat steps 6 through 10.
8. (Optional) To add another document extractor, repeat steps 4 to 6.
9. On the **Document Extractors** page, click **Save**.

## Add document and column transformers

When you configure a feed crawler, Search uses transformers to index structured documents like CSV and JSON. [Additional transformers can be added and configured](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/additional-configurations-and-settings/transformers/transformers.html "Transformers") to further alter data before indexing.

To add document and column transformers:

1. In the left pane, click **Transformers** and then, in the **Transformers** section, click ![pencil_grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/pencil_grey.png) **Edit**.
2. On the **Transformers** page, in the left pane, click **Document Transformers**.
3. In the **Document Transformers** pane, [add a document transformer](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/create-document-and-attribute-transformers.html#UUID-c389e7cf-a576-5566-2e76-de3cfc9212f1_section-idm1743454764186570 "Add a document transformer").
4. To add another document transformer, repeat step 3.
5. In the left pane, next to **Column Transformers**, click ![Plus_add__grey.png](https://doc.sitecore.com/api/markdown-image/search/en/users/search-user-guide/images/Plus_add__grey.png) **Column Transformer**.
6. In the **Column Transformers** pane, [add a column transformer](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/configure-indexing/create-document-and-attribute-transformers.html#UUID-c389e7cf-a576-5566-2e76-de3cfc9212f1_section-idm1743454764193594 "Add an attribute transformer").
7. To add another column transformer, repeat step 6.
8. Click **Save**.

## Set incremental updates

Uploading a file via SFTP will trigger the source, but you can also make updates to your source via API updates like `Patch` or `Puts`. To update the source with API requests, enable [incremental updates](https://doc.sitecore.com/search/en/users/search-user-guide/index-items/keep-indexes-up-to-date/enable-incremental-updates-for-a-crawler.html "Enable incremental updates for a crawler").

To enable incremental updates to your source:

1. On the menu bar, click **Sources** and then select your feed crawler source.
2. In the **Source Settings** menu, click **Incremental Updates**.
3. Click the **Enable Incremental Updates** switch to enable the feature, then click **Save**.

## Publish the source

You must publish the source to start the first scan and index.

To publish the source:

- To open the **Publish Source** dialog, in the upper right corner of the source page, click **Publish**.