---
title: "Manage site sources"
slug: "manage-site-sources"
abstract: "How to create, reindex, and delete content search sources in SitecoreAI."
---
# Manage site sources

> [!important]
> Feature availability is part of a phased rollout. Your organization may not see this functionality yet. It will become available when your environment is included in the rollout.

A site source is an index of content that has been collected from a website through crawling. When you create a site source, SitecoreAI discovers pages from a website you specify, extracts content, and stores it in a search index that can be used by search components.

When a visitor uses a search component connected to a site source, the results come from the webpages indexed by that site source.

Unlike content sources, which ingest content from the **Content editor**, site sources collect content directly from websites using sitemap-based discovery and crawling.

To index linked resources, such as documents referenced from crawled pages, see [Anchor indexing](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/manage-site-sources/anchor-indexing.html "Anchor indexing").

> [!important]
> Some websites use firewalls, bot protection, or web application firewall (WAF) services that can block crawlers. If your website restricts incoming traffic, you might need to [allowlist SitecoreAI crawler IP addresses](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/manage-site-sources/sitecore-ip-allowlist.html "Sitecore IP allowlist") before the site can be indexed successfully.
> [!tip]
> To index data from the Content editor, [use a content source](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/manage-content-sources.html "Manage content sources").

## Create a site source

A site source crawls and indexes a website. When creating a site source, you configure rules that tell the crawler which sites to crawl and how to extract data on those sites. SitecoreAI then finds pages using a sitemap and optional starting URLs, extracts configured fields from each page, and stores the results in a search index.

> [!tip]
> You can save a source as a draft while you configure it and continue working on it later.

1. On the menu bar, click **Content** > **Search sources**, and then click **Create Source**.
2. In the **Create new search source** dialog, click **Site Source**.
3. In the **Source Name** field, enter a name for your source. Optionally, add a **Description** for your source.
4. Click **Custom Url**, and enter the URL of the website you would like to index content from. Click **Confirm** :mdiCheck:, then choose a **Re-index Schedule** and click **Next Step**.
5. Use the **Configure indexing rules** menu to see which sitemaps SitecoreAI has identified. If necessary, click **Add Sitemap** to manually add a sitemap to this list, and then click **Next Step**.
6. Click the **Locales** drop-down list to choose the languages that this source will support, and then click **Next Step**.
7. On the **URL pattern filters** menu, configure URL pattern filters:
    - To control which pages are indexed, click the **Policy** drop-down list and select either **Allow** or **Disallow**.
    - Click the **Rule** drop-down list to select an operator. For example, "begins with".
    - In the **Path pattern** field, enter a value that triggers the rule. For example, "/sitecore".
    - Optionally, use the **Preview** menu to see which fields are excluded or included based on this rule.
    - Click **Next Step**.
8. On the [**Anchor indexing**](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/manage-site-sources/anchor-indexing.html) menu, configure how SitecoreAI indexes resources from pages in this source:
   - Enter a name for the rule, then choose how many links per page to check against it.
   - In the **Match** drop-down list, select **All** or **Any** to specify whether all or any of the conditions must match.
   - Add a condition that identifies the links to index. For example, configure a condition to match links that end with `.pdf`.
   - To refine which links are indexed, add conditions or rules. To index other types of linked resources, add another anchor document.
   - To test your configuration, enter a URL in the **Test extraction** field and click **Test extraction**. The results show how your rules apply to links on that page.
9.  On the **Configure fields** menu, configure extraction and fields:

    > [!important]
    > When you mark fields as Searchable, the Allowance meter will increase by the number of languages selected for this source. Once the allowance is reached, you cannot mark additional fields as Searchable.

    - In the **Test Extraction** field, enter a URL to preview the data that will be extracted from the URL.
    - Configure the [fields](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/understanding-search-sources/fields.html "Fields") in your source and their behavior by selecting the check boxes.
    - Optionally, click **Add field** to configure a new field that the crawler will index.
    - Optionally click **Edit** :mdiPencilOutline: to configure which HTML elements, meta tags, or CSS selectors are used to extract the field during crawling.
    - Click **Next Step**.
10.  [Enable the advanced settings](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/understanding-search-sources/advanced-settings.html "Advanced settings") you want to use in your source and then click **Save**.

## Reindex content for a site source

Because information on a website can change unexpectedly, it is important to keep site source indexes up to date with the latest information.

When you reindex a site source, SitecoreAI rebuilds the underlying index and seamlessly switches the source to use the updated index. This ensures that search experiences connected to the source continue to return results while the reindexing process completes.

To reindex content for a source:

1. On the menu bar, click **Content** > **Search sources**.
2. Next to the source you want to reindex content for, click **Reindex content**.

## View a site source

Clicking on a source allows you to view and edit that source's configurations, including fields and rules. Clicking on a site source also shows details of the most recent indexing job, such as how many items were indexed successfully.

To view a site source:

1. On the menu bar, click **Content** > **Search sources**.
2. Click a source.
3. Click any of the following tabs:
    - **Crawl** - to display crawl job details for this source.
    - **Fields** - to display the list of fields indexed by the source, and their configurations.
    - **Source Config** - to display sitemaps and sitemap configurations for this source.
    - **Pinning Rules** - to display a list of [pinning rules](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/understanding-search-sources/rules.html "Add pin rules to your source") active on the source. Click **Add a pin** to create a pin rule.
    - **Boost/Bury Rules** - to display a list of boost and bury rules active on the source. Click **Add new rule** to create a new boost or bury rule.
    - **Settings** - to display the source settings, such as the source name, ID, and advanced settings.
    - **Preview** - displays a preview of the content indexed in this source, and the number of fields that have been indexed for each item.

## View job run details

Clicking on a sitemap source also allows you to see indexing jobs details, such as the number of sitemap items included and excluded, or if a crawl is unsuccessful.

To view job run details:

1. On the menu bar, click **Content** > **Search sources**.
2. Click a source.
3. In the **Crawl** tab, click a job to see details related to that crawl job, such as job status, duration, and number of items indexed.

## Delete a site source

> [!important]
> When you delete a source, search components connected to that source do not return results. You can select a new source in the Source Configuration Manager menu.

To delete a source:

1. On the menu bar, click **Content** > **Search sources**.
2. Next to the source you want to delete, **Actions** :mdiDotsHorizontal:.
3. Click **Delete**.
4. To confirm the deletion, click **Delete**.

## Configure a search component

After you have created a source, the next step is to [configure a search experience component](https://doc.sitecore.com/sai/en/users/sitecoreai/search-experiences/search-experience-components/configure-a-search-component.html "Configure a search component") so your search experience can display the content indexed by that source.
