Manage site sources
Feature availability is part of a phased rollout. Your organization may not see this functionality yet. It will become available when your environment is included in the rollout.
A site source is an index of content that has been collected from a website through crawling. When you create a site source, SitecoreAI discovers pages from a website you specify, extracts content, and stores it in a search index that can be used by search components.
When a visitor uses a search component connected to a site source, the results come from the webpages indexed by that site source.
Unlike content sources, which ingest content from the Content editor, site sources collect content directly from websites using sitemap-based discovery and crawling.
Some websites use firewalls, bot protection, or web application firewall (WAF) services that can block crawlers. If your website restricts incoming traffic, you might need to allowlist SitecoreAI crawler IP addresses before the site can be indexed successfully.
To index data from the Content editor, use a content source.
Create a site source
A site source crawls and indexes a website. When creating a site source, you configure rules that tell the crawler which sites to crawl and how to extract data on those sites. SitecoreAI then finds pages using a sitemap and optional starting URLs, extracts configured fields from each page, and stores the results in a search index.
-
On the menu bar, click Content > Search sources, and then click Create Source.
-
In the Create new search source dialog, click Site Source.
-
In the Source Name field, enter a name for your source. Optionally, add a Description for your source.
-
Click Custom Url, and enter the URL of the website you would like to index content from. Click Confirm
, then choose a Re-index Schedule and click Next Step. -
Use the Configure indexing rules menu to see which sitemaps SitecoreAI has identified. If necessary, click Add Sitemap to manually add a sitemap to this list, and then click Next Step.
-
Click the Locales drop-down list to choose the languages that this source will support, and then click Next Step.
-
On the URL pattern filters menu, configure URL pattern filters:
-
To control which pages are indexed, click the Policy drop-down list and select either Allow or Disallow.
-
Click the Rule drop-down list to select an operator. For example, "begins with".
-
In the Path pattern field, enter a value that triggers the rule. For example, "/sitecore".
-
Optionally, use the Preview menu to see which fields are excluded or included based on this rule.
-
Click Next Step.
-
-
On the Configure fields menu, configure extraction and fields:
ImportantWhen you mark fields as Searchable, the Allowance meter will increase by the number of languages selected for this source. Once the allowance is reached, you cannot mark additional fields as Searchable.
-
In the Test Extraction field, enter a URL to preview the data that will be extracted from the URL.
-
Configure the fields in your source and their behavior by selecting the Include, Searchable, and Filterable check boxes.
-
Optionally, click Add field to configure a new field that the crawler will index.
-
Optionally click Edit
to configure which HTML elements, meta tags, or CSS selectors are used to extract the field during crawling. -
Click Next Step.
-
-
Enable the advanced settings you want to use in your source and then click Save.
Reindex content for a site source
Because information on a website can change unexpectedly, it is important to keep site source indexed up to date with the latest information.
When you reindex a site source, SitecoreAI rebuilds the underlying index and seamlessly switches the source to use the updated index. This ensures that search experiences connected to the source continue to return results while the reindexing process completes.
To reindex content for a source:
-
On the menu bar, click Content > Search sources.
-
Next to the source you want to reindex content for, click Reindex content.
View a site source
Clicking on a source allows you to view and edit that source's configurations, including fields and rules. Clicking on a site source also shows details of the most recent indexing job, such as how many items were indexed successfully.
To view a site source:
-
On the menu bar, click Content > Search sources.
-
Click a source.
-
Click any of the following tabs:
-
Crawl - to display crawl job details for this source.
-
Fields - to display the list of fields indexed by the source, and their configurations.
-
Source Config - to display sitemaps and sitemap configurations for this source.
-
Pinning Rules - to display a list of pinning rules active on the source. Click Add a pin to create a pin rule.
-
Boost/Bury Rules - to display a list of boost and bury rules active on the source. Click Add new rule to create a new boost or bury rule.
-
Settings - to display the source settings, such as the source name, ID, and advanced settings.
-
View job run details
Clicking on a sitemap source also allows you to see indexing jobs details, such as the number of sitemap items included and excluded. If a crawl is unsuccessful,
To view job run details:
-
On the menu bar, click Content > Search sources.
-
Click a source.
-
In the Crawl tab, click a job to see details related to that crawl job, such as job status, duration, and number of items indexed.
Delete a site source
When you delete a source, search components connected to that source do not return results. You can select a new source in the Source Configuration Manager menu.
To delete a source:
-
On the menu bar, click Content > Search sources.
-
Next to the source you want to delete, Actions
. -
Click Delete.
-
To confirm the deletion, click Delete.
Configure a search component
After you have created a source, the next step is to configure a search component so your search experience can display the content indexed by that source.