Sitecore Experience Platform

Rebuild the xDB index in Azure Search

Abstract

How to rebuild the xDB index, check the rebuild status, and check if the index is active.

This topic describes how to rebuild the xDB index in Azure Search. Index rebuild functionality was only introduced in 9.0 Update 2, but you can take steps to refresh the index in earlier Sitecore 9 versions.

The xConnect Search Indexer uses the connection strings to create any missing indexes on startup or at the beginning of the rebuild process. There are two indexes:

  • The primary index, named xdb by default.

  • The secondary index, named <Primary Index>-secondary (making it xdb-secondary).

You do not need a dedicated connection string for the secondary index, but if your installation requires it, you can configure one.

Rebuild the index

To rebuild the index:

  1. Open xConnect Collection Search role console (you can use the Kudu Debug console). By default, the xConnect Search Indexer is hosted as a WebJob in the context of the xConnect Collection Search role.

  2. Go to the xConnect Search Indexer folder at \<xConnect role root>\App_Data\jobs\continuous\IndexWorker\

    azr-rebuild-11.png
  3. Start the index rebuild process with this command: .\XConnectSearchIndexer.exe -rr

  4. The xConnect Search Indexer returns Done to indicate that the rebuild process has started.

Requests are automatically directed to the active index, when the rebuild process is complete. You do not need to make any changes to the connection strings.

Important

Rebuilding the reporting database does not trigger a rebuild of the index. You must rebuild the index separately.

Rebuild to a different subscription or index name

You can also use the rebuild process when you want to:

When the primary index is the active index

If the primary index is the current active index, complete the following steps to rebuild to a different index name or a different subscription.

  1. Confirm that the primary index (identified by the collection.search connection string) is the current active index.

  2. Create a patch file for the <xConnect role root>\App_Data\Config\Sitecore\CollectionSearch\sc.Xdb.Collection.IndexReader.AzureSearch.xml configuration file and add it to the <xConnect role root>\App_Data\config\global folder.

  3. In the patch file, specify a secondary connection string:

    <?xml version="1.0" encoding="utf-8" ?>
    <Settings>
        <Sitecore>
            <XConnect>
                <CollectionSearch>
                    <Services>
                        <IAzureSecondaryWebClientFactory>
                            <Options>
                                <ConnectionStringName>
                                    collection.search.secondary
                                </ConnectionStringName>
                            </Options>
                        </IAzureSecondaryWebClientFactory>
                    </Services>
                </CollectionSearch>
            </XConnect>
        </Sitecore>
    </Settings>
  4. Open the <xConnect role root>\App_Config\ConnectionStrings.config configuration file.

  5. Add a new connection string with the same name that you specified previously in step 2 and add in your subscription, index name, and API key. In this example, the secondary connection string name is collection.search.secondary and the index name is xdb-custom-secondary:

    <add name="collection.search.secondary" connectionString="serviceUrl=https://[service].search.windows.net/;indexName=xdb-custom-secondary;apiKey=[API Key]"/>
    <add name="collection.search" connectionString="serviceUrl=https://[service].search.windows.net/;indexName=xdb;apiKey=[API Key]" />
    
  6. Repeat on any other instances of the xConnect Collection Search service role and the xConnect Search Indexer.

  7. Rebuild the index. When the rebuild is complete, xdb-custom-secondary becomes the active index. You do not need to restart the indexer before rebuilding. If an index does not exist, the indexer creates it.

  8. When the rebuild is complete, change the default connection string (collection.search) so that the primary index is recreated in the new subscription.

  9. At this point you can delete the original index and the original subscription. Make sure no roles are referencing the original subscription.

When the secondary index is the active index

If the secondary index is the current active index, complete the following steps to rebuild to a different subscription. In this scenario, you do not need to configure a custom secondary connection string.

  1. Confirm that the primary index (identified by the collection.search connection string) is not the current active index.

  2. Change the primary collection.search connection string to point to your new subscription or index name.

    <add name="collection.search" connectionString="serviceUrl=https://[service].search.windows.net/;indexName=xdb;apiKey=[NEW API KEY]" /> <!-- MAINTENANCE> -->
  3. Rebuild the index. When the rebuild is complete, xdb becomes the active index in the new subscription, and xdb-secondary becomes the maintenance index.

  4. When the rebuild is complete, you can delete the old subscription that contains the old indexes.

Note

If you want to change the name of the secondary index, you must do this before rebuilding from the primary to the secondary index. You can not change the name of the secondary index when it is active.

Checking if a rebuild is complete

There are several ways to check if a rebuild is complete:

  • Check the xConnect Search Indexer logs. The Rebuilding process. Stage: Finished text indicates that the rebuild process is complete.

    az-rebuild-41.png
  • Check the status of the xdb-rebuild-status document with this command: $filter=id eq 'xdb-rebuild-status'&$select=id,rebuildstate

    A RebuildState of 6 indicates that rebuild has completed.

  • With a PowerShell, you can monitor the index rebuild progress.

Checking if an index is active

The Azure Search provider uses two physical cores: The primary and the secondary. At any time, one core is actively used for search and indexing of live data, and the other is used for maintenance operations such as rebuild. After a rebuild, the xConnect Search Indexer swaps the physical cores but does not rename them.

To know which physical index is currently active, follow these steps:

  1. Open one of the indexes in Azure Search.

  2. Perform this search: $filter=id eq 'indexconfiguration'&$select=id,isactivecore

  3. The IsActiveCore field is true, if the index is active.

To save space, you can delete deprecated index data from the old core.

Note

If your are rebuilding the index to remove all sensitive PII (Personally Identifying Information), make sure you delete the old core. Otherwise it still contains this data.

Refresh the content of a 9.0 Initial Release and 9.0 Update 1 index

There is no index rebuild process available for Azure Search between Sitecore 9.0 Initial Release and 9.0 Update 1. However, you can use this procedure to refresh the content of the index:

  1. Stop the indexer WebJob.

  2. Delete the index.

  3. Restart the indexer.

Important

This procedure recreates the index and adds data which has changed within the last five days (the default retention period in SQL). Data older than the value of the retention period is not be added to the index. The process relies on the xDB Collection provider change tracking retention period being high enough to maintain a record of all incoming changes until the indexer has processed all existing data. Note that setting a very high change tracking retention period has detrimental effects on performance.

Important

Rebuilding the reporting database does not trigger a rebuild of the index. You must rebuild the index separately.