Skip to main content

Rebuild the xDB index in Solr

Abstract

How to create a rebuild core, start the rebuilding process, confirm the rebuild and clean up the rebuild core.

This topic describes how to rebuild the xDB Search index, if you are using the Solr search provider. Before rebuilding, ensure that your environment meets the following requirements:

  • There are two xConnect Solr cores: the live core (usually xdb) and the rebuild core, which has a _rebuild suffix (like xdb_rebuild).

  • Both cores are based on the basic_configs file.

  • xConnect schema modifications have been applied to both cores.

Important

Rebuilding the reporting database does not trigger a rebuild of the index. You must rebuild the index separately.

Invoke the rebuild request command

The indexer changes process type, access way, and working folder depending on where it is hosted:

Sitecore hosted on-prem

Sitecore hosted in cloud

Process type

A Windows service on the xConnect Collection Search instance.

An Azure App Service WebJob on the xConnect Collection Search instance.

Access way

PowerShell.

The Kudu debug console.

<WORKING> folder

<xConnect instance>\App_Data\jobs\continuous\IndexWorker\

D:\local\Temp\jobs\continuous\IndexWorker\<SUBFOLDER>\

Each time Azure redeploys the WebJob, it creates a uniquely named temporary subfolder, for example bp003r3f.h2g/, for storing the indexer working files.

To invoke the rebuild request command:

  1. Go to the server where the xConnect Search Indexer is running and open a command line.

  2. Go to the xConnect Collection Search IndexWorker folder.

  3. Run Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild). The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

Create a rebuild core

If you do not have a rebuild core, complete the following steps. If you have a rebuild core, skip this section.

Note

For Managed Cloud solutions, skip this section, because xdb_rebuild is provisioned out of the box.

To create a rebuild core:

  1. Go to the Solr core folder: <Solr root>\server\solr

  2. Create a copy of your core with a _rebuild suffix. If your core is named xdb, name the copy xdb_rebuild.

  3. Open the xdb_rebuild folder and delete the core.properties file and the data folder.

  4. Open a browser and go to the Solr administrative interface (usually https://localhost:8983/solr/).

  5. Click Core Admin in the left-hand menu:

    Core Admin in the Solr administrative interface

  6. Click Add Core and type in the name and path (InstanceDir) of your rebuild core (leave other properties unchanged):

    Add Core in the Solr administrative interface

Create aliases

To create aliases for your collections:

  1. In the Solr administration UI, go to the Collections page using one of the nodes you created.

  2. Click Create Alias.

  3. Create an alias named xdb_rebuild and in the Collections list, click xdb_rebuild_internal.

  4. Create a second alias named xdb and select xdb_internal in the Collections list.

  5. To see a list of all aliases, use the URL https://<solrurl>/solr/admin/collections?action=LISTALIASES.

  6. Apply the schema modifications from the <xconnect-root>\App_Data\solrcommands\schema.json file to both collections using the Solr Schema API.

Note

Use xdb and xdb_rebuild in your connection strings, not xdb_internal and xdb_rebuild_internal. xdb and xdb_rebuild are the default names of the primary and secondary indexes. If you change the names of the indexes, you must update all connection strings.

Note

Don't give a collection and an alias the same name.

You have now created a SolrCloud farm. Before you can use it with xConnect, configure the xConnect roles to use SolrCloud.

Rebuild the core

You can't trigger an index rebuild from a user interface or from the xConnect Client API. Instead, trigger the rebuild process in one of the following ways:

Invoke the rebuild request command

To invoke the rebuild request command:

  1. Go to the server where the xConnect Search Indexer is running and open a command line.

  2. Go to the xConnect Collection Search IndexWorker folder: <xConnect Collection Search service root>\App_Data\jobs\continuous\IndexWorker.

  3. Run Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild). The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

    Note

    If you run a RequestRebuild command when an index rebuild process is running, the existing process is cancelled and a new one is started.

Manually post a rebuild document

To manually post a rebuild document:

  • Post the following document directly from PowerShell to the Solr live core: Invoke-WebRequest https://localhost:8983:/solr -Method Post -Body '{"id":"xdb-rebuild-status","lastupdated_dt":"2017-05-24T10:15:47.603436Z","rebuildstate_l":1,"progresstotal_d":0.0,"progresscurrent_d":0.0}'

Confirm rebuild

When the rebuild is complete, the cores are swapped, and the rebuilt core becomes the live core. When you swap cores, names are swapped, but physical folder locations aren't swapped:

Name Before Rebuild

Name After Rebuild

Core Location

xdb (live)

xdb_rebuild

C:\Solr-8.2.0_https\server\solr\xdb

xdb_rebuild

xdb (live)

C:\Solr-8.2.0_https\server\solr\xdb_rebuild

At this point, the rebuild core is no longer in use. However, it still has all data that was in the live core before the swap.

Manually swapping cores back

If you need to swap back (not rebuild a second time), you must first make sure the cores were recently swapped. Swapping back only works if:

  • The xDB Collection provider change tracking data still holds all changes since the cores were swapped.

  • The xConnect Search can be offline for a period of time. This is necessary for the cores that are swapped back to get up to date by pulling the above changes. The smaller the time window since the swap, the less time this takes.

Assuming you meet these requirements, proceed with the following instructions. If you do not meet them, you can not successfully swap cores.

  1. Stop the indexer.

    Warning

    Failing to stop the indexer before swapping back the cores might cause the indexer to skip data.

  2. Swap the cores by using the Solr swap command.

  3. Start the indexer again.

Clean up the rebuild core

To save space, you can delete deprecated index data from the old core.

To save space, you can delete deprecated index data from the old core.

Note

Make sure you delete the old core, if your reason for rebuilding is to remove all PII sensitive data from the index. The old core still contains this data.