Skip to main content

Rebuild the xDB index in Solr

Abstract

How to create a rebuild core, start the rebuilding process, confirm the rebuild and clean up the rebuild core.

This topic describes how to rebuild the xDB Search index, if you are using the Solr search provider. Before rebuilding, ensure that your environment meets the following requirements:

  • There are two xConnect Solr cores: the live core (usually xdb) and the rebuild core, which has a _rebuild suffix (like xdb_rebuild).

  • Both cores are based on the basic_configs file.

  • xConnect schema modifications have been applied to both cores.

Important

Rebuilding the reporting database does not trigger a rebuild of the index. You must rebuild the index separately.

Invoke the rebuild request command

The indexer changes process type, access way, and working folder depending on where it is hosted:

Sitecore hosted on-prem

Sitecore hosted in cloud

Process type

A Windows service on the xConnect Collection Search instance.

An Azure App Service WebJob on the xConnect Collection Search instance.

Access way

PowerShell.

The Kudu debug console.

<WORKING> folder

<xConnect instance>\App_Data\jobs\continuous\IndexWorker\

D:\local\Temp\jobs\continuous\IndexWorker\<SUBFOLDER>\

Each time Azure redeploys the WebJob, it creates a uniquely named temporary subfolder, for example bp003r3f.h2g/, for storing the indexer working files.

To invoke the rebuild request command:

  1. Go to the server where the xConnect Search Indexer is running and open a command line.

  2. Go to the xConnect Collection Search IndexWorker folder.

  3. Run Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild). The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

Create a rebuild core

If you do not have a rebuild core, complete the following steps. If you have a rebuild core, skip this section.

To create a rebuild core:

  1. Go to the Solr core folder: <Solr root>\server\solr

  2. Create a copy of your core with a _rebuild suffix. If your core is named xdb, name the copy xdb_rebuild.

  3. Open the xdb_rebuild folder and delete the core.properties file and the data folder.

  4. Open a browser and go to the Solr administrative interface (usually https://localhost:8983/solr/).

  5. Click Core Admin in the left-hand menu:

    Core Admin in the Solr administrative interface

  6. Click Add Core and type in the name and path (InstanceDir) of your rebuild core (leave other properties unchanged):

    Add Core in the Solr administrative interface

Rebuild the core

You cannot trigger an index rebuild from a user interface or from the xConnect Client API. You must trigger the rebuild process in one of the following ways:

Invoke the rebuild request command

To invoke the rebuild request command:

  1. Go to the server where the xConnect Search Indexer is running and open a command line.

  2. Go to the xConnect Collection Search IndexWorker folder: <xConnect Collection Search service root>\App_Data\jobs\continuous\IndexWorker.

  3. Run Sitecore.XConnectSearchIndexer.exe -rr (alternatively Sitecore.XConnectSearchIndexer.exe -requestrebuild). The command registers a small document in the live core signaling that the rebuild should be started. The xConnect Search Indexer will notice the document and start the rebuild process.

    Note

    If you run a RequestRebuild command when an index rebuild process is running, the existing process is cancelled and a new one is started.

Manually post a rebuild document

To manually post a rebuild document:

  • Post the following document directly from PowerShell to the Solr live core: Invoke-WebRequest https://localhost:8983:/solr -Method Post -Body '{"id":"xdb-rebuild-status","lastupdated_dt":"2017-05-24T10:15:47.603436Z","rebuildstate_l":1,"progresstotal_d":0.0,"progresscurrent_d":0.0}'

Confirm rebuild

When the rebuild is complete, the cores are swapped, and the rebuilt core becomes the live core. When you swap cores, names are swapped, but physical folder locations are not swapped:

Name Before Rebuild

Name After Rebuild

Core Location

xdb (live)

xdb_rebuild

C:\Solr-8.2.0_https\server\solr\xdb

xdb_rebuild

xdb (live)

C:\Solr-8.2.0_https\server\solr\xdb_rebuild

At this point, the rebuild core is no longer in use. However, it still has all data that was in the live core before the swap.

Manually swapping cores back

If you need to swap back (not rebuild a second time), you must first make sure the cores were recently swapped. Swapping back only works if:

  • The xDB Collection provider change tracking data still holds all changes since the cores were swapped.

  • The xConnect Search can be offline for a period of time. This is necessary for the cores that are swapped back to get up to date by pulling the above changes. The smaller the time window since the swap, the less time this takes.

Assuming you meet these requirements, proceed with the following instructions. If you do not meet them, you can not successfully swap cores.

  1. Stop the indexer.

    Warning

    Failing to stop the indexer before swapping back the cores might cause the indexer to skip data.

  2. Swap the cores by using the Solr swap command.

  3. Start the indexer again.

Clean up the rebuild core

To save space, you can delete deprecated index data from the old core.

To save space, you can delete deprecated index data from the old core.

Note

Make sure you delete the old core, if your reason for rebuilding is to remove all PII sensitive data from the index. The old core still contains this data.