Tracking changes to the xDB index

Abstract

Overview of the xDB change tracking settings, and how they affect live indexing, index rebuild, and performance.

The xDB tracks incoming changes to contacts, interactions, and facets. The xConnect Search Indexer uses this data to synchronize the index with data in the collection database

Tracking changes with the xDB Collection SQL provider

If you are using the xDB Collection SQL provider, xConnect uses the SQL Server change tracking feature. The change tracking retention period determines how long changes are stored in SQL Server.

The Database Properties dialog box

We recommend that you do the following.

  • Enable auto cleanup on all shards. This is true by default.

  • Monitor the indexer closely.

  • Keep the change tracking retention period as small as possible.

Warning

When you set the change tracking value, we recommend that you allow enough time for your IT department to be able to fix the indexer during any potential system downtime. If they can fix it during downtime, you might avoid having to rebuild the index.  By default, the value is set to 5 days.

For more information on how to configure the xConnect search indexer, see: #UUID-ee071fd4-9a8e-0497-5e35-42c884165317.

Important

Azure Search index rebuild is only available in 9.0 Update 2 and later. If you are using an earlier version, you might have to set the index retention period to a higher value.

Impact on live indexing

The retention period affects the indexer’s ability to resume indexing after it has been offline. If the retention period is less than the downtime period, the indexer cannot index the data that was collected during the downtime. In this scenario, you must rebuild the index from scratch.

Impact on index rebuild

An index rebuild occurs in two logical steps:

  1. Indexing of all data up until the moment when the index rebuild was triggered.

  2. Indexing of all data collected after the index rebuild was triggered.

If step 1 takes longer than the configured retention period, the indexer cannot index all data collected in step 2. If this happens, you must do the following before triggering another index rebuild:

  • Increase the retention period on every shard.

  • Temporarily set auto cleanup to false on every shard.

Impact on performance

Each table with change tracking enabled has an internal table that stores data about each changed row.

Different retention times and table sizes affect performance in the following ways:

  • Short retention times produce high performance and save space. However, if the system fails, you must rebuild the indexer.

  • A longer retention period results in larger internal tables. This increases the overall database size, which can cause performance issues in a busy system.

  • Larger internal tables makes the Auto Cleanup process slower. However, we recommend that you do not disable the auto cleanup process. If you disable the auto cleanup process, the system gains even larger tables over time.

Tracking changes with the xDB Collection MongoDB provider

If you are using the xDB Collection MongoDB provider, xConnect uses a custom collection named [Changes] with a TTL index. The retention period is controlled programatically, and it uses the same default as SQL change tracking (5 days):

<!--
The `expireAfterSeconds` option for the Changes collection's TTL index.
Default: 432000 (5 days).
-->
<ExpireAfterSeconds>432000</ExpireAfterSeconds>

For more information, see Configuring the xConnect Search Indexer