Batch indexing

Current version: 10.1

xConnect increments a sync token in the storage layer each time a batch is submitted to xConnect. If the client’s SyncToken property is set when a search is performed, results are only returned when indexing has caught up to at least that number.

Important

The guarantee that all batches have been indexed when search results are returned is local to the client instance. If you submit data to xConnect in parallel threads, client A might return search results before client B.

The latest sync token is returned to the client alongside the results of the batch:

RequestResponse
await client.SubmitAsync();

var searchSynchronizationToken = client.SyncToken;
Note

Change tracking is not enabled for device profiles.

If you submit 5 batches of 10,000 contacts, indexing is complete for those 10,000 contacts when the indexing worker has indexed changes up to and including the fifth batch. This batch is represented by a specific sync token. To confirm that indexing is complete for those 50,000 contacts:

  1. Submit five batches.

  2. Save a reference to the SyncToken that is returned with the final batch. For example, List Manager stores sync tokens in the list definition items.

  3. In the code that checks whether or not indexing is complete, set the SyncToken property to the saved token.

  4. Perform a dummy search using this client. Results will only be returned if indexing has caught up to the sync token that was supplied.

It does not matter what you search for - no results will be returned until indexing has caught up to at least the number specified. If the SyncToken is null, results are returned immediately - the client does not wait for indexing to catch up to a particular sync token.

Scenario: Saving and using a SyncToken

In the following scenario, xConnect client A adds a large number of contacts and subscribes them to a new list by setting the ListSubscriptions facet. On a separate page, xConnect client B is using contact search to return all contacts in that list. However, client B’s SyncToken property is not set - therefore, results will be returned before indexing is complete and the results count will not be accurate.

Two separate operations on contacts could conflict if you don't use a sync token

To ensure that results are only returned when indexing is complete, save the SyncToken returned to client A. For example, you might maintain a SQL table of list IDs and the sync token from the latest batch of contacts to be added to that list:

Job ID

Subscription List ID

Sync Token

1

a256fe54-9915-4779-afe1-511b74795dde

6e26fe54-9915-4779-afe1-511b74795dde

Before performing a search, set the SyncToken property to the saved value as shown. Results will not be returned until indexing is complete.

RequestResponse
using Sitecore.XConnect;
using Sitecore.XConnect.Client;
using Sitecore.XConnect.Collection.Model;
using System;
using System.Linq;

namespace Documentation
{
    public class ChangeTracking
    {
        // Async example
        public async void ExampleAsync()
        {
            using (XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                // Manually set sync token
                client.SyncToken = Guid.Parse("6e26fe54-9915-4779-afe1-511b74795dde").ToByteArray();

                try
                {
                    var listId = Guid.Parse("a256fe54-9915-4779-afe1-511b74795dde");
                    var query = client.Contacts.Where(c => c.ListSubscriptions().Subscriptions.Any(s => s.ListDefinitionId == listId));

                    // Will only return results when indexing is complete
                    var results = await query.GetBatchEnumerator();
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception or timeout
                }
            }
        }

        // Sync Example
        public void ExampleSync()
        {
            using (XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                // Manually set sync token
                client.SyncToken = Guid.Parse("6e26fe54-9915-4779-afe1-511b74795dde").ToByteArray();

                try
                {
                    var listId = Guid.Parse("a256fe54-9915-4779-afe1-511b74795dde");
                    var query = client.Contacts.Where(c => c.ListSubscriptions().Subscriptions.Any(s => s.ListDefinitionId == listId));

                    // Will only return results when indexing is complete
                    var results = query.GetBatchEnumeratorSync();
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception or timeout
                }
            }
        }
    }
}

Do you have some feedback for us?

If you have suggestions for improving this article,