Configuring the xConnect Search Indexer

This topic describes the configuration settings that affect the xConnect Search Indexer. In a single machine on-premise deployment, indexer configuration is located under under C:\>Path to xConnect>\App_data\jobs\continuous\IndexWorker\App_data\Config.

Important

The xConnect Search Indexer does not use all of the configuration settings available in the \App_data\Config sub-folder - for example, the indexer does not use the settings in sc.Xdb.Collection.RepositorySettings.xml.

Change tracking retention period

The xConnect Search Indexer is affected by the underlying provider’s change tracking retention period configuration.

sc.XConnect.SearchIndexer.xml

The following settings control how often the indexer checks for changes.

File Path: C:\<Path to xConnect>\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\SearchIndexer\sc.XConnect.SearchIndexer.xml

Setting

Description

Frequency

Determines how often the indexer checks for changes. If the indexer is busy for 0.20 seconds, it will way 0.05 seconds before trying again. If the indexer is busy for longer than 0.25 seconds, it will check for changes immediately after indexing the previous set.

DelayAfterError

Determines how long the indexer waits before retrying after encountering an error.

DelayAfterRecurringError

Determines how long the indexer waits before retrying after the RecurringErrorThreshold has been met. Increase value to reduce number of log entries in the event of a recurring error.

RecurringErrorThreshold

Determines how many times an error can occur before the DelayAfterRecurringError setting takes effect.

sc.Xdb.Collection.IndexerSettings.xml

The following settings control memory usage during index rebuild.

File Path: C:\<Path to xConnect>\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\SearchIndexer\sc.Xdb.Collection.IndexerSettings.xml

Setting

Description

IncomingDataLagOnCompletion

At the end of an index rebuild, the indexer must catch up with incoming data. When the indexer is behind by less than the value specified by IncomingDataLagOnCompletion, the cores are swapped.

ParallelizationDegree

Determines how many parallel streams of data are processed during an index rebuild. Use together with BatchSize to tune memory usage during an index rebuild.

Reduce if an index rebuild is consuming too much memory. Can be increased if the indexer is not consuming very much memory.

BatchSize

Determines how many contacts or interactions are loaded at one per parallel stream during an index rebuild. Use together with ParallelizationDegree to tune memory usage during an index rebuild. Reduce if an index rebuild is consuming too much memory. Can be increased if the indexer is not consuming very much memory.

SplitRecordsThreshold

Limits the number of records loaded into memory (per core) by the indexer at any one time. Decrease if indexer is using too much memory. To disable splitting, set the value to to 0, a negative value, or remove the element completely.

sc.Xdb.Collection.Data.Sql.xml

The following settings apply if you are using the SQL xDB Collection provider.

File Path: C:\<Path to xConnect>\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\Collection\sc.Xdb.Collection.Data.Sql.xml

Setting

Description

NumberOfChangeVersions

Determines the number of transactions (not records) that will be returned from the change table. For example, 500 transactions may include much more than 500 changes.

The value of this setting is a high value (50000) by default to prevent the indexer from falling behind and keeping the number of pending changes low. Because a high value will increase the indexer’s memory requirements, consider decreasing the value of SplitRecordsThreshold if the indexer is consuming too much memory. Reducing the NumberOfChangeVersions setting will have a limited effect, as only a small amount of data (IDs, facet keys, and types of change) is loaded from the change table.

GetChangesCommandTimeoutInSeconds

Determines timeout for the get changes command. Increase this value if you experience timeouts during indexing.

CommandTimeoutInSeconds

Determines timeout for collection provider commands such as GET and SAVE.

AbsoluteExpiration

The SQL provider caches meta-data about sharded cluster configuration. This setting controls expiration of the cache.

sc.Xdb.Collection.Data.MongoDb.xml

The following settings apply if you are using the MongoDB xDB Collection provider.

File Path: C:\<Path to xConnect>\App_data\jobs\continuous\IndexWorker\App_data\Config\Sitecore\Collection\sc.Xdb.Collection.Data.MongoDb.xml

sc.Xdb.Collection.IndexWriter.AzureSearch.xml

The following settings are specific to the Azure Search provider.

Setting

Description

MaximumUpdateBatchSize

Controls number of contacts or interactions that are sent in a single post. Cannot be larger than 1000. Decrease if large contacts or interactions lead to rejected posts.

ParallelizationDegree

Consider increasing if you are using multiple Azure Search partitions. Be aware that increasing this value increases memory consumption. It is recommended that you monitor memory usage of a busy indexer when changing this value. Monitor using IndexWriteAvgTime and IndexWriteAvgBatchSize performance counters.

Note: If this setting is removed from configuration, the provider will fall back to the number of cores available on the host machine.

MaximumRetryDelayMilliseconds

Controls the maximum amount of time the provider will wait when Azure Search fails temporarily due to load. Consider increasing this value if Azure Search consistently returns a 503 error, as this may allow the system time to return related errors.It is recommended that you analyze your Azure Search traffic early and frequently.

RetryCount

Determines how many times the provider retries before reporting an error to the indexer. An error is logged when the retry count is exceeded. Modify together with MaximumRetryDelayMilliseconds.

MaximumWaitTimeoutMilliseconds

Determines maximum time the indexer will wait for changes to be indexed. If the limit is hit, indexer will retry indexing and the following entry will be added to the logs: “Waiting documents in index is failed by timeout”. Recurring timeout issues indicate that Azure Search is under heavy load. Consider using Azure Search partitions or increasing the value of the setting.

DataReplicationTimeoutMilliseconds

If you are using Azure Search replicas, this setting determines the amount of time to wait after initially detecting changes before signalling to the xConnect Search service that changes are available to be searched. The delay allows time for data to propagate across replicas. There is currently no way to determine if data is available in all replicas, and it is therefore difficult to tune this setting.

Can be set to 0 if replicas are not being used.

sc.Xdb.Collection.IndexWriter.SOLR.xml

The following settings are specific to the Solr search provider.

Setting

Description

MaximumUpdateBatchSize

Determines the number of contacts or interactions in a single post.

MaximumDeleteBatchSize

Determines the number of contact or interactions deletes in a single post. Although deletes usually concern IDs, deleting of child documents may be affected by Solr’s OR clause limit.

MaximumCommitMilliseconds

Determines how fast data is soft committed (commitWithin) in Solr.

ParallelizationDegree

Consider increasing if you are using multiple Solr replicas. Be aware that increasing this value increases memory consumption. It is recommended that you monitor memory usage of a busy indexer when changing this value. Monitor using IndexWriteAvgTime and IndexWriteAvgBatchSize performance counters.

Note: If this setting is removed from configuration, the provider will fall back to the number of cores available on the host machine.

MaximumRetryDelayMilliseconds

Controls the maximum amount of time the provider will wait when Solr fails temporarily due to load. Consider increasing this value if Solr consistently returns a 503 error as this may allow the system time to return related errors.

RetryCount

Determines how many times the provider retries before reporting an error to the indexer. An error is logged when the retry count is exceeded. Modify together with MaximumRetryDelayMilliseconds.

Encoding

Determines encoding for Solr posts. Only utf-8 has been tested.