Understanding product synchronization performance

Current version: 8.1

Commerce Connect has its own product data model and Product Synchronization service layer  for exchanging product data with one or more external commerce systems. Product synchronization is potentially very time consuming, but you can configure some elements of your system to reduce the time needed to synchronize products between the external commerce system and Sitecore.

This topic describes the following elements that can impact product synchronization performance:

Item buckets

When a new item is created in an item bucket, it is automatically placed in the root folder. In order to move it into its correct location within the bucket, the bucket needs to be synchronized.

Item bucket synchronization versus product synchronization

Item buckets use synchronization that works on items locally within the bounds of the bucket. Product synchronization occurs across boundaries between Sitecore and external systems, and it does not work on items, but on a product domain model.

Item bucket synchronization strategies

The product synchronization process offers two different strategies for when the bucket is synchronized. The strategy that is best for your situation depends on the size of the catalog and especially the number of products being added during synchronizations. The number of products being added often depends on whether the catalog contains seasonal products and on the type of products being sold. The two strategies are:

  • Instant bucket synchronization – this is the default strategy. Within a bucket, it is possible to synchronize a single item, possibly containing subitems, by calling the BucketManager.MoveItemIntoBucket(entityItem, root) method. When a single product is synchronized, this strategy is always used. With bulk product synchronization, the process breaks up the list of products and handles the products individually, although possibly in parallel. Depending on the number of items being added to the bucket, this strategy can become unfeasible.

  • Delayed bucket synchronization –. with this strategy, a temporary bucket is used for new product items and bucket synchronization is delayed until all products have been processed and new product items have been created in the root of the bucket. After synchronization, the bucket content is moved to the main bucket. This eliminates the time spent during bucket synchronization on accessing all existing bucket items in order to check if the new product items already exist. The time reduction can be significant, especially with big catalogs. You can configure delayed bucket synchronization and the use of a temporary bucket in pipelines; see the configuration file: /App_Config/Include/Sitecore.Commerce.Products.Delayed- SyncProductRepository.config.disabled.

Multithreading

Multithreading is built into the product synchronization. By default, a single thread is spawned from the processors that synchronize products, manufacturers, types, resources, divisions, and specifications. The threads are spawned for each repository being synchronized. The number of threads can be configured in the Sitecore.Commerce.Products.config file by adding the ProductSynchronization. NumberOfThreads setting and specifying the number of concurrent threads.

Delayed item events and indexing

Commerce Connect disables the triggering of item events and indexing during synchronization to avoid wasting resources on firing events or maintaining indexes before synchronization is done. Indexing is turned on after synchronization has finished. The following context and disablers are instantiated during synchronization:

  • Sitecore.SecurityModel.SecurityDisabler()

  • Sitecore.Data.Proxies.EventDisabler()

  • Sitecore.Data.Proxies.ProxyDisabler()

  • Sitecore.Data.DatabaseCacheDisabler()

  • Sitecore.Data.BulkUpdateContext()

Limited calls between systems

Reading product data once and then processing it in multiple pipelines reduces the number of calls between Sitecore and the external systems. All product entities in Commerce Connect are synchronized using the Commerce Connect pipeline, which reads the data from the external system in the individual pipelines. Synchronizing a single product can amount to many calls between the systems, and each call takes time and resources. However, product data can be read once and then passed on to the individual subpipelines for processing, which reduces the number of calls between the systems.

External resources

Resources can be located externally. Resources in Sitecore are stored as media items in the Media Library. Media items are binary blobs and can be large and time-consuming to import into Sitecore. Therefore, resources can either be imported into the Sitecore Media Library or simply referred to externally. If resources are imported, they are stored in a bucketed Products folder under the Media Library. If not imported, they can be referred to by a URI stored on the resource reference item.

Product IDs

The external product repository is always regarded as the main repository, which by default owns the products. This makes the ID of the products and artifacts in the external system the primary key. In Sitecore, the IDs of the corresponding items for products and artifacts are generated by Commerce Connect instead of relying on the default Sitecore implementation that automatically generates a new GUID. The default implementation is based on the MD5 hash algorithm and has the following format:

Item.ID = MD5.ComputeHash(Prefix + ExternalID);

By using a hash algorithm, a direct mapping between the IDs coming from the external system and the item IDs in Sitecore is generated. This has the following benefits:

  • There is no need for mapping tables, which take up space and are time-consuming to maintain and query.

  • Getting the item IDs is very fast.

  • There is no need to search for an item in Sitecore – by using the external ID as input to the hashing algorithm, the Sitecore item ID is calculated instantaneously.

Do you have some feedback for us?

If you have suggestions for improving this article,