Walkthrough: Importing catalog data

Version: 10.3

This walkthrough describes how to import a ZIP file containing catalog data (previously exported from a Sitecore XC environment), into another Sitecore XC environment. For details on the expected data structure for the import operation, see the Walkthrough: Exporting catalog and inventory data topic.

Note

Sitecore XC currently supports full catalog imports only.

During standard operations, the Commerce Engine Search service uses the incremental index minion as an automated process that periodically adds new catalog items to an indexing list. This automated process ensures the discoverability of new catalog items. To help optimize performance during catalog import, the Catalog application programming interface (API) uses policy keys that prevent incremental indexing during the catalog data import process. This reduces the total time required to import catalog data.  

After the catalog data is imported, you can proceed with importing any related inventory data set (if required).  Finally, you must rebuild catalog indexes manually at the Commerce Engine level and at the Sitecore platform level.

To import catalog data and inventory sets (if required), and rebuild indexes, perform the following procedures:

Optimize configuration for importing large catalog data set

Depending on catalog size and complexity of data structure within the catalog, importing catalog data can take a long time or fail in some cases. You should make the following configuration changes before you start importing large, complex data sets:

Before you begin importing catalog data, you must modify the Internet Information Services (IIS) settings (in the IIS Manager and in the web.config file) to reduce the risk of timeouts occurring during the data export process. This is important when exporting a very large catalog data set.

To reconfigure the service used to import catalog data:

  1. Start the Internet Information Services (IIS) Manager and, in the navigation tree, click Application Pools.

  2. On the Application Pools page, click the name of the relevant application pool (for example, CommerceAuthoring_Sc). 

  3. In the Actions pane, in the Edit Application Pool section, click Advanced Settings and, in the Advance Settings dialog box, set the following values to 0:

    Section

    Parameter

    Value

    CPU

      Limit Interval (minutes) 

    0

    Process Model

     Idle Time-out (minutes)

    0

    Recycling

    Regular Time Interval (minutes)

    0

  4. Click OK.

  5. Open the web.config file for the service you use to import the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\web.config), and modify the settings marked in bold in the following configuration sample: 

    Note

    Before you change the IIS settings, take note of their original values so that you know what to set them back to after you have finished importing catalogs.

    RequestResponse
    <?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <security>
      <requestFiltering>
        <!-- For Microsoft IIS (Internet Information Services), maxAllowedContentLength specifies the maximum length of content in a request, in bytes. -->
        <requestLimits maxAllowedContentLength="2147483648" />
      </requestFiltering>
    </security>
    <handlers>
      <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
    </handlers>
    <modules runAllManagedModulesForAllRequests="false">
      <remove name="WebDAVModule" />
    </modules>
    <aspNetCore processPath=".\Sitecore.Commerce.Engine.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" requestTimeout="00:00:00" stdoutLogFile=".\logs\stdout" />
  </system.webServer>
</configuration>

  6. Reset the Internet Information Services.

Change the workflow entity caching policy configuration

For optimal performance of catalog import operation, change the caching configuration of the workflow entity ("EntityFullName": "Sitecore.Commerce.Plugin.Workflow.Workflow").

To change the caching policy for the workflow entity:

  1. On the instance of the Authoring service you use to import the catalog data, open the memory cache policy set file, for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\data\Environments\Plugin.Caching.PolicySet-1.0.0.json).

  2. Ensure that the Sitecore.Commerce.Plugin.Workflow.Workflow entity section has the parameter AllowCaching set to true as shown in the following example: 

    RequestResponse
    {
   "$type": "Sitecore.Commerce.Core.EntityCachingPolicy, Sitecore.Commerce.Core",
     "EntityFullName": "Sitecore.Commerce.Plugin.Workflow.Workflow",
     "AllowCaching": true,
     "Priority": "Normal",
     "Expiration": 86400000,
     "HasNegativeCaching": false,
     "CacheName": "Workflows",
     "EntityIdPrefix": "Entity-Workflow"
},
    Note

    Optionally, you can enable logging for collecting information while importing catalog data. Open the config.json file for the Commerce Engine service you use to import the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\config.json) and set the log level values for "Default", "Microsoft" and "System" logs to "Information".

    RequestResponse
    "Serilog": {
  "Using": [
    "Serilog.Sinks.Literate",
    "Serilog.Sinks.File",
    "Serilog.Sinks.ApplicationInsights"
  ],
  "MinimumLevel": {
    "Default": "Information",
    "Override": {
      "Microsoft": "Information",
      "System": "Information"
  }
},

Change the SQL timeout configuration

Change SQL timeout configuration to make sure the connection to the SQL database remains open and avoid to allow long processes to complete.

To change the SQL timeout configuration:

  1. On the instance of the Authoring service you use to import the catalog data, open the Plugin.SQL.PolicySet-1.0.0.json file (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\data\Environments\Plugin.SQL.PolicySet-1.0.0.json

  2. Set the value of the "ConnectTimeout" and "CommandTimeout" properties to "0".

    Note

    Take note of the current property values so that you can set them back to their original values after you have finished importing catalogs.

Bootstrap the Commerce Engine

After you made all your changes to the Commerce Engine configuration, bootstrap the Commerce Engine.

Increase the threshold of Sitecore.Diagnostics.MemoryMonitorHook

When importing large catalogs, you should increase the threshold value of the Sitecore.Diagnostics.MemoryMonitorHook hook to ensure that there is sufficient memory to allow the import operation to complete successfully.

To change the Sitecore.Diagnostics.MemoryMonitorHook threshold value:

  1. Open the wwwroot\<host>\App_Config\Sitecore.config file.

  2. In the "Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel" section, increase the "Threshold" value . The following example shows an example of default hooks configuration:

    RequestResponse
    <hooks>
    <hook type="Sitecore.Diagnostics.HealthMonitorHook, Sitecore.Kernel" />
    <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
      <param desc="Threshold">2GB</param>
      <param desc="Check interval">00:00:05</param>
      <param desc="Minimum time between log entries">00:01:00</param>
      <ClearCaches>false</ClearCaches>
      <GarbageCollect>false</GarbageCollect>
      <AdjustLoadFactor>false</AdjustLoadFactor>
    </hook>
  </hooks>

Import a Commerce catalog (for deployments not using containers)

Note

If your Commerce deployment uses containers, follow instructions in this procedure instead.

This task describes how to import a ZIP file containing catalog data (previously exported from a Sitecore XC environment), into another Sitecore XC environment. This procedure is based on the Postman sample provided with the Sitecore Commerce Engine Software Development Kit (SDK).

Note

If your ZIP file is larger than 1 GB, copy the file to your local machine (for example, to the wwwroot\data\Catalogs folder of your Commerce Engine service), and use the Import Local Catalogs request in Postman, as described below.

The following instructions assume that you have installed and set up the Postman application, imported the sample collections (from the Sitecore Commerce Engine SDK) into Postman, configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import catalog product data:

  1. Open Postman, and in the Collections pane, expand the CatalogAPISamples collection.

  2. Open the Catalog - API folder.  

    • If your ZIP file is smaller than 1 GB, click the Import Catalogs (with publishing) request.

      Note

      When you import with publishing, you bypass the publishing workflow. As a result, sellable items are imported and do not need to transition from the various status (for example, draft, waiting for approval, and so on) in order to be published to the content delivery server.

      (The Postman samples also include a request to import catalogs without publishing. In this case, imported sellable items are in the draft status and must then go though the publishing workflow.)

    • If your ZIP file is larger than 1 GB, click the Import Local Catalogs request.

  3. In the Environment field, specify the Commerce environment where you need to import data  (for example, the HabitatAuthoring environment).

  4. Click the Headers tab and, in the HTTP request, specify the following header key and value:

    • Key: PolicyKeys

    • Value: IgnoreIndexDeletedSitecoreItem|IgnoreAddEntityToIndexList|IgnoreIndexUpdatedSitecoreItem|IgnoreLocalizeEntity

  5. On the Body tab, specify the file you want to import. 

    • If you are executing the Import Catalogs (with publishing) request, in the importFile key value, click Select Files and browse to and click the ZIP file that contains the catalog data to import.

    • If you are executing the Import Local Catalogs request, in the "importFilePath" parameter, enter the name of the local ZIP file that contains the catalog data to import.

    The following table describes the parameters you can set for catalog data import:

    Parameter

    Value

    Description

    "mode"

    "replace"

    Deletes all existing catalogs, and adds the new catalog file to import.

    Caution

    The replace mode is the default value. This mode deletes all existing catalog data (even if the new catalogs have different names.) If you want to keep existing catalog data, use the add mode.

    "add"

    Adds the new catalog data file, and keeps all existing catalogs.

    Note

    If an existing catalog has the same name as the catalog file to import, the content of the existing catalog is replaced with the content of the new catalog file.

    "batchSize"

    900

    The number of items to process in a batch.

    Important

    Reduce the batch size to a smaller number if an SQL timeout error occurs.

    "errorThreshold"

    "10"

    Indicates the maximum number of errors allowed during the import operation. After this threshold is reached, the import operation fails. Default value is 10.

  6. To begin the catalog import operation, click Send. 

    You can use the DevOps API to check the status of long running import commands.

    Note

    If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

  7. To verify the status of the import operation, check the returned response code:

    • When the ResponseCode returns the value Failed, the number of errors encountered during the import operation exceeds the configured error threshold.

    • When the ResponseCode returns the value CompletedWithErrors, the number of errors encountered during the import operation is within the configured error threshold.

    • When the ResponseCode returns the value Succeeded, the operation completed without error.

  8. Repeat this procedure for each additional catalog .zip file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your catalogs (and there is no related inventory data to import), you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

  9. After you finish importing the catalog into your Sitecore deployment environment, you configure the catalogs in the Sitecore Content Editor to associate the catalog with your storefront site.

Import catalog data in the engine_authoring container

Note

This procedure only applies to Commerce deployment using containers. If your deployment does not use containers, follow instructions in this procedure instead.

This task describes how to import a ZIP file containing catalog data (previously exported from a Sitecore XC environment), into the engine_authoring container. This procedure is based on the Import Local Catalogs API request in the Postman samples provided with the Sitecore Commerce Engine Software Development Kit (SDK).

The following instructions assume that you have installed and set up the Postman application, imported the sample collections (from the Sitecore Commerce Engine SDK) into Postman, configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import catalog product data into the engine_authoring container:

  1. Place the catalog ZIP file to import in the following folder: C:\engine\catalogs\ .

  2. In the Collections pane, expand the CatalogAPISamples collection.

  3. Open the Catalog - API folder and click the Import Local Catalogs request.

  4. In the Environment field, specify the Commerce environment where you need to import data  (for example, the HabitatAuthoring environment).

  5. Click the Headers tab and, in the HTTP request, specify the following header key and value:

    • Key: PolicyKeys

    • Value: IgnoreIndexDeletedSitecoreItem|IgnoredEntityToIndexList|IgnoreIndexUpdatedSitecoreItem|IgnoreLocalizeEntity

  6. On the Body tab, specify parameter values for the catalog data import:

    The following table describes the parameters you can set for catalog data import:

    Parameter

    Value

    Description

    "importFilePath":

    none

    Relative path to the file to import: "..\catalogs\<filename>"

    "mode":

    "replace"

    Deletes all existing catalogs, and adds the new catalog file to import.

    Caution

    The replace mode is the default value. This mode deletes all existing catalog data (even if the new catalogs have different names.) If you want to keep existing catalog data, use the add mode.

    "add"

    Adds the new catalog data file, and keeps all existing catalogs.

    Note

    If an existing catalog has the same name as the catalog file to import, the content of the existing catalog is replaced with the content of the new catalog file.

    "batchSize":

    "900"

    Number of items to process in a batch.

    Important

    Reduce the batch size to a smaller number if an SQL timeout error occurs.

    "errorThreshold":

    "10"

    Indicates the maximum number of errors allowed during the import operation. After this threshold is reached, the import operation fails.

    "publish":

    false

    Specifies whether to import sellable items in the draft status or in the published status.

    When set to false (default), sellable items are imported in the draft status and must transition through the approval work flow before they can be published to the content delivery server.

    When set to true, sellable items are imported in the published status, and published to the content delivery server without going through the approval workflow.

  7. To begin the catalog import operation, click Send. 

    You can use the DevOps API to check the status of long running import commands.

    Note

    If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

  8. To verify the status of the import operation, check the returned response code:

    • When the ResponseCode returns the value Failed, the number of errors encountered during the operation exceeds the configured error threshold.

    • When the ResponseCode returns the value CompletedWithErrors, the number of errors encountered during the operation is within the configured error threshold.

    • When the ResponseCode returns the value Succeeded, the operation completed without error.

  9. Repeat this complete procedure for each additional catalog .zip file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your catalogs (and there is no related inventory data to import), you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

  10. After you finish importing the catalog into your Sitecore deployment environment, you configure the catalogs in the Sitecore Content Editor to associate the catalog with your storefront site.

Import product inventory data (for deployments not using containers)

Note

If your Commerce deployment uses containers, follow instructions in this procedure instead.

After you have finished importing catalog data files into your deployment, you can import related inventory data.

Note

If your ZIP file is larger than 1 GB, copy the file to your local machine (for example, to the wwwroot\data\Catalogs folder of your Commerce Engine service) and use the Import Local Catalogs request, as described in the procedure that follows.

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine software development kit (SDK), configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import catalog inventory data:

  1. If the Sitecore Identity token has expired, execute another GetToken request to get a new token.

  2. In the Collections pane, expand the InventoryAPISamples collection.

  3. Open the Inventory - API folder.

    • If your ZIP file is smaller than 1 GB, click the Import Inventory Sets request.

    • If your ZIP file is larger than 1 GB, click the Import Local Inventory Sets request.

  4. In the Environment field, specify the Commerce environment where you need to import data (for example, the HabitatAuthoring environment).

  5. On the Body tab, specify the file you want to import.

    • If you are executing the Import Inventory Sets request, in the importFile key value, click Select Files and browse to and click the ZIP file that contains the inventory data to import.

    • If you are executing the Import Local Inventory Sets request, in the "importFilePath"parameter, enter the name of the local ZIP file that contains the inventory data to import.

  6. Set the import operation parameters as required:

    Parameter

    Value

    Description

    "importFilePath":

    "<relative path>"

    Relative path to the inventory data file to import, for example "..\catalogs\<filename>"

    "mode":

    "replace"

    Deletes all existing inventory data, and adds the new inventory file data.

    Caution

    The replace mode is the default value. This mode deletes all existing data (even if the new data set have different names.)

    If you want to keep existing data, use the add mode.

    "add"

    Adds the new data file, and keeps all existing inventory data.

    Note

    If an existing inventory data set has the same name as the data set file to import, the content of the existing data set is replaced with the content of the new data file.

    "batchSize":

    "900"

    Number of items to process per batch.

    Important

    Reduce the batch size to a smaller number if an SQL timeout error occurs.

    "errorThreshold":

    "10"

    Indicates the maximum number of errors allowed during the import operation. After this threshold is reached, the import operation fails.

  7. To begin the import operation, click Send.

  8. To verify the status of the import operation, check the returned response code:

    • When the ResponseCode returns the value Failed, the number of errors encountered during the operation exceeds the configured error threshold.

    • When the ResponseCod returns the value CompletedWithErrors, the number of errors encountered during the operation is within the configured error threshold.

    • When the ResponseCode returns the value Succeeded, the operation completed without error.

  9. Repeat this procedure for each additional product inventory ZIP file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your inventory sets, you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

  10. If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

Import product inventory data in the engine_authoring container

Note

This procedure only applies to Commerce deployment using containers. If your deployment does not uses containers, follow the instructions in this procedure instead.

After you have finished importing catalog data files into your containerized Commerce deployment, you can import related inventory data into the engine_authoring container.

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine software development kit (SDK), configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import product inventory data:

  1. Place the ZIP file containing inventory data to import in the following folder: C:\engine\catalogs\.

  2. If the Sitecore Identity token has expired, execute another GetToken request to get a new token.

  3. In the Collections pane, expand the InventoryAPISamples collection.

  4. Open the Inventory - API folder, and click the Import Local Inventory Sets request.

  5. In the Environment field, specify the Commerce environment where you need to import data (for example, the HabitatAuthoring environment).

  6. On the Body tab, specify parameter values for the inventory data import:

    Parameter

    Value

    Description

    "importFilePath":

    "<relative path>"

    Relative path to the inventory data file to import, for example "..\catalogs\<filename>"

    "mode":

    "replace"

    Deletes all existing inventory data, and adds the new inventory file data.

    Caution

    The replace mode is the default value. This mode deletes all existing data (even if the new data set have different names.)

    If you want to keep existing data, use the add mode.

    "add"

    Adds the new data file, and keeps all existing inventory data.

    Note

    If an existing inventory data set has the same name as the data set file to import, the content of the existing data set is replaced with the content of the new data file.

    "batchSize":

    "900"

    Number of items to process per batch.

    Important

    Reduce the batch size to a smaller number if an SQL timeout error occurs.

    "errorThreshold":

    "10"

    Indicates the maximum number of errors allowed during the import operation. After this threshold is reached, the import operation fails.

  7. To begin the import operation, click Send.

  8. To verify the status of the import operation, check the returned response code:

    • When the ResponseCode returns the value Failed, the number of errors encountered during the operation exceeds the configured error threshold.

    • When the ResponseCode returns the value CompletedWithErrors, the number of errors encountered during the operation is within the configured error threshold.

    • When the ResponseCode returns the value Succeeded, the operation completed without error.

  9. Repeat this procedure for each additional product inventory ZIP file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your inventory sets, you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

  10. If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information"  back to "Warning").

Index Commerce Engine catalog items

After you have completed the catalog and inventory data import, you must perform a full indexing of catalog items by manually initiating the Commerce Engine full index minion. This procedure uses the Postman samples provided in the Sitecore Commerce Engine SDK.

To invoke the full index minion: 

  1. In the Postman Collections pane, expand the SitecoreCommerce_DevOps collection.

  2. Open the Minions folder, and click the Run FullIndex Minion - Catalogs items request. 

  3. In the Body pane, set the environment to the Authoring Environment (for example, "environmentName":"{{AuthoringEnvironment}}").

  4. To execute the request, click Send.

    Note

    After you have performed a full indexing of catalog items, you must manually rebuild Sitecore XP indexes.

Do you have some feedback for us?

If you have suggestions for improving this article,