1. Administration

Walkthrough: Importing catalog data

Version:

This walkthrough describes how to import a ZIP file containing catalog data (previously exported from a Sitecore XC environment), into another Sitecore XC environment. For details on the expected data structure for the import operation, see the Walkthrough: Exporting catalog and inventory data topic.

Important

You can only import catalog data into an environment that contains the same Storefront site as the one from which you previously exported it. If you attempt to import the catalog data into an environment that contains a different Storefront site, the import operation fails with an error indicating that the catalog is already associated to another site.

Note

Sitecore XC currently supports full catalog imports only.

During standard operations, the Commerce Engine Search service uses the incremental index minion as an automated process that periodically adds new catalog items to an indexing list. This automated process ensures the discoverability of new catalog items. To help optimize performance during catalog import, the Catalog application programming interface (API) uses policy keys that prevent incremental indexing during the catalog data import process. This reduces the total time required to import catalog data.  

After the catalog data is imported, you can proceed with importing any related inventory data set (if required).  Finally, you must rebuild catalog indexes manually at the Commerce Engine level and at the Sitecore platform level.

To import catalog data and inventory sets (if required), and rebuild indexes, perform the following procedures:

Before you begin importing catalog data, you must modify the Internet Information Services (IIS) settings (in the IIS Manager and in the web.config file) to reduce the risk of timeouts occurring during the data export process. This is important when exporting a very large catalog data set.

To reconfigure the service used to import catalog data:

  1. Start the Internet Information Services (IIS) Manager and, in the navigation tree, click Application Pools.

  2. On the Application Pools page, click the name of the relevant application pool (for example, CommerceAuthoring_Sc9). 

  3. In the Actions pane, in the Edit Application Pool section, click Advanced Settings and, in the Advance Settings dialog box, set the following values to 0:

    Section

    Parameter

    Value

    CPU

      Limit Interval (minutes) 

    0

    Process Model

     Idle Time-out (minutes)

    0

    Recycling

    Regular Time Interval (minutes)

    0

  4. Click OK.

  5. Open the web.config file for the service you use to import the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc9\web.config), and modify the settings marked in bold in the following configuration sample: 

    Note

    Before you change the IIS settings, take note of their original values so that you know what to set them back to after you have finished importing catalogs.

    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
      <system.webServer>
        <security>
          <requestFiltering>
            <!-- For Microsoft IIS (Internet Information Services), maxAllowedContentLength specifies the maximum length of content in a request, in bytes. -->
            <requestLimits maxAllowedContentLength="2147483648" />
          </requestFiltering>
        </security>
        <handlers>
          <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
        </handlers>
        <modules runAllManagedModulesForAllRequests="false">
          <remove name="WebDAVModule" />
        </modules>
        <aspNetCore processPath=".\Sitecore.Commerce.Engine.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" requestTimeout="00:00:00"stdoutLogFile=".\logs\stdout" />
      </system.webServer>
    </configuration>
  6. In the memory cache plugin file for the service you use to import the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc9\wwwroot\data\EnvironmentsPlugin.AuthoringMemoryCache.PolicySet-1.0.0.json) ensure that the Sitecore.Commerce.Plugin.Workflow.Workflow entity section has the parameter AllowCaching set to true

    If not, change the value to true and then bootstrap the Commerce Engine to persist the change.

    Note

    Optionally, you can enable logging for collecting information while importing catalog data. Open the config.json file for the Commerce Engine service you use to import the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc9\wwwroot\config.json) and set the log level values for "Default""Microsoft" and "System" logs to "Information".

    "Serilog":  {
       "Using": [
          "Serilog.Sinks.Literate",
          "Serilog.Sinks.File"
       ]
       "MinimumLevel": {
         "Default": "Information",
         "Override": {
           "Microsoft": "Information"
           "System": "Information",
       }
    }
  7. Reset the Internet Information Services.

Import a Commerce catalog

This task describes how to import a ZIP file containing catalog data (previously exported from a Sitecore XC environment), into another Sitecore XC environment. This procedure is based on the Postman sample provided with the Sitecore Commerce Engine Software Development Kit (SDK).

Note

If your ZIP file is larger than 1 GB, copy the file to your local machine (for example, to the wwwroot\data\Catalogs folder of your Commerce Engine service) and use the Import Local Catalogs request, as described below.

For Azure deployments, upload the ZIP file to your Azure web app service using your FTP/S deployment credentials.

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine SDK, configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import catalog product data:

  1. In the Collections pane, expand the CatalogAPISamples collection.

  2. Open the Catalog - API folder.  

    • If your ZIP file is smaller than 1 GB, click the Import Catalogs (with publishing) request.

    • If your ZIP file is larger than 1 GB, click the Import Local Catalogs request.

    Note

    When you import with publishing, you by-pass the publishing workflow. As a result, sellable items are imported and do not need to transition from the various status (for example, draft, waiting for approval, and so on) in order to be published to the content delivery server.

    (The Postman samples also include a request to import catalogs without publishing. In this case, imported sellable items are in the draft status and must then go though the publishing workflow.)

  3. In the Environment field, specify the Commerce environment where you need to import data  (for example, the HabitatAuthoring environment).

  4. Click the Headers tab and, in the HTTP request, specify the following header key and value:

    • Key: PolicyKeys

    • Value: IgnoreIndexDeletedSitecoreItem|IgnoreAddEntityToIndexList|IgnoreIndexUpdatedSitecoreItem|IgnoreLocalizeEntity

  5. On the Body tab, specify the file you want to import. 

    • If you are executing the Import Catalogs (with publishing) request, in the importFile key value, click Choose Files and browse to and click the ZIP file that contains the catalog data to import.

    • If you are executing the Import Local Catalogs request, in the "importFilePath" parameter, specify the name of the local ZIP file that contains the catalog data to import. For Azure deployments, in the "batchSize" parameter, set the value to "-1" (to use the batch size in the ZIP file).

    The following table describes the parameters for catalog data import:

    Key

    Value

    Description

    mode

    replace

    Deletes all existing catalogs, and adds the new catalog file to import.

    Caution

    The replace mode is the default value. This mode deletes all existing catalog data (even if the new catalogs have different names.) If you want to keep existing catalog data, use the add mode.

    add

    Adds the new catalog data file, and keeps all existing catalogs.

    Note

    If an existing catalog has the same name as the catalog file to import, the content of the existing catalog is replaced with the content of the new catalog file.

    errorThreshold

    <int>

    Indicates the maximum number of errors allowed during the import operation. After this threshold is reached, the import operation fails.

  6. To begin the catalog import operation, click Send

    You can use the DevOps API to check the status of long running import commands.

    Note

    If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

  7. Repeat this procedure for each additional catalog .zip file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your catalogs (and there is no related inventory data to import), you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

Import product inventory data

After you have finished importing catalog data files into your deployment, you can import related inventory data.

Note

If your ZIP file is larger than 1 GB, copy the file to your local machine (for example, to the wwwroot\data\Catalogs folder of your Commerce Engine service) and use the Import Local Catalogs request, as described below.

For Azure deployments, upload the ZIP file to your Azure web app service using your FTP/S deployment credentials.

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine software development kit (SDK), configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To import catalog inventory data:

  1. If the Sitecore Identity token has expired, execute another GetToken request to get a new token.

  2. In the Collections pane, expand the InventoryAPISamples collection.

  3. Open the Inventory - API folder.

    • If your ZIP file is smaller than 1 GB, click the Import Inventory Sets request.

    • If your ZIP file is larger than 1 GB, click the Import Local Inventory Sets request.

  4. In the Environment field, specify the Commerce environment where you need to import data (for example, the HabitatAuthoring environment).

  5. On the Body tab, specify the file you want to import.

    • If you are executing the Import Inventory Sets request, in the importFile key value, click Choose Files and browse to and click the ZIP file that contains the inventory data to import.

    • If you are executing the Import Local Inventory Sets request, in the "importFilePath" parameter, specify the name of the local ZIP file that contains the inventory data to import. For Azure deployments, in the "batchSize" parameter, set the value to "-1" (to use the batch size in the ZIP file).

  6. To begin the import operation, click Send.

  7. Repeat this procedure for each additional product inventory ZIP file that you want to import into your Sitecore Commerce deployment.

    Caution

    After you have finished importing all your inventory sets, you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

    Note

    If you have configured logging to collect information during the import operation, set the logging levels back to their original value (for example, from "Information"  back to "Warning").

Index Commerce Engine catalog items

After you have completed the catalog and inventory data import, you must perform a full indexing of catalog items by manually initiating the Commerce Engine full index minion. This procedure uses the Postman samples provided in the Sitecore Commerce Engine SDK.

To invoke the full index minion: 

  1. In the Postman Collections pane, expand the SitecoreCommerce_DevOps collection.

  2. Open the Minions folder and click the Run FullIndex Minion - Catalogs items request. 

  3. In the Body pane, set the environment to the Authoring Environment (for example, "environmentName":"{{AuthoringEnvironment}}").

  4. To execute the request, click Send.

    Note

    After you have performed a full indexing of catalog items, you must manually rebuild Sitecore XP indexes.

If you have suggestions for improving this article, let us know!