Walkthrough: Exporting catalog and inventory data

Current version: 10.2

This topic describes how to export catalog and inventory data from a Sitecore XC environment. 

Sitecore XC provides an application programming interface (API) that you use to export catalog data and inventory sets from a Commerce environment. For example, you can export a Commerce catalog from a development environment and then import it into a different environment, such as a test or a staging environment. 

See the Generated data structure section for details on the data structure of the exported data.

To export catalog and inventory data:

Before you begin exporting catalog data, you must modify the Internet Information Services (IIS) settings (in the IIS Manager and in the web.config file) to reduce the risk of timeouts from occurring during the data export process. This is important when exporting a very large catalog data set.

To change the IIS configuration and related settings in the web.config file: 

  1. Start  Internet Information Services (IIS) Manager and, in the navigation tree, click Application Pools.

  2. On the Application Pools page, click the name of the relevant application pool (for example, CommerceAuthoring_Sc). 

  3. In the Actions pane, in the Edit Application Pool section, click Advanced Settings and, in the Advance Settings dialog box, set the following parameter values to 0 (zero):

    Section

    Parameter

    Value

    CPU

      Limit Interval (minutes) 

    0

    Process Model

     Idle Time-out (minutes)

    0

    Recycling

    Regular Time Interval (minutes)

    0

  4. Click OK.

  5. Open the web.config  file for the service you use to export the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\web.config) and modify the settings marked in bold in the following configuration sample:

    Note

    Before you change the IIS settings as shown in the following example, take note of their original values so that you know what to set them back to after you have finished exporting catalogs.

    RequestResponse
    <?xml version="1.0" encoding="utf-8"?>
    <configuration>
      <system.webServer>
        <security>
          <requestFiltering>
            <!-- For Microsoft IIS (Internet Information Services), maxAllowedContentLength specifies the maximum length of content in a request, in bytes. -->
            <requestLimits maxAllowedContentLength="2147483648" />
          </requestFiltering>
        </security>
        <handlers>
          <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
        </handlers>
        <modules runAllManagedModulesForAllRequests="false">
          <remove name="WebDAVModule" />
        </modules>
        <aspNetCore processPath=".\Sitecore.Commerce.Engine.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" requestTimeout="00:00:00"  stdoutLogFile=".\logs\stdout" />
      </system.webServer>
    </configuration>
    Note

    Optionally, you can enable logging for collecting information while the catalog export process is running.  To do this, open the config.json file for the service you use to export the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\config.json) and set the log level values to "Information".

    RequestResponse
    "Serilog": {
      "Using": [
        "Serilog.Sinks.Literate",
        "Serilog.Sinks.File",
        "Serilog.Sinks.ApplicationInsights"
      ],
      "MinimumLevel": {
        "Default": "Information",
        "Override": {
          "Microsoft": "Information",
          "System": "Information"
       }
    },
  6. Reset the Internet Information Services.

Change the SQL timeout configuration

Change the SQL timeout configuration to make sure the connection to the SQL database remains open and avoid to allow long processes to complete.

To change the SQL timeout configuration:

  1. On the instance of the Authoring service you use to import the catalog data, open the Plugin.SQL.PolicySet-1.0.0.json file (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\data\Environments\Plugin.SQL.PolicySet-1.0.0.json

  2. Set the value of the "ConnectTimeout" and "CommandTimeout" properties to "0".

    Note

    Take note of the current property values so that you can set them back to their original values after you have finished importing catalogs.

Bootstrap the Commerce Engine

After you made your changes to the Commerce Engine configuration, bootstrap the Commerce Engine.

Export a Commerce catalog data set

This example is based on the Postman sample provided with the Sitecore Commerce Engine SDK. 

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine SDK, configured your environment, and retrieved a bearer token to access the Commerce Engine API.

Note

As an alternative to using Postman to execute the export catalog API request, this example shows how to use a cURL command in a Bash script.

To export your Commerce XC catalog data set using Postman:

  1. In the Postman Collections pane, expand the CatalogAPISamples collection.

  2. Open the Catalog - API folder and select the Export Catalogs request ({{ServiceHost}}/{{ShopsApi}}/ExportCatalogs()).

  3. In the Environment field, specify the Commerce environment that contains the catalog data you need to export.  For example, the HabitatAuthoring environment.

  4. On the Headers tab, provide the following required key values for your deployment:

    Header key

    Value

    ShopName

    {{ShopName}}

    ShopperId

    {{ShopperId}}

    Language

    {{Language}}

    Currency

    {{Currency}}

    Environment

    {{Environment}}

    GeoLocation

    {{GeoLocation}}

    CustomerId

    {{CustomerId}}

    Authorization

    {{SitecoreIdToken}}

  5. On the Body tab, modify the export settings based on your specific requirements. The following shows the default settings:

    RequestResponse
    {
    "fileName":"ExportCatalogs.zip",
    "mode":"full",
    "maximumItemsPerFile":"200",
    "excludeInventory":false
    }

    When the excludeInventory parameter is set to true, the ExportCatalog call does not include inventory information.

  6. To begin the catalog export operation, click Send

  7. Repeat this procedure for each additional catalog that you want to export from your Sitecore Commerce deployment.

    Note

    If you have configured logging to collect information during the export operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

    Caution

    After you have finished exporting all your catalogs (and there is no related inventory data to export), you must set the settings you previously changed in IIS Manager and in the web.config file back to their original values. 

    Otherwise, proceed with exporting inventory data.

Example of a catalog data export using cURL in a Bash script

The following example shows how to export a catalog data set using a cURL command in a Bash script as an alternative to using the Postman tool to send the HTTP request.

Note

Before you execute your script, you must:

  • Update the value <YOUR HOSTNAME> with the name of the Commerce Engine instance running the Authoring service in your deployment (the default value is localhost:5000).

  • Update the value <YOUR TOKEN> in the Authorization request header with a valid token value.

  • Update the Environment header key with the name of the Commerce environment that contains the catalog data you need to export. By default, the script specifies the HabitatAuthoring environment.

  • Set file permissions by running the chmod +x <your_base_script>.sh.

  • Modify the script to specify your machine name and any other information specific to your deployment environment.

RequestResponse
#!/bin/bash

# Git Bash:
# $ chmod +x WebRequest_ExportCatalogs.sh
# $ ./WebRequest_ExportCatalogs.sh

curl --request POST \
  --url 'https://<YOUR HOSTNAME>/api/ExportCatalogs%28%29' \
  --output ExportCatalogs.zip \
  --insecure \
  --header 'Authorization: Bearer <YOUR TOKEN>' \
  --header 'cache-control: no-cache' \
  --header 'Content-Type: application/json' \
  --header 'Currency: USD' \
  --header 'CustomerId: CustomerId' \
  --header 'Environment: HabitatAuthoring' \
  --header 'GeoLocation: IpAddress=1.0.0.0' \
  --header 'Language: en-US' \
  --header 'ShopName: CommerceEngineDefaultStorefront' \
  --header 'ShopperId: ShopperId' \
  --data '{
    "fileName":"ExportCatalogs.zip",
    "mode":"full",
    "maximumItemsPerFile":"200"
}'

Export a Commerce inventory set

After you have exported your catalog, you can export the associated inventory sets, if you do not have inventory data to export.

This procedure uses the Postman samples provided in the Sitecore Commerce Engine SDK. 

Note

As an alternative to using Postman to execute the export inventory API request, this example shows how to use a cURL command in a Bash script.

To export inventory sets data:

  1. If the Sitecore Identity token has expired, execute another GetToken request to get a new token.

  2. In the Collections pane, expand the InventoryAPISamples collection.

  3. Open the Inventory - API folder, and select the Export Inventory Sets request.

  4. In the Environment field, specify the environment that contains the data that you want to export (for example, the HabitatAuthoring environment).

  5. On the Headers tab, provide the following required key values for the deployment:

    Header key

    Value

    ShopName

    {{ShopName}}

    ShopperId

    {{ShopperId}}

    Language

    {{Language}}

    Currency

    {{Currency}}

    Environment

    {{Environment}}

    GeoLocation

    {{GeoLocation}}

    CustomerId

    {{CustomerId}}

    Authorization

    {{SitecoreIdToken}}

  6. To begin exporting the inventory sets, click Send

  7. Repeat this procedure for each additional product inventory set that you want to export into your Sitecore Commerce deployment.

    Caution

    After you have finished exporting all your catalogs, you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

    Note

    If you have configured logging to collect information during the export operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

Example of an inventory data export using cURL in a Bash script

The following shows an example of how to export an inventory set using a cURL command in a Bash script as an alternative to using the Postman tool to send the HTTP request.

Note

Before you execute your script, you must:

  • Update the value <YOUR HOSTNAME> with the name of the Commerce Engine instance running the Authoring service in your deployment (the default value is localhost:5000).

  • Update the value <YOUR TOKEN> in the Authorization request header with a valid token value.

  • Update the Environment header key with the name of the Commerce environment that contains the data that you need to export. By default, the script specifies the HabitatAuthoring environment.

  • Set the file permissions by running the command chmod +x <your_base_script>.sh.

  • Modify the script to specify your machine name and environment, or any other information specific to your deployment.

RequestResponse
#!/bin/bash

# Git Bash:
# $ chmod +x WebRequest_ExportInventorySets.sh
# $ ./WebRequest_ExportInventorySets.sh

curl --request PUT \
  curl --request POST \
  --url 'https://<YOUR HOSTNAME>/api/ExportCatalogs%28%29' \
  --output ExportCatalogs.zip \
  --insecure \
  --header 'Authorization: Bearer <YOUR TOKEN>' \
  --header 'cache-control: no-cache' \
  --header 'Content-Type: application/json' \
  --header 'Currency: USD' \
  --header 'CustomerId: CustomerId' \
  --header 'Environment: HabitatAuthoring' \
  --header 'GeoLocation: IpAddress=1.0.0.0' \
  --header 'Language: en-US' \
  --header 'ShopName: CommerceEngineDefaultStorefront' \
  --header 'ShopperId: ShopperId' \
  --data '{
    "fileName":"ExportInventorySets.zip",
    "mode":"full",
    "maximumItemsPerFile":"200"
}'

Generated data structure

When you export catalog or inventory data, the operation generates a ZIP file that contains the data in a specific format. The system expects this same data structure when you import data back into the system.

The ZIP file contains a number of JSON files that define the exported Commerce entities, represented as JSON objects. Each file name identifies the entity type. For example, a file named Category.1.json contains the first page of a list of category entities, in JSON-serialized format.

Note

The catalog export feature only supports full catalog data exports. Catalog data is exported into a .zip file that you can import into the system later. Because all items in a file are imported in parallel, smaller numbers of Commerce entities are batched together in each file, for improved performance.

The ZIP file also includes a Relationships folder that contains a set of JSON files that define the relationships between the entities in the root-level JSON files. The file name indicates the relationship type and the source entity, and the file itself contains a list of JSON objects. Each JSON object defines the source entity, the target entity, and the type of relationship between them.

For example, a file named CatalogToCategory.Habitat_Master.1.json contains information about the relationships between the Habitat_Master catalog and the categories within the catalog:

RequestResponse
[{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-Assortments",
    "RelationshipType": "CatalogToCategory"
},
{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-Departments",
    "RelationshipType": "CatalogToCategory"
},
{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-FeaturedProducts",
    "RelationshipType": "CatalogToCategory"
}]

Do you have some feedback for us?

If you have suggestions for improving this article,