Walkthrough: Exporting catalog and inventory data

Before you begin exporting catalog data, you must modify the Internet Information Services (IIS) settings (in the IIS Manager and in the web.config file) to reduce the risk of timeouts from occurring during the data export process. This is important when exporting a very large catalog data set.

To change the IIS configuration and related settings in the web.config file: 

1. Start  Internet Information Services (IIS) Manager and, in the navigation tree, click Application Pools.

2. On the Application Pools page, click the name of the relevant application pool (for example, CommerceAuthoring_Sc). 

3. In the Actions pane, in the Edit Application Pool section, click Advanced Settings and, in the Advance Settings dialog box, set the following parameter values to 0 (zero):

   Section	Parameter	Value
   CPU	Limit Interval (minutes)	0
   Process Model	Idle Time-out (minutes)	0
   Recycling	Regular Time Interval (minutes)	0

4. Click OK.

5. Open the web.config  file for the service you use to export the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\web.config) and modify the settings marked in bold in the following configuration sample:

   Note

   Before you change the IIS settings as shown in the following example, take note of their original values so that you know what to set them back to after you have finished exporting catalogs.

   <?xml version="1.0" encoding="utf-8"?>
   <configuration>
     <system.webServer>
       <security>
         <requestFiltering>
           <!-- For Microsoft IIS (Internet Information Services), maxAllowedContentLength specifies the maximum length of content in a request, in bytes. -->
           <requestLimits maxAllowedContentLength="2147483648" />
         </requestFiltering>
       </security>
       <handlers>
         <add name="aspNetCore" path="*" verb="*" modules="AspNetCoreModule" resourceType="Unspecified" />
       </handlers>
       <modules runAllManagedModulesForAllRequests="false">
         <remove name="WebDAVModule" />
       </modules>
       <aspNetCore processPath=".\Sitecore.Commerce.Engine.exe" arguments="" forwardWindowsAuthToken="false" stdoutLogEnabled="false" requestTimeout="`00:00`:00"  stdoutLogFile=".\logs\stdout" />
     </system.webServer>
   </configuration>

   Note

   Optionally, you can enable logging for collecting information while the catalog export process is running.  To do this, open the config.json file for the service you use to export the catalog data (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\config.json) and set the log level values to "Information".

   "Serilog": { "Using": [ "Serilog.Sinks.Literate", "Serilog.Sinks.File", "Serilog.Sinks.ApplicationInsights" ], "MinimumLevel": { "Default": "Information", "Override": { "Microsoft": "Information", "System": "Information" } },

6. Reset the Internet Information Services.

Change the SQL timeout configuration to make sure the connection to the SQL database remains open and avoid to allow long processes to complete.

To change the SQL timeout configuration:

1. On the instance of the Authoring service you use to import the catalog data, open the Plugin.SQL.PolicySet-1.0.0.json file (for example, C:\inetpub\wwwroot\CommerceAuthoring_Sc\wwwroot\data\Environments\Plugin.SQL.PolicySet-1.0.0.json

2. Set the value of the "ConnectTimeout" and "CommandTimeout" properties to "0".

   Note

   Take note of the current property values so that you can set them back to their original values after you have finished importing catalogs.

After you made your changes to the Commerce Engine configuration, bootstrap the Commerce Engine.

This example is based on the Postman sample provided with the Sitecore Commerce Engine SDK. 

The following instructions assume that you have installed and set up the Postman application, imported the sample collections from the Sitecore Commerce Engine SDK, configured your environment, and retrieved a bearer token to access the Commerce Engine API.

To export your Commerce XC catalog data set using Postman:

1. In the Postman Collections pane, expand the CatalogAPISamples collection.

2. Open the Catalog - API folder and select the Export Catalogs request ({{ServiceHost}}/{{ShopsApi}}/ExportCatalogs()).

3. In the Environment field, specify the Commerce environment that contains the catalog data you need to export.  For example, the HabitatAuthoring environment.

4. On the Headers tab, provide the following required key values for your deployment:

   Header key	Value
   ShopName	{{ShopName}}
   ShopperId	{{ShopperId}}
   Language	{{Language}}
   Currency	{{Currency}}
   Environment	{{Environment}}
   GeoLocation	{{GeoLocation}}
   CustomerId	{{CustomerId}}
   Authorization	{{SitecoreIdToken}}

5. On the Body tab, modify the export settings based on your specific requirements. The following shows the default settings:

   {
   "fileName":"ExportCatalogs.zip",
   "mode":"full",
   "maximumItemsPerFile":"200",
   "excludeInventory":false
   }

   When the excludeInventory parameter is set to true, the ExportCatalog call does not include inventory information.

6. To begin the catalog export operation, click Send. 

7. Repeat this procedure for each additional catalog that you want to export from your Sitecore Commerce deployment.

   Note

   If you have configured logging to collect information during the export operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

   Caution

   After you have finished exporting all your catalogs (and there is no related inventory data to export), you must set the settings you previously changed in IIS Manager and in the web.config file back to their original values. 

   Otherwise, proceed with exporting inventory data.

#!/bin/bash

# Git Bash:
# $ chmod +x WebRequest_ExportCatalogs.sh
# $ ./WebRequest_ExportCatalogs.sh

curl --request POST \
  --url 'https://<YOUR HOSTNAME>/api/ExportCatalogs%28%29' \
  --output ExportCatalogs.zip \
  --insecure \
  --header 'Authorization: Bearer <YOUR TOKEN>' \
  --header 'cache-control: no-cache' \
  --header 'Content-Type: application/json' \
  --header 'Currency: USD' \
  --header 'CustomerId: CustomerId' \
  --header 'Environment: HabitatAuthoring' \
  --header 'GeoLocation: IpAddress=1.0.0.0' \
  --header 'Language: en-US' \
  --header 'ShopName: CommerceEngineDefaultStorefront' \
  --header 'ShopperId: ShopperId' \
  --data '{
    "fileName":"ExportCatalogs.zip",
    "mode":"full",
    "maximumItemsPerFile":"200"
}'

After you have exported your catalog, you can export the associated inventory sets, if you do not have inventory data to export.

This procedure uses the Postman samples provided in the Sitecore Commerce Engine SDK. 

To export inventory sets data:

1. If the Sitecore Identity token has expired, execute another GetToken request to get a new token.

2. In the Collections pane, expand the InventoryAPISamples collection.

3. Open the Inventory - API folder, and select the Export Inventory Sets request.

4. In the Environment field, specify the environment that contains the data that you want to export (for example, the HabitatAuthoring environment).

5. On the Headers tab, provide the following required key values for the deployment:

   Header key	Value
   ShopName	{{ShopName}}
   ShopperId	{{ShopperId}}
   Language	{{Language}}
   Currency	{{Currency}}
   Environment	{{Environment}}
   GeoLocation	{{GeoLocation}}
   CustomerId	{{CustomerId}}
   Authorization	{{SitecoreIdToken}}

6. To begin exporting the inventory sets, click Send. 

7. Repeat this procedure for each additional product inventory set that you want to export into your Sitecore Commerce deployment.

   Caution

   After you have finished exporting all your catalogs, you must set the settings you previously changed in the IIS Manager and in the web.config file back to their original values. 

   Note

   If you have configured logging to collect information during the export operation, set the logging levels back to their original value (for example, from "Information" back to "Warning").

#!/bin/bash

# Git Bash:
# $ chmod +x WebRequest_ExportInventorySets.sh
# $ ./WebRequest_ExportInventorySets.sh

curl --request PUT \
  curl --request POST \
  --url 'https://<YOUR HOSTNAME>/api/ExportCatalogs%28%29' \
  --output ExportCatalogs.zip \
  --insecure \
  --header 'Authorization: Bearer <YOUR TOKEN>' \
  --header 'cache-control: no-cache' \
  --header 'Content-Type: application/json' \
  --header 'Currency: USD' \
  --header 'CustomerId: CustomerId' \
  --header 'Environment: HabitatAuthoring' \
  --header 'GeoLocation: IpAddress=1.0.0.0' \
  --header 'Language: en-US' \
  --header 'ShopName: CommerceEngineDefaultStorefront' \
  --header 'ShopperId: ShopperId' \
  --data '{
    "fileName":"ExportInventorySets.zip",
    "mode":"full",
    "maximumItemsPerFile":"200"
}'

[{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-Assortments",
    "RelationshipType": "CatalogToCategory"
},
{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-Departments",
    "RelationshipType": "CatalogToCategory"
},
{
    $type : Sitecore.Commerce.Plugin.Catalog.RelationshipArgument, Sitecore.Commerce.Plugin.Catalog
    "SourceName": "Entity-Catalog-Habitat_Master",
    "TargetName": "Entity-Category-Habitat_Master-FeaturedProducts",
    "RelationshipType": "CatalogToCategory"
}]