Configuring a storage provider

Version: 10.4

A storage provider lets you export data files to a location that you specify in a storage profile configuration. You can define multiple profiles for each provider. When you register a new data export task, you specify which profile to use for the task.

By default, the xConnect Data Export Tool includes the following storage providers:

Network folder
Azure Blob Storage
Azure File Storage

You can add your own custom providers to, for example, store data files on a NAS or a cloud provider other than Azure.

To make changes to a provider or profile or define a new profile for, for example, the xp1 topology, you must update all the following configuration files:

Container/pod	Configuration file path
Xdbcollection	Not applicable. Xdbcollection is configured in a global file, for example, `xconnect-xp0.json`, which results in the storage provider configuration being applied on the xConnect Processing role.
Cortexprocessing	`C:\inetpub\wwwroot\App_data\config\sitecore\Processing\sc.XConnect.DataTools.Workers.xml`
CortexprocessingWorker	`C:\service\App_Data\Config\Sitecore\Processing\sc.XConnect.DataTools.Workers.xml`

You must configure the following:

Connection string (for Azure providers)
Storage provider retry settings
Storage profile name, connection name, and location

Configure a connection string for Azure

For the Azure Blob Storage and Azure File Storage providers, you must specify a connection string for Azure.

To configure a connection string: do this:

Construct the connection string. For example:

RequestResponse

<add name="xconnect.dataexport.storage.account" connectionString="DefaultEndpointsProtocol=https;AccountName=teststorageaccount;AccountKey=QakK0uiWK3Tp75rOpngXwQ0pNtJTzsQGQSQVVVRloFpEiKYyPapiPGNb0yM5WgOC8+LMijRbKMzp4H+crTLcfA==;EndpointSuffix=core.windows.net" />

In the following configuration files, add the connection string.

Container/pod	Connection string configuration file
Cortexprocessing	`C:\inetpub\wwwroot\app_config\ConnectionStrings.config`
CortexprocessingWorker	`C:\service\App_config\ConnectionStrings.config`

In the sc.XConnect.DataTools.Workers.xml configuration files, in the profile nodes, in the StorageAccountConnectionStringName parameter, specify the connection string name, for example xconnect.dataexport.storage.account.

Configure a storage provider

For each storage provider, you can configure exponential retry settings to make the provider more resilient to network issues.

You can use the following retry options:

Max Retries – the maximum number of retry attempts before the provider gives up.
Delay In Seconds – the interval on which to base retry delays. Every time the provider tries to complete the process the interval between tries is augmented by the delay. If you set the delay to, for example, 5 seconds, if the provider is interrupted it waits 5 seconds before trying again. If it is interrupted again, it waits 5 + 5 = 10 seconds, and so on. The delay cannot be greater than the value in Max Delay In Seconds.
Max Delay In Seconds – the maximum permissible delay between retry attempts.

For example, for the xp1 topology, you specify the options in the sc.XConnect.DataTools.Workers.xml files, in the following nodes:

Storage provider	Provider node
Network folder	`Sitecore/Processing/Services/StorageProviders.NetworkFolderProvider`
Azure Blob Storage	`Sitecore/Processing/Services/StorageProviders.AzureBlobStorageProvider`
Azure File Storage	`Sitecore/Processing/Services/StorageProviders.AzureFileStorageProvider`

In this example, the default storage provider configuration for Azure Blob Storage is as follows:

RequestResponse

<StorageProviders.AzureBlobStorageProvider>
    <Type>Sitecore.XConnect.DataTools.CortexWorkers.DataExport.StorageProviders.AzureBlobStorageProvider, Sitecore.XConnect.DataTools.CortexWorkers</Type>
    <As>Sitecore.XConnect.DataTools.CortexWorkers.DataExport.StorageProviders.IStorageProvider, Sitecore.XConnect.DataTools.CortexWorkers</As>
    <LifeTime>Singleton</LifeTime>
    <Options>
        <ExponentialRetry>
            <MaxRetries>5</MaxRetries>
            <DelayInSeconds>1</DelayInSeconds>
            <MaxDelayInSeconds>5</MaxDelayInSeconds>
        </ExponentialRetry>
    </Options>
</StorageProviders.AzureBlobStorageProvider>

Configure a storage profile

The Data Export Tool comes with a default profile for each storage provider. You can define your own custom profiles by adding them to the sc.XConnect.DataTools.Workers.xml files. Each custom profile belongs to one storage provider, but you can have any number of profiles for each provider.

To configure a storage profile:

Specify the following settings:
- ProfileName – a unique profile name that is used during the data export task registration.
- StorageAccountConnectionStringName – for Azure storage providers, the name of the connection string to the Azure Storage account defined in the Processing role and Processing engine.
- Location – the name of location that holds the exported data files. The name of the setting depends on which storage provider you are configuring.

You specify the settings in the following nodes in the sc.XConnect.DataTools.Workers.xml files:

Storage provider	Profile node	Location description
Network folder	`Sitecore/Processing/Services/StorageProviders.NetworkFolderStorageProfile.SampleProfile/Options`	In the `<Path>` parameter, specify the local or network folder where the export files are stored. If the local or network folder does not exist, Sitecore rejects the export task.
Azure Blob Storage	`Sitecore/Processing/Services/StorageProviders.AzureBlobStorageProfile.SampleProfile/Options`	In the `<ContainerName>` parameter, specify the Azure container where the export files are stored. If the container does not exist in your Azure Storage account, Sitecore rejects the export task.
Azure File Storage	`Sitecore/Processing/Services/StorageProviders.AzureFileStorageProfile.SampleProfile/Options`	In the `<ShareName>` and `<DirectoryName>` parameters, specify the Azure share and directory where the export files are stored. If the share and directory do not exist in your Azure Storage account, Sitecore rejects the export task.

Storage provider

Profile node

Location description

Network folder

Sitecore/Processing/Services/StorageProviders.NetworkFolderStorageProfile.SampleProfile/Options

In the <Path> parameter, specify the local or network folder where the export files are stored.

If the local or network folder does not exist, Sitecore rejects the export task.

Azure Blob Storage

Sitecore/Processing/Services/StorageProviders.AzureBlobStorageProfile.SampleProfile/Options

In the <ContainerName> parameter, specify the Azure container where the export files are stored.

If the container does not exist in your Azure Storage account, Sitecore rejects the export task.

Azure File Storage

Sitecore/Processing/Services/StorageProviders.AzureFileStorageProfile.SampleProfile/Options

In the <ShareName> and <DirectoryName> parameters, specify the Azure share and directory where the export files are stored.

If the share and directory do not exist in your Azure Storage account, Sitecore rejects the export task.

For example, the default profile configuration for Azure Blob Storage is as follows:

RequestResponse

<StorageProviders.AzureBlobStorageProfile.SampleProfile>
    <Type>Sitecore.XConnect.DataTools.CortexWorkers.DataExport.StorageProviders.AzureBlobStorageProfile, Sitecore.XConnect.DataTools.CortexWorkers</Type>
    <As>Sitecore.XConnect.DataTools.CortexWorkers.DataExport.StorageProviders.AzureBlobStorageProfile, Sitecore.XConnect.DataTools.CortexWorkers</As>
    <LifeTime>Singleton</LifeTime>
    <Options>
        <ProfileName>SampleAzureBlobStorageProfile</ProfileName>
        <!--
            Defines the name of the connection string with credentials to the predefined Azure Storage Account.
        -->
        <StorageAccountConnectionStringName>xconnect.dataexport.storage.account</StorageAccountConnectionStringName>
        <!--
            Defines the name of the predefined Azure Blob Container.
        -->
        <ContainerName>samplecontainer</ContainerName>
    </Options>
</StorageProviders.AzureBlobStorageProfile.SampleProfile>

Do you have some feedback for us?

If you have suggestions for improving this article,