Data extraction

Extracting contacts and interactions

Version:

Use the CreateContactEnumerator() and CreateInteractionEnumerator() methods to extract contacts and interactions. You must specify a batch size and use expand options to specify which facets should be returned with each result.

You cannot run data extraction synchronously.

Note

For an example of a data extraction feature used to extract and migrate data to another Sitecore instance, see the xConnect Migration code sample.

Extracting contacts

The following example demonstrates how to extract all contacts. Results are returned in batches of 200. The CreateContactEnumerator() method accepts a ContactExpandOptions object, which determines which contact facets and related interactions are returned.

Important

Exported JSON will include PII (Personally Identifiable Information) if you choose to extract facets marked [PIISensitive].

using Sitecore.XConnect;
using Sitecore.XConnect.Collection.Model;
using System;
using System.Collections.Generic;

namespace Documentation
{
    public class DataExtraction
    {
        // Async interaction
        public async void ExampleAsync()
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    // Extract contacts
                    var contactCursor = await client.CreateContactEnumerator(new ContactExpandOptions(PersonalInformation.DefaultFacetKey)
                    {
                        Interactions = new RelatedInteractionsExpandOptions(IpInfo.DefaultFacetKey)
                        {
                            StartDateTime = DateTime.UtcNow.AddDays(-300),
                            EndDateTime = DateTime.UtcNow.AddDays(-200)
                        }
                    }
                    , 200);
                    var totalContactsCount = contactCursor.TotalCount;

                    while (await contactCursor.MoveNext())
                    {
                        // Batch of 200
                        var currentBatch = contactCursor.Current;

                        // Write batch to third party or do something else
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Manage exceptions
                }
            }
        }
    }
}

Note

Data extraction returns all contacts, including contacts that were obsoleted during a merge.

Extracting interactions

The following example demonstrates how to extract all interactions that were saved before a specified cut-off date. Results are returned in batches of 200. The CreateInteractionEnumerator() method accepts a InteractionExpandOptions object, which determines which interaction facets and related contact facets are returned.

using Sitecore.XConnect;
using Sitecore.XConnect.Collection.Model;
using System;
using System.Collections.Generic;

namespace Documentation
{
    public class DataExtraction
    {
        // Async interaction
        public async void ExampleAsync()
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    // Extract interactions
                    var interactionCursor = await client.CreateInteractionEnumerator(DateTime.UtcNow, new InteractionExpandOptions(IpInfo.DefaultFacetKey)
                    {
                        Contact = new RelatedContactExpandOptions(PersonalInformation.DefaultFacetKey)
                    }
                    , 200);
                    var totalInteractionsCount = interactionCursor.TotalCount;

                    while (await interactionCursor.MoveNext())
                    {
                        // Batch of 200
                        var currentBatch = interactionCursor.Current;

                        // Write batch to third party or do something else
                    }

                }
                catch (XdbExecutionException ex)
                {
                    // Manage exceptions
                }
            }
        }
    }
}

Specifying a date range

The CreateInteractionEnumerator() method accepts the following optional constructor parameters that work together with the cutOffDate parameter:

minStartDateTime - the minimum interaction StartDateTime (inclusive).
maxStartDateTime- the maximum interaction StartDateTime (exclusive).

An interaction’s StartDateTime property represents when the interaction started, whereas the LastModified property represents when the interaction was saved to xConnect. If an interaction is imported from an external system at scheduled intervals, the LastModified date is likely to greater than the StartDateTime.

By specifying a start date range and a cut-off date, you can extract interactions that occurred between two dates, but exclude any interactions within that range that were saved to xConnect after a particular date.

A cut-off date is used to limit the interactions that are extracted from xConnect

Note

By default, the cutOffDate uses the creation time of the extraction cursor. Because the extraction can take time to process, it does not include any new contacts and interactions after that time.

The following example will return all interactions that started between 1st November and 10th November, but exclude any interactions that were saved after 8th November:

using Sitecore.XConnect;
using Sitecore.XConnect.Collection.Model;
using System;
using System.Collections.Generic;

namespace Documentation
{
    public class DataExtraction
    {
        // Async interaction
        public async void ExampleAsync()
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    var startDate = new DateTime(2017, 11, 01).ToUniversalTime(); // 1st November 2017
                    var endDate = new DateTime(2017, 11, 10).ToUniversalTime(); // 10th November 2017
                    var cutOff = new DateTime(2017, 11, 08).ToUniversalTime(); // 8th November 2017

                    // Extract interactions
                    var dateRangeInteractionCursor = await client.CreateInteractionEnumerator(cutOff, startDate, endDate, new InteractionExpandOptions(IpInfo.DefaultFacetKey)
                    {
                        Contact = new RelatedContactExpandOptions(PersonalInformation.DefaultFacetKey)
                    }
                    , 200);

                    var totalDateRangeInteractionsCount = interactionCursor.TotalCount;

                    while (await interactionCursor.MoveNext())
                    {
                        // Batch of 200
                        var currentBatch = dateRangeInteractionCursor.Current;
                    }

                }
                catch (XdbExecutionException ex)
                {
                    // Manage exceptions
                }
            }
        }
    }
}

Using a cursor mark

You can use a cursor mark to resume data extraction if the process is interrupted. This applies to both contact and interaction data extraction. The following example demonstrates how to pass a cursor mark into the CreateContactEnumerator() if an exception is thrown:

using Sitecore.XConnect;
using Sitecore.XConnect.Collection.Model;
using System;
using System.Collections.Generic;

namespace Documentation
{
    public class DataExtractionBookmark
    {
        // Async interaction
        public async void ExampleAsync(byte[] b)
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.XConnectClientConfiguration.GetClient())
            {
                IAsyncEntityBatchEnumerator<Contact> contactCursor = null;
                var batchSize = 200;
                byte[] bookmark = b;

                try
                {
                    var expandOptions = new RelatedInteractionsExpandOptions(IpInfo.DefaultFacetKey)
                    {
                        StartDateTime = DateTime.UtcNow.AddDays(-300),
                        EndDateTime = DateTime.UtcNow.AddDays(-200)
                    };

                    if (bookmark == null)
                    {
                        // Extract contacts - no bookmark
                        contactCursor = await client.CreateContactEnumerator(new ContactExpandOptions(PersonalInformation.DefaultFacetKey)
                        {
                            Interactions = expandOptions
                        }
                        , batchSize);
                    }
                    else
                    {
                        // Extract contacts - with bookmark
                        contactCursor = await client.CreateContactEnumerator(bookmark, new ContactExpandOptions(PersonalInformation.DefaultFacetKey)
                        {
                            Interactions = expandOptions
                        }
                        , batchSize);
                    }

                    var totalContactsCount = contactCursor.TotalCount;

                    while (await contactCursor.MoveNext())
                    {
                        // Batch of 200
                        var currentBatch = contactCursor.Current;
                        bookmark = contactCursor.GetBookmark();
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Something happened!
                    // Restart at cursor mark
                    ExampleAsync(bookmark);
                }
            }
        }
    }
}

Shard location information is not stored within the cursor. This means that cursor mark will expire if a shard is moved to a different server. If a cursor mark expires, you must start the data extraction process from the beginning.

Contact merge and data extraction

See Merge contacts for more information about how data extraction handles:

Contacts that have previously been merged.
Contact merges that occur whilst data extraction is ongoing.

If you have suggestions for improving this article, let us know!