Handling mixed batches

Current version: 10.2

Batches of mixed operation types (such as adding and updating contacts) may result in more complicated exception handling. Consider the following scenario:

  • A batch of 2000 contacts are imported from a CSV

  • The CSV contains 10 pieces of facet information - such as name, Twitter username, and e-mail address

  • The contacts could be new or existing

  • The existing contacts may or may not already have existing facet data

  • Conflicts may occur during the import

Assume that all contacts are identified by their Twitter username or e-mail address. In the following example, decisions have been made to minimize the number of exceptions that might be thrown:

  • Existing contacts are identified by calling .GetAsync<Contact> at the very start. Whilst this is a separate call to xConnect, it means that we do not try to create a new contact for records that already exist. Creating a new contact with an identifier that already exists results in an exception, which means we would have to retry those operations.

  • If a contact exists but their facet data has not changed, we do not send an update - it is not necessary.

  • All ‘add’ and ‘update operations’ are still submitted as a single batch - however, we have minimized the risk of exceptions occurring.

Sample console application

The following sample console application allows you to:

  • Pre-populate the xDB with known contacts.

  • Simulates adding a batch of new and existing contacts.

To use the sample console application:

  1. Create a new console application in Visual Studio.

  2. Add the following NuGet packages from the Sitecore feed:

    • Sitecore.XConnect.Client

    • Sitecore.XConnect.Collection.Model

    • Sitecore.XConnect

  3. Paste the following code into Program.cs:

    RequestResponse
    using Sitecore.XConnect;
    using Sitecore.XConnect.Client;
    using Sitecore.XConnect.Client.WebApi;
    using Sitecore.XConnect.Collection.Model;
    using Sitecore.XConnect.Schema;
    using Sitecore.Xdb.Common.Web;
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Threading.Tasks;
    using Sitecore.XConnect.Operations;
    using Sitecore.Xdb.Common.Web;
    using Console = System.Console;
    using PersonalInformation = Sitecore.XConnect.Collection.Model.PersonalInformation;
    
    namespace Sitecore.Documentation
    {
        public class Program
        {
            private static int charcount = 20; 
    
            private static void Main(string[] args)
            {
                MainAsync(args).ConfigureAwait(false).GetAwaiter().GetResult();
                Console.ForegroundColor = ConsoleColor.DarkGreen;
                Console.WriteLine("");
                Console.WriteLine("END OF PROGRAM.");
                Console.ReadKey();
            }
    
            private static async Task MainAsync(string[] args)
            {
                CertificateHttpClientHandlerModifierOptions options =
                    CertificateHttpClientHandlerModifierOptions.Parse(
                        "StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=5496B9D84850AFE5A3554B7B2D23F3B5C79CA53A");
    
                var certificateModifier = new CertificateHttpClientHandlerModifier(options);
    
                List<IHttpClientModifier> clientModifiers = new List<IHttpClientModifier>();
                var timeoutClientModifier = new TimeoutHttpClientModifier(new TimeSpan(0, 0, 20));
                clientModifiers.Add(timeoutClientModifier);
    
                var collectionClient = new CollectionWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"),
                    clientModifiers, new[] {certificateModifier});
                var searchClient = new SearchWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers,
                    new[] {certificateModifier});
                var configurationClient = new ConfigurationWebApiClient(
                    new Uri("https://tut4xconnect.dev.local/configuration"), clientModifiers, new[] {certificateModifier});
    
                var cfg = new XConnectClientConfiguration(
                    new XdbRuntimeModel(CollectionModel.Model), collectionClient, searchClient, configurationClient);
    
                try
                {
                    await cfg.InitializeAsync();
    
                    // Print xConnect if configuration is valid
                    var arr = new[]
                    {
                        @"            ______                                                       __     ",
                        @"           /      \                                                     |  \    ",
                        @" __    __ |  $$$$$$\  ______   _______   _______    ______    _______  _| $$_   ",
                        @"|  \  /  \| $$   \$$ /      \ |       \ |       \  /      \  /       \|   $$ \  ",
                        @"\$$\/  $$| $$      |  $$$$$$\| $$$$$$$\| $$$$$$$\|  $$$$$$\|  $$$$$$$ \$$$$$$   ",
                        @" >$$  $$ | $$   __ | $$  | $$| $$  | $$| $$  | $$| $$    $$| $$        | $$ __  ",
                        @" /  $$$$\ | $$__/  \| $$__/ $$| $$  | $$| $$  | $$| $$$$$$$$| $$_____   | $$|  \",
                        @"|  $$ \$$\ \$$    $$ \$$    $$| $$  | $$| $$  | $$ \$$     \ \$$     \   \$$  $$",
                        @" \$$   \$$  \$$$$$$   \$$$$$$  \$$   \$$ \$$   \$$  \$$$$$$$  \$$$$$$$    \$$$$ "
                    };
                    Console.WindowWidth = 160;
                    foreach (string line in arr)
                        Console.WriteLine(line);
    
                }
                catch (XdbModelConflictException ce)
                {
                    Console.WriteLine("ERROR:" + ce.Message);
                    return;
                }
    
                // Initialize a client using the validated configuration
                using (var client = new XConnectClient(cfg))
                {
                    try
                    {
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.Yellow;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Prefill the xDB with some test data:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        var newContacts = GenerateListOfPeople();
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Contacts created in xDB:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        Console.WriteLine(new string('-', charcount * 4));
    
                        foreach (var p in newContacts)
                        {
                            var newContact = new Contact();
                            client.AddContactIdentifier(newContact,
                                new ContactIdentifier(p.Source, p.Identifier, ContactIdentifierType.Known));
                            client.SetFacet(newContact,
                                new PersonalInformation() {FirstName = p.Firstname, LastName = p.LastName});
                            client.AddContact(newContact);
    
                            var row = p.Firstname.PadRight(charcount)
                                      + "| " + p.LastName.PadRight(charcount)
                                      + "| " + p.Identifier.PadRight(charcount)
                                      + "| " + p.Source.PadRight(charcount);
    
                            Console.WriteLine(row);
                            Console.WriteLine(new string('-', charcount * 4));
                        }
    
                        Console.WriteLine();
    
                        try
                        {
                            client.Submit();
                        }
                        catch (XdbExecutionException ex)
                        {
                            Console.WriteLine(ex.Message);
                        }
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Create contact data that you want to import - mix new and existing contacts!:");
                        Console.WriteLine("#############################################################################");
                        Console.ForegroundColor = ConsoleColor.White;
                        Console.WriteLine();
    
                        var newAndUpdatedContacts = GenerateListOfPeople();
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Contact data to import:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        Console.WriteLine(new string('-', charcount * 4));
    
                        foreach (var p in newAndUpdatedContacts)
                        {
                            var row = p.Firstname.PadRight(charcount)
                                      + "| " + p.LastName.PadRight(charcount)
                                      + "| " + p.Identifier.PadRight(charcount)
                                      + "| " + p.Source.PadRight(charcount);
    
                            Console.WriteLine(row);
                            Console.WriteLine(new string('-', charcount * 4));
                        }
    
                       
                        // Create identifiers from Identifier/Source properties so that we can retrieve contacts
                        var identifiers = newAndUpdatedContacts.Select(x => new IdentifiedContactReference(x.Source, x.Identifier)).ToArray();
    
                        // Retrieve all contacts using list of identifiers - make sure you also retrieve the PersonalInfo facet in the expand options,
                        // as we want to write to these.
                        var contactsTask = client.GetAsync<Contact>(identifiers, new ContactExecutionOptions(                        new ContactExpandOptions(PersonalInformation.DefaultFacetKey)));
    
                        var contactResults = await contactsTask;
    
                        if (contactResults != null && contactResults.Any())
                        {
                            foreach (var result in contactResults)
                            {
                                if (result.Exists)
                                {
                                    // Get existing contact's identifiers
                                    var contactIdentifiers = result.Entity.Identifiers.Select(i => i.Identifier);
    
                                    var csvEntry = newAndUpdatedContacts.FirstOrDefault(p
    => result.Entity.Identifiers.Select(c => p.Identifier ==
    c.Identifier).FirstOrDefault());
    
                                    Console.WriteLine();
    
                                    Console.ForegroundColor = ConsoleColor.Cyan;
                                    Console.WriteLine("Updating existing contact: " + csvEntry.Identifier + "/" + csvEntry.Source);
                                    Console.ForegroundColor = ConsoleColor.White;
    
                                    if (csvEntry != null)
                                    {
                                        csvEntry.ContactExists = true;
    
                                        var personalInfoFacet =
                                            result.Entity.GetFacet<PersonalInformation>(PersonalInformation
                                                .DefaultFacetKey);
    
                                        if (personalInfoFacet != null)
                                        {
                                                
                                            Console.WriteLine("Contact has PersonalInformation facet.");
                                            // Check if facet is identical to what's in the CSV - if so, ignore. No need to send unnecessarny facet updates.
    
                                            bool FacetHasChanged = false;
    
                                            if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.FirstName,
                                                csvEntry.Firstname))
                                            {
                                                Console.WriteLine("First name was: " + personalInfoFacet.FirstName);
                                                personalInfoFacet.FirstName = csvEntry.Firstname;
                                                FacetHasChanged = true;
                                                Console.WriteLine("First name changed to: " + csvEntry.Firstname);
                                            }
    
                                            if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.LastName,
                                                csvEntry.LastName))
                                            {
                                                Console.WriteLine("Last name was: " + personalInfoFacet.LastName);
                                                personalInfoFacet.LastName = csvEntry.LastName;
                                                FacetHasChanged = true;
                                                Console.WriteLine("Last name changed to: " + csvEntry.LastName);
                                            }
    
                                            if (FacetHasChanged)
                                            {
                                                client.SetFacet<PersonalInformation>(result.Entity,
                                                    PersonalInformation.DefaultFacetKey, personalInfoFacet);
                                            }
                                            else
                                            {
                                                Console.WriteLine("No facet data has changed.");
                                            }
                                        }
                                        else
                                        {
                                            Console.WriteLine("Contact does not have PersonalInfo facet - creating new facet.");
                                            // Set PersonalInfo facet data - note that we are not performing any validation; if first name and surname were not provided, those values will be null or empty.
                                            // This method adds a SetFacetOperation for each contact and adds it to the batch.
                                            client.SetFacet<PersonalInformation>(result.Entity,
                                                PersonalInformation.DefaultFacetKey, new PersonalInformation()
                                                {
                                                    FirstName = csvEntry.Firstname,
                                                    LastName = csvEntry.LastName
                                                });
                                        }
                                    }
                                }
                            }
                        }
    
                        // Now deal with new contacts
                        foreach (var csvEntry in newAndUpdatedContacts.Where(x => !x.ContactExists))
                        {
                            Console.ForegroundColor = ConsoleColor.Cyan;                        ;
                            Console.WriteLine("Creating new contact: " + csvEntry.Identifier + "/" + csvEntry.Source);
                            Console.ForegroundColor = ConsoleColor.White;
    
                            Console.WriteLine();
    
                            // Create a new contact object for each row
                            var contact = new Contact();
    
                            // In this example, we have elected not to add contacts that do not have a Identifier naem
                            if (!string.IsNullOrEmpty(csvEntry.Identifier))
                            {
                                // In this instance, Identifier names are the unique identifier.
                                client.AddContactIdentifier(contact,
                                    new ContactIdentifier(csvEntry.Source, csvEntry.Identifier, ContactIdentifierType.Known));
    
                                client.SetFacet<PersonalInformation>(contact, PersonalInformation.DefaultFacetKey,
                                    new PersonalInformation()
                                    {
                                        FirstName = csvEntry.Firstname,
                                        LastName = csvEntry.LastName
                                    });
    
                                // Add the contact - this method creates an AddContactOperation for each contact and adds it to the batch.
                                client.AddContact(contact);
                            }
                        }
    
                        await client.SubmitAsync();
                    }
                    catch (XdbExecutionException ex)
                    {
                        // Handle conflicts that may have happened when udpating existing contacts
                        var setPersonalOperations = ex.GetOperations(client)
                            .OfType<SetFacetOperation<PersonalInformation>>()
                            .Where(x => x.Result.Status == SaveResultStatus.Conflict);
    
                        // Handle contacts that already exist
                        var contactExists = ex.GetOperations(client).OfType<AddContactOperation>()
                            .Where(x => x.Result.Status == SaveResultStatus.AlreadyExists);
                    }
                }
            }
    
            private static List<Person> GenerateListOfPeople()
            {
                var contacts = new List<Person>();
                var createMoreContacts = true;
    
                string source = string.Empty;
                string identifier = string.Empty;
                string firstname = string.Empty;
                string lastname = string.Empty;
    
                int count = 1;
    
                while (createMoreContacts == true)
                {
                    Console.ForegroundColor = ConsoleColor.Yellow;
                    Console.WriteLine("Contact #" + count);
                    Console.ForegroundColor = ConsoleColor.White;
                    count++;
                    Console.Write("Contact identifier source: ");
                    source = Console.ReadLine();
                    Console.Write("Contact identifier: ");
                    identifier = Console.ReadLine();
                    Console.Write("First name: ");
                    firstname = Console.ReadLine();
                    Console.Write("Last name: ");
                    lastname = Console.ReadLine();
    
                    var person = new Person()
                    {
                        Firstname = firstname,
                        LastName = lastname,
                        Identifier = identifier,
                        Source = source
                    };
    
                    contacts.Add(person);
                    
                    Console.WriteLine();
                    Console.Write("Create more contacts (Y/N)?: ");
                    var createMore = Console.ReadLine();
                    Console.WriteLine();
    
    
                    if (createMore.ToLowerInvariant() == "n")
                    {
                        createMoreContacts = false;
                    }
                }
                return contacts;
            }
        }
    
        public class Person
        {
            public string Firstname { get; set; }
            public string LastName { get; set; }
            public string Identifier { get; set; }
            public string Source { get; set; }
            public bool ContactExists { get; set; }
        }
    
        public static class Helpers
        { 
            public static bool IsIdenticalOrEmptyOrNull(string current, string imported)
            {
                if (current.Equals(imported) || string.IsNullOrEmpty(imported))
                {
                    return true;
                }
    
                return false;
            }
        }
    }
  4. Replace sample xConnect service URL (https://xconnect.dev.local/).

  5. Replace the sample certificate thumbprint. You can find your certificate thumbprint in the <xconnect-root>\App_Config\AppSettings.config file.

  6. Press F5 to run the application. The following example shows the console application's output:

    RequestResponse
                ______                                                       __
               /      \                                                     |  \
     __    __ |  $$$$$$\  ______   _______   _______    ______    _______  _| $$_
    |  \  /  \| $$   \$$ /      \ |       \ |       \  /      \  /       \|   $$ \
    \$$\/  $$| $$      |  $$$$$$\| $$$$$$$\| $$$$$$$\|  $$$$$$\|  $$$$$$$ \$$$$$$
     >$$  $$ | $$   __ | $$  | $$| $$  | $$| $$  | $$| $$    $$| $$        | $$ __
     /  $$$$\ | $$__/  \| $$__/ $$| $$  | $$| $$  | $$| $$$$$$$$| $$_____   | $$|  \
    |  $$ \$$\ \$$    $$ \$$    $$| $$  | $$| $$  | $$ \$$     \ \$$     \   \$$  $$
     \$$   \$$  \$$$$$$   \$$$$$$  \$$   \$$ \$$   \$$  \$$$$$$$  \$$$$$$$    \$$$$
    
    #############################################################################
    Prefill the xDB with some test data:
    #############################################################################
    
    Contact #1
    Contact identifier source: source45
    Contact identifier: mhwelander
    First name: Martina
    Last name: Welander
    
    Create more contacts (Y/N)?: Y
    
    Contact #2
    Contact identifier source: source45
    Contact identifier: myrtlesitecore
    First name: Myrtle
    Last name: Sitecore
    
    Create more contacts (Y/N)?: N
    
    #############################################################################
    Contacts created in xDB:
    #############################################################################
    
    --------------------------------------------------------------------------------
    Martina             | Welander            | mhwelander          | source45
    --------------------------------------------------------------------------------
    Myrtle              | Sitecore            | myrtlesitecore      | source45
    --------------------------------------------------------------------------------
    
    #############################################################################
    Create contact data that you want to import - mix new and existing contacts!:
    #############################################################################
    
    Contact #1
    Contact identifier source: source45
    Contact identifier: mhwelander
    First name: Martina Updated
    Last name: Welander Updated
    
    Create more contacts (Y/N)?: Y
    
    Contact #2
    Contact identifier source: source45
    Contact identifier: bobmcbob
    First name: Bob
    Last name: McBob
    
    Create more contacts (Y/N)?: N
    
    #############################################################################
    Contact data to import:
    #############################################################################
    
    --------------------------------------------------------------------------------
    Martina Updated     | Welander Updated    | mhwelander          | source45
    --------------------------------------------------------------------------------
    Bob                 | McBob               | bobmcbob            | source45
    --------------------------------------------------------------------------------
    
    Updating existing contact: mhwelander/source45
    Contact has PersonalInformation facet.
    First name was: Martina
    First name changed to: Martina Updated
    Last name was: Welander
    Last name changed to: Welander Updated
    Creating new contact: bobmcbob/source45
    
    
    END OF PROGRAM.

Do you have some feedback for us?

If you have suggestions for improving this article,