Handling mixed batches

Abstract

Sample console application to pre-populate the xDB with known contacts and simulate adding a batch of new and existing contacts.

Batches of mixed operation types (such as adding and updating contacts) may result in more complicated exception handling. Consider the following scenario:

  • A batch of 2000 contacts are imported from a CSV

  • The CSV contains 10 pieces of facet information - such as name, Twitter username, and e-mail address

  • The contacts could be new or existing

  • The existing contacts may or may not already have existing facet data

  • Conflicts may occur during the import

Assume that all contacts are identified by their Twitter username or email address. In the following example, decisions have been made to minimize the number of exceptions that might be thrown:

  • Existing contacts are identified by calling .GetAsync<Contact> at the very start. Whilst this is a separate call to xConnect, it means that we do not try to create a new contact for records that already exist. Creating a new contact with an identifier that already exists results in an exception, which means we would have to retry those operations.

  • If a contact exists but their facet data has not changed, we do not send an update - it is not necessary.

  • All ‘add’ and ‘update operations’ are still submitted as a single batch - however, we have minimized the risk of exceptions occurring.

The following sample console application allows you to:

  • Pre-populate the xDB with known contacts.

  • Simulates adding a batch of new and existing contacts.

To use the sample console application:

  1. Create a new console application in Visual Studio.

  2. Add the following NuGet packages from the Sitecore feed:

    • Sitecore.XConnect.Client

    • Sitecore.XConnect.Collection.Model

    • Sitecore.XConnect

  3. Paste the following code into Program.cs:

    using Sitecore.XConnect;
    using Sitecore.XConnect.Client;
    using Sitecore.XConnect.Client.WebApi;
    using Sitecore.XConnect.Collection.Model;
    using Sitecore.XConnect.Schema;
    using Sitecore.Xdb.Common.Web;
    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Threading.Tasks;
    using Sitecore.XConnect.Operations;
    using Sitecore.Xdb.Common.Web;
    using Console = System.Console;
    using PersonalInformation = Sitecore.XConnect.Collection.Model.PersonalInformation;
    
    namespace Sitecore.Documentation
    {
        public class Program
        {
            private static int charcount = 20; 
    
            private static void Main(string[] args)
            {
                MainAsync(args).ConfigureAwait(false).GetAwaiter().GetResult();
                Console.ForegroundColor = ConsoleColor.DarkGreen;
                Console.WriteLine("");
                Console.WriteLine("END OF PROGRAM.");
                Console.ReadKey();
            }
    
            private static async Task MainAsync(string[] args)
            {
                CertificateHttpClientHandlerModifierOptions options =
                    CertificateHttpClientHandlerModifierOptions.Parse(
                        "StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=5496B9D84850AFE5A3554B7B2D23F3B5C79CA53A");
    
                var certificateModifier = new CertificateHttpClientHandlerModifier(options);
    
                List<IHttpClientModifier> clientModifiers = new List<IHttpClientModifier>();
                var timeoutClientModifier = new TimeoutHttpClientModifier(new TimeSpan(0, 0, 20));
                clientModifiers.Add(timeoutClientModifier);
    
                var collectionClient = new CollectionWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"),
                    clientModifiers, new[] {certificateModifier});
                var searchClient = new SearchWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers,
                    new[] {certificateModifier});
                var configurationClient = new ConfigurationWebApiClient(
                    new Uri("https://tut4xconnect.dev.local/configuration"), clientModifiers, new[] {certificateModifier});
    
                var cfg = new XConnectClientConfiguration(
                    new XdbRuntimeModel(CollectionModel.Model), collectionClient, searchClient, configurationClient);
    
                try
                {
                    await cfg.InitializeAsync();
    
                    // Print xConnect if configuration is valid
                    var arr = new[]
                    {
                        @"            ______                                                       __     ",
                        @"           /      \                                                     |  \    ",
                        @" __    __ |  $$$$$$\  ______   _______   _______    ______    _______  _| $$_   ",
                        @"|  \  /  \| $$   \$$ /      \ |       \ |       \  /      \  /       \|   $$ \  ",
                        @"\$$\/  $$| $$      |  $$$$$$\| $$$$$$$\| $$$$$$$\|  $$$$$$\|  $$$$$$$ \$$$$$$   ",
                        @" >$$  $$ | $$   __ | $$  | $$| $$  | $$| $$  | $$| $$    $$| $$        | $$ __  ",
                        @" /  $$$$\ | $$__/  \| $$__/ $$| $$  | $$| $$  | $$| $$$$$$$$| $$_____   | $$|  \",
                        @"|  $$ \$$\ \$$    $$ \$$    $$| $$  | $$| $$  | $$ \$$     \ \$$     \   \$$  $$",
                        @" \$$   \$$  \$$$$$$   \$$$$$$  \$$   \$$ \$$   \$$  \$$$$$$$  \$$$$$$$    \$$$$ "
                    };
                    Console.WindowWidth = 160;
                    foreach (string line in arr)
                        Console.WriteLine(line);
    
                }
                catch (XdbModelConflictException ce)
                {
                    Console.WriteLine("ERROR:" + ce.Message);
                    return;
                }
    
                // Initialize a client using the validated configuration
                using (var client = new XConnectClient(cfg))
                {
                    try
                    {
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.Yellow;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Prefill the xDB with some test data:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        var newContacts = GenerateListOfPeople();
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Contacts created in xDB:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        Console.WriteLine(new String('-', charcount * 4));
    
                        foreach (var p in newContacts)
                        {
                            var newContact = new Contact();
                            client.AddContactIdentifier(newContact,
                                new ContactIdentifier(p.Source, p.Identifier, ContactIdentifierType.Known));
                            client.SetFacet(newContact,
                                new PersonalInformation() {FirstName = p.Firstname, LastName = p.LastName});
                            client.AddContact(newContact);
    
                            var row = p.Firstname.PadRight(charcount)
                                      + "| " + p.LastName.PadRight(charcount)
                                      + "| " + p.Identifier.PadRight(charcount)
                                      + "| " + p.Source.PadRight(charcount);
    
                            Console.WriteLine(row);
                            Console.WriteLine(new String('-', charcount * 4));
                        }
    
                        Console.WriteLine();
    
                        try
                        {
                            client.Submit();
                        }
                        catch (XdbExecutionException ex)
                        {
                            Console.WriteLine(ex.Message);
                        }
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Create contact data that you want to import - mix new and existing contacts!:");
                        Console.WriteLine("#############################################################################");
                        Console.ForegroundColor = ConsoleColor.White;
                        Console.WriteLine();
    
                        var newAndUpdatedContacts = GenerateListOfPeople();
    
                        Console.ForegroundColor = ConsoleColor.Green;
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine("Contact data to import:");
                        Console.WriteLine("#############################################################################");
                        Console.WriteLine();
                        Console.ForegroundColor = ConsoleColor.White;
    
                        Console.WriteLine(new String('-', charcount * 4));
    
                        foreach (var p in newAndUpdatedContacts)
                        {
                            var row = p.Firstname.PadRight(charcount)
                                      + "| " + p.LastName.PadRight(charcount)
                                      + "| " + p.Identifier.PadRight(charcount)
                                      + "| " + p.Source.PadRight(charcount);
    
                            Console.WriteLine(row);
                            Console.WriteLine(new String('-', charcount * 4));
                        }
    
                       
                        // Create identifiers from Identifier/Source properties so that we can retrieve contacts
                        var identifiers = newAndUpdatedContacts.Select(x => new IdentifiedContactReference(x.Source, x.Identifier)).ToArray();
    
                        /// Retrieve all contacts using list of identifiers - make sure you also retrieve the PersonalInfo facet in the expand options,
                        /// as we want to write to these.
                        var contactsTask = client.GetAsync<Contact>(identifiers,
                            new ContactExpandOptions(PersonalInformation.DefaultFacetKey));
    
                        var contactResults = await contactsTask;
    
                        if (contactResults != null && contactResults.Any())
                        {
                            foreach (var result in contactResults)
                            {
                                if (result.Exists)
                                {
                                    // Get existing contact's identifiers
                                    var contactIdentifiers = result.Entity.Identifiers.Select(i => i.Identifier);
    
                                    var csvEntry = newAndUpdatedContacts.Where(p =>
                                            result.Entity.Identifiers.Select(c => p.Identifier == c.Identifier)
                                                .FirstOrDefault())
                                        .FirstOrDefault();
    
                                    Console.WriteLine();
    
                                    Console.ForegroundColor = ConsoleColor.Cyan;
                                    Console.WriteLine("Updating existing contact: " + csvEntry.Identifier + "/" + csvEntry.Source);
                                    Console.ForegroundColor = ConsoleColor.White;
    
                                    if (csvEntry != null)
                                    {
                                        csvEntry.ContactExists = true;
    
                                        var personalInfoFacet =
                                            result.Entity.GetFacet<PersonalInformation>(PersonalInformation
                                                .DefaultFacetKey);
    
                                        if (personalInfoFacet != null)
                                        {
                                                
                                            Console.WriteLine("Contact has PersonalInformation facet.");
                                            /// Check if facet is identical to what's in the CSV - if so, ignore. No need to send unnecessarny facet updates.
    
                                            bool FacetHasChanged = false;
    
                                            if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.FirstName,
                                                csvEntry.Firstname))
                                            {
                                                Console.WriteLine("First name was: " + personalInfoFacet.FirstName);
                                                personalInfoFacet.FirstName = csvEntry.Firstname;
                                                FacetHasChanged = true;
                                                Console.WriteLine("First name changed to: " + csvEntry.Firstname);
                                            }
    
                                            if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.LastName,
                                                csvEntry.LastName))
                                            {
                                                Console.WriteLine("Last name was: " + personalInfoFacet.LastName);
                                                personalInfoFacet.LastName = csvEntry.LastName;
                                                FacetHasChanged = true;
                                                Console.WriteLine("Last name changed to: " + csvEntry.LastName);
                                            }
    
                                            if (FacetHasChanged)
                                            {
                                                client.SetFacet<PersonalInformation>(result.Entity,
                                                    PersonalInformation.DefaultFacetKey, personalInfoFacet);
                                            }
                                            else
                                            {
                                                Console.WriteLine("No facet data has changed.");
                                            }
                                        }
                                        else
                                        {
                                            Console.WriteLine("Contact does not have PersonalInfo facet - creating new facet.");
                                            /// Set PersonalInfo facet data - note that we are not performing any validation; if first name and surname were not provided, those values will be null or empty.
                                            /// This method adds a SetFacetOperation for each contact and adds it to the batch.
                                            client.SetFacet<PersonalInformation>(result.Entity,
                                                PersonalInformation.DefaultFacetKey, new PersonalInformation()
                                                {
                                                    FirstName = csvEntry.Firstname,
                                                    LastName = csvEntry.LastName
                                                });
                                        }
                                    }
                                }
                            }
                        }
    
                        // Now deal with new contacts
                        foreach (var csvEntry in newAndUpdatedContacts.Where(x => !x.ContactExists))
                        {
                            Console.ForegroundColor = ConsoleColor.Cyan;                        ;
                            Console.WriteLine("Creating new contact: " + csvEntry.Identifier + "/" + csvEntry.Source);
                            Console.ForegroundColor = ConsoleColor.White;
    
                            Console.WriteLine();
    
                            /// Create a new contact object for each row
                            var contact = new Contact();
    
                            /// In this example, we have elected not to add contacts that do not have a Identifier naem
                            if (!String.IsNullOrEmpty(csvEntry.Identifier))
                            {
                                /// In this instance, Identifier names are the unique identifier.
                                client.AddContactIdentifier(contact,
                                    new ContactIdentifier(csvEntry.Source, csvEntry.Identifier, ContactIdentifierType.Known));
    
                                client.SetFacet<PersonalInformation>(contact, PersonalInformation.DefaultFacetKey,
                                    new PersonalInformation()
                                    {
                                        FirstName = csvEntry.Firstname,
                                        LastName = csvEntry.LastName
                                    });
    
                                /// Add the contact - this method creates an AddContactOperation for each contact and adds it to the batch.
                                client.AddContact(contact);
                            }
                        }
    
                        await client.SubmitAsync();
                    }
                    catch (XdbExecutionException ex)
                    {
                        // Handle conflicts that may have happened when udpating existing contacts
                        var setPersonalOperations = ex.GetOperations(client)
                            .OfType<SetFacetOperation<PersonalInformation>>()
                            .Where(x => x.Result.Status == SaveResultStatus.Conflict);
    
                        // Handle contacts that already exist
                        var contactExists = ex.GetOperations(client).OfType<AddContactOperation>()
                            .Where(x => x.Result.Status == SaveResultStatus.AlreadyExists);
                    }
                }
            }
    
            private static List<Person> GenerateListOfPeople()
            {
                var contacts = new List<Person>();
                var createMoreContacts = true;
    
                string source = String.Empty;
                string identifier = String.Empty;
                string firstname = String.Empty;
                string lastname = String.Empty;
    
                int count = 1;
    
                while (createMoreContacts == true)
                {
                    Console.ForegroundColor = ConsoleColor.Yellow;
                    Console.WriteLine("Contact #" + count);
                    Console.ForegroundColor = ConsoleColor.White;
                    count++;
                    Console.Write("Contact identifier source: ");
                    source = Console.ReadLine();
                    Console.Write("Contact identifier: ");
                    identifier = Console.ReadLine();
                    Console.Write("First name: ");
                    firstname = Console.ReadLine();
                    Console.Write("Last name: ");
                    lastname = Console.ReadLine();
    
                    var person = new Person()
                    {
                        Firstname = firstname,
                        LastName = lastname,
                        Identifier = identifier,
                        Source = source
                    };
    
                    contacts.Add(person);
                    
                    Console.WriteLine();
                    Console.Write("Create more contacts (Y/N)?: ");
                    var createMore = Console.ReadLine();
                    Console.WriteLine();
    
    
                    if (createMore.ToLowerInvariant() == "n")
                    {
                        createMoreContacts = false;
                    }
                }
    
                return contacts;
            }
        }
    
        public class Person
        {
            public string Firstname { get; set; }
            public string LastName { get; set; }
            public string Identifier { get; set; }
            public string Source { get; set; }
            public bool ContactExists { get; set; }
        }
    
        public static class Helpers
        { 
    
            public static bool IsIdenticalOrEmptyOrNull(string current, string imported)
            {
                if (current.Equals(imported) || String.IsNullOrEmpty(imported))
                {
                    return true;
                }
    
                return false;
            }
        }
    }
  4. Replace sample xConnect service URL (https://xconnect.dev.local/).

  5. Replace the sample certificate thumbprint. You can find your certificate thumbprint in the <xconnect-root>\App_Config\AppSettings.config file.

  6. Press F5 to run the application. The following example shows the console application's output:

                ______                                                       __
               /      \                                                     |  \
     __    __ |  $$$$$$\  ______   _______   _______    ______    _______  _| $$_
    |  \  /  \| $$   \$$ /      \ |       \ |       \  /      \  /       \|   $$ \
    \$$\/  $$| $$      |  $$$$$$\| $$$$$$$\| $$$$$$$\|  $$$$$$\|  $$$$$$$ \$$$$$$
     >$$  $$ | $$   __ | $$  | $$| $$  | $$| $$  | $$| $$    $$| $$        | $$ __
     /  $$$$\ | $$__/  \| $$__/ $$| $$  | $$| $$  | $$| $$$$$$$$| $$_____   | $$|  \
    |  $$ \$$\ \$$    $$ \$$    $$| $$  | $$| $$  | $$ \$$     \ \$$     \   \$$  $$
     \$$   \$$  \$$$$$$   \$$$$$$  \$$   \$$ \$$   \$$  \$$$$$$$  \$$$$$$$    \$$$$
    
    #############################################################################
    Prefill the xDB with some test data:
    #############################################################################
    
    Contact #1
    Contact identifier source: source45
    Contact identifier: mhwelander
    First name: Martina
    Last name: Welander
    
    Create more contacts (Y/N)?: Y
    
    Contact #2
    Contact identifier source: source45
    Contact identifier: myrtlesitecore
    First name: Myrtle
    Last name: Sitecore
    
    Create more contacts (Y/N)?: N
    
    #############################################################################
    Contacts created in xDB:
    #############################################################################
    
    --------------------------------------------------------------------------------
    Martina             | Welander            | mhwelander          | source45
    --------------------------------------------------------------------------------
    Myrtle              | Sitecore            | myrtlesitecore      | source45
    --------------------------------------------------------------------------------
    
    #############################################################################
    Create contact data that you want to import - mix new and existing contacts!:
    #############################################################################
    
    Contact #1
    Contact identifier source: source45
    Contact identifier: mhwelander
    First name: Martina Updated
    Last name: Welander Updated
    
    Create more contacts (Y/N)?: Y
    
    Contact #2
    Contact identifier source: source45
    Contact identifier: bobmcbob
    First name: Bob
    Last name: McBob
    
    Create more contacts (Y/N)?: N
    
    #############################################################################
    Contact data to import:
    #############################################################################
    
    --------------------------------------------------------------------------------
    Martina Updated     | Welander Updated    | mhwelander          | source45
    --------------------------------------------------------------------------------
    Bob                 | McBob               | bobmcbob            | source45
    --------------------------------------------------------------------------------
    
    Updating existing contact: mhwelander/source45
    Contact has PersonalInformation facet.
    First name was: Martina
    First name changed to: Martina Updated
    Last name was: Welander
    Last name changed to: Welander Updated
    Creating new contact: bobmcbob/source45
    
    
    END OF PROGRAM.