Handling mixed batches
Batches of mixed operation types (such as adding and updating contacts) may result in more complicated exception handling. Consider the following scenario:
-
A batch of 2000 contacts are imported from a CSV
-
The CSV contains 10 pieces of facet information - such as name, Twitter username, and e-mail address
-
The contacts could be new or existing
-
The existing contacts may or may not already have existing facet data
-
Conflicts may occur during the import
Assume that all contacts are identified by their Twitter username or e-mail address. In the following example, decisions have been made to minimize the number of exceptions that might be thrown:
-
Existing contacts are identified by calling
.GetAsync<Contact>
at the very start. Whilst this is a separate call to xConnect, it means that we do not try to create a new contact for records that already exist. Creating a new contact with an identifier that already exists results in an exception, which means we would have to retry those operations. -
If a contact exists but their facet data has not changed, we do not send an update - it is not necessary.
-
All ‘add’ and ‘update operations’ are still submitted as a single batch - however, we have minimized the risk of exceptions occurring.
Sample console application
The following sample console application allows you to:
-
Pre-populate the xDB with known contacts.
-
Simulates adding a batch of new and existing contacts.
To use the sample console application:
-
Create a new console application in Visual Studio.
-
Add the following NuGet packages from the Sitecore feed:
-
Sitecore.XConnect.Client
-
Sitecore.XConnect.Collection.Model
-
Sitecore.XConnect
-
-
Paste the following code into
Program.cs
:RequestResponsec#using Sitecore.XConnect; using Sitecore.XConnect.Client; using Sitecore.XConnect.Client.WebApi; using Sitecore.XConnect.Collection.Model; using Sitecore.XConnect.Schema; using Sitecore.Xdb.Common.Web; using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; using Sitecore.XConnect.Operations; using Sitecore.Xdb.Common.Web; using Console = System.Console; using PersonalInformation = Sitecore.XConnect.Collection.Model.PersonalInformation; namespace Sitecore.Documentation { public class Program { private static int charcount = 20; private static void Main(string[] args) { MainAsync(args).ConfigureAwait(false).GetAwaiter().GetResult(); Console.ForegroundColor = ConsoleColor.DarkGreen; Console.WriteLine(""); Console.WriteLine("END OF PROGRAM."); Console.ReadKey(); } private static async Task MainAsync(string[] args) { CertificateHttpClientHandlerModifierOptions options = CertificateHttpClientHandlerModifierOptions.Parse( "StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=5496B9D84850AFE5A3554B7B2D23F3B5C79CA53A"); var certificateModifier = new CertificateHttpClientHandlerModifier(options); List<IHttpClientModifier> clientModifiers = new List<IHttpClientModifier>(); var timeoutClientModifier = new TimeoutHttpClientModifier(new TimeSpan(0, 0, 20)); clientModifiers.Add(timeoutClientModifier); var collectionClient = new CollectionWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers, new[] {certificateModifier}); var searchClient = new SearchWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers, new[] {certificateModifier}); var configurationClient = new ConfigurationWebApiClient( new Uri("https://tut4xconnect.dev.local/configuration"), clientModifiers, new[] {certificateModifier}); var cfg = new XConnectClientConfiguration( new XdbRuntimeModel(CollectionModel.Model), collectionClient, searchClient, configurationClient); try { await cfg.InitializeAsync(); // Print xConnect if configuration is valid var arr = new[] { @" ______ __ ", @" / \ | \ ", @" __ __ | $$$$$$\ ______ _______ _______ ______ _______ _| $$_ ", @"| \ / \| $$ \$$ / \ | \ | \ / \ / \| $$ \ ", @"\$$\/ $$| $$ | $$$$$$\| $$$$$$$\| $$$$$$$\| $$$$$$\| $$$$$$$ \$$$$$$ ", @" >$$ $$ | $$ __ | $$ | $$| $$ | $$| $$ | $$| $$ $$| $$ | $$ __ ", @" / $$$$\ | $$__/ \| $$__/ $$| $$ | $$| $$ | $$| $$$$$$$$| $$_____ | $$| \", @"| $$ \$$\ \$$ $$ \$$ $$| $$ | $$| $$ | $$ \$$ \ \$$ \ \$$ $$", @" \$$ \$$ \$$$$$$ \$$$$$$ \$$ \$$ \$$ \$$ \$$$$$$$ \$$$$$$$ \$$$$ " }; Console.WindowWidth = 160; foreach (string line in arr) Console.WriteLine(line); } catch (XdbModelConflictException ce) { Console.WriteLine("ERROR:" + ce.Message); return; } // Initialize a client using the validated configuration using (var client = new XConnectClient(cfg)) { try { Console.WriteLine(); Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine("#############################################################################"); Console.WriteLine("Prefill the xDB with some test data:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; var newContacts = GenerateListOfPeople(); Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Contacts created in xDB:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(new string('-', charcount * 4)); foreach (var p in newContacts) { var newContact = new Contact(); client.AddContactIdentifier(newContact, new ContactIdentifier(p.Source, p.Identifier, ContactIdentifierType.Known)); client.SetFacet(newContact, new PersonalInformation() {FirstName = p.Firstname, LastName = p.LastName}); client.AddContact(newContact); var row = p.Firstname.PadRight(charcount) + "| " + p.LastName.PadRight(charcount) + "| " + p.Identifier.PadRight(charcount) + "| " + p.Source.PadRight(charcount); Console.WriteLine(row); Console.WriteLine(new string('-', charcount * 4)); } Console.WriteLine(); try { client.Submit(); } catch (XdbExecutionException ex) { Console.WriteLine(ex.Message); } Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Create contact data that you want to import - mix new and existing contacts!:"); Console.WriteLine("#############################################################################"); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(); var newAndUpdatedContacts = GenerateListOfPeople(); Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Contact data to import:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(new string('-', charcount * 4)); foreach (var p in newAndUpdatedContacts) { var row = p.Firstname.PadRight(charcount) + "| " + p.LastName.PadRight(charcount) + "| " + p.Identifier.PadRight(charcount) + "| " + p.Source.PadRight(charcount); Console.WriteLine(row); Console.WriteLine(new string('-', charcount * 4)); } // Create identifiers from Identifier/Source properties so that we can retrieve contacts var identifiers = newAndUpdatedContacts.Select(x => new IdentifiedContactReference(x.Source, x.Identifier)).ToArray(); // Retrieve all contacts using list of identifiers - make sure you also retrieve the PersonalInfo facet in the expand options, // as we want to write to these. var contactsTask = client.GetAsync<Contact>(identifiers, new ContactExecutionOptions( new ContactExpandOptions(PersonalInformation.DefaultFacetKey))); var contactResults = await contactsTask; if (contactResults != null && contactResults.Any()) { foreach (var result in contactResults) { if (result.Exists) { // Get existing contact's identifiers var contactIdentifiers = result.Entity.Identifiers.Select(i => i.Identifier); var csvEntry = newAndUpdatedContacts.FirstOrDefault(p => result.Entity.Identifiers.Select(c => p.Identifier == c.Identifier).FirstOrDefault()); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.Cyan; Console.WriteLine("Updating existing contact: " + csvEntry.Identifier + "/" + csvEntry.Source); Console.ForegroundColor = ConsoleColor.White; if (csvEntry != null) { csvEntry.ContactExists = true; var personalInfoFacet = result.Entity.GetFacet<PersonalInformation>(PersonalInformation .DefaultFacetKey); if (personalInfoFacet != null) { Console.WriteLine("Contact has PersonalInformation facet."); // Check if facet is identical to what's in the CSV - if so, ignore. No need to send unnecessarny facet updates. bool FacetHasChanged = false; if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.FirstName, csvEntry.Firstname)) { Console.WriteLine("First name was: " + personalInfoFacet.FirstName); personalInfoFacet.FirstName = csvEntry.Firstname; FacetHasChanged = true; Console.WriteLine("First name changed to: " + csvEntry.Firstname); } if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.LastName, csvEntry.LastName)) { Console.WriteLine("Last name was: " + personalInfoFacet.LastName); personalInfoFacet.LastName = csvEntry.LastName; FacetHasChanged = true; Console.WriteLine("Last name changed to: " + csvEntry.LastName); } if (FacetHasChanged) { client.SetFacet<PersonalInformation>(result.Entity, PersonalInformation.DefaultFacetKey, personalInfoFacet); } else { Console.WriteLine("No facet data has changed."); } } else { Console.WriteLine("Contact does not have PersonalInfo facet - creating new facet."); // Set PersonalInfo facet data - note that we are not performing any validation; if first name and surname were not provided, those values will be null or empty. // This method adds a SetFacetOperation for each contact and adds it to the batch. client.SetFacet<PersonalInformation>(result.Entity, PersonalInformation.DefaultFacetKey, new PersonalInformation() { FirstName = csvEntry.Firstname, LastName = csvEntry.LastName }); } } } } } // Now deal with new contacts foreach (var csvEntry in newAndUpdatedContacts.Where(x => !x.ContactExists)) { Console.ForegroundColor = ConsoleColor.Cyan; ; Console.WriteLine("Creating new contact: " + csvEntry.Identifier + "/" + csvEntry.Source); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(); // Create a new contact object for each row var contact = new Contact(); // In this example, we have elected not to add contacts that do not have a Identifier naem if (!string.IsNullOrEmpty(csvEntry.Identifier)) { // In this instance, Identifier names are the unique identifier. client.AddContactIdentifier(contact, new ContactIdentifier(csvEntry.Source, csvEntry.Identifier, ContactIdentifierType.Known)); client.SetFacet<PersonalInformation>(contact, PersonalInformation.DefaultFacetKey, new PersonalInformation() { FirstName = csvEntry.Firstname, LastName = csvEntry.LastName }); // Add the contact - this method creates an AddContactOperation for each contact and adds it to the batch. client.AddContact(contact); } } await client.SubmitAsync(); } catch (XdbExecutionException ex) { // Handle conflicts that may have happened when udpating existing contacts var setPersonalOperations = ex.GetOperations(client) .OfType<SetFacetOperation<PersonalInformation>>() .Where(x => x.Result.Status == SaveResultStatus.Conflict); // Handle contacts that already exist var contactExists = ex.GetOperations(client).OfType<AddContactOperation>() .Where(x => x.Result.Status == SaveResultStatus.AlreadyExists); } } } private static List<Person> GenerateListOfPeople() { var contacts = new List<Person>(); var createMoreContacts = true; string source = string.Empty; string identifier = string.Empty; string firstname = string.Empty; string lastname = string.Empty; int count = 1; while (createMoreContacts == true) { Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine("Contact #" + count); Console.ForegroundColor = ConsoleColor.White; count++; Console.Write("Contact identifier source: "); source = Console.ReadLine(); Console.Write("Contact identifier: "); identifier = Console.ReadLine(); Console.Write("First name: "); firstname = Console.ReadLine(); Console.Write("Last name: "); lastname = Console.ReadLine(); var person = new Person() { Firstname = firstname, LastName = lastname, Identifier = identifier, Source = source }; contacts.Add(person); Console.WriteLine(); Console.Write("Create more contacts (Y/N)?: "); var createMore = Console.ReadLine(); Console.WriteLine(); if (createMore.ToLowerInvariant() == "n") { createMoreContacts = false; } } return contacts; } } public class Person { public string Firstname { get; set; } public string LastName { get; set; } public string Identifier { get; set; } public string Source { get; set; } public bool ContactExists { get; set; } } public static class Helpers { public static bool IsIdenticalOrEmptyOrNull(string current, string imported) { if (current.Equals(imported) || string.IsNullOrEmpty(imported)) { return true; } return false; } } }
-
Replace sample xConnect service URL (
https://xconnect.dev.local/
). -
Replace the sample certificate thumbprint. You can find your certificate thumbprint in the
<xconnect-root>\App_Config\AppSettings.config
file. -
Press F5 to run the application. The following example shows the console application's output:
RequestResponseshell______ __ / \ | \ __ __ | $$$$$$\ ______ _______ _______ ______ _______ _| $$_ | \ / \| $$ \$$ / \ | \ | \ / \ / \| $$ \ \$$\/ $$| $$ | $$$$$$\| $$$$$$$\| $$$$$$$\| $$$$$$\| $$$$$$$ \$$$$$$ >$$ $$ | $$ __ | $$ | $$| $$ | $$| $$ | $$| $$ $$| $$ | $$ __ / $$$$\ | $$__/ \| $$__/ $$| $$ | $$| $$ | $$| $$$$$$$$| $$_____ | $$| \ | $$ \$$\ \$$ $$ \$$ $$| $$ | $$| $$ | $$ \$$ \ \$$ \ \$$ $$ \$$ \$$ \$$$$$$ \$$$$$$ \$$ \$$ \$$ \$$ \$$$$$$$ \$$$$$$$ \$$$$ ############################################################################# Prefill the xDB with some test data: ############################################################################# Contact #1 Contact identifier source: source45 Contact identifier: mhwelander First name: Martina Last name: Welander Create more contacts (Y/N)?: Y Contact #2 Contact identifier source: source45 Contact identifier: myrtlesitecore First name: Myrtle Last name: Sitecore Create more contacts (Y/N)?: N ############################################################################# Contacts created in xDB: ############################################################################# -------------------------------------------------------------------------------- Martina | Welander | mhwelander | source45 -------------------------------------------------------------------------------- Myrtle | Sitecore | myrtlesitecore | source45 -------------------------------------------------------------------------------- ############################################################################# Create contact data that you want to import - mix new and existing contacts!: ############################################################################# Contact #1 Contact identifier source: source45 Contact identifier: mhwelander First name: Martina Updated Last name: Welander Updated Create more contacts (Y/N)?: Y Contact #2 Contact identifier source: source45 Contact identifier: bobmcbob First name: Bob Last name: McBob Create more contacts (Y/N)?: N ############################################################################# Contact data to import: ############################################################################# -------------------------------------------------------------------------------- Martina Updated | Welander Updated | mhwelander | source45 -------------------------------------------------------------------------------- Bob | McBob | bobmcbob | source45 -------------------------------------------------------------------------------- Updating existing contact: mhwelander/source45 Contact has PersonalInformation facet. First name was: Martina First name changed to: Martina Updated Last name was: Welander Last name changed to: Welander Updated Creating new contact: bobmcbob/source45 END OF PROGRAM.