Handling mixed batches
Batches of mixed operation types (such as adding and updating contacts) may result in more complicated exception handling. Consider the following scenario:
-
A batch of 2000 contacts are imported from a CSV
-
The CSV contains 10 pieces of facet information - such as name, Twitter username, and e-mail address
-
The contacts could be new or existing
-
The existing contacts may or may not already have existing facet data
-
Conflicts may occur during the import
Assume that all contacts are identified by their Twitter username or email address. In the following example, decisions have been made to minimize the number of exceptions that might be thrown:
-
Existing contacts are identified by calling
.GetAsync<Contact>
at the very start. Whilst this is a separate call to xConnect, it means that we do not try to create a new contact for records that already exist. Creating a new contact with an identifier that already exists results in an exception, which means we would have to retry those operations. -
If a contact exists but their facet data has not changed, we do not send an update - it is not necessary.
-
All ‘add’ and ‘update operations’ are still submitted as a single batch - however, we have minimized the risk of exceptions occurring.
Sample console application
The following sample console application allows you to:
-
Pre-populate the xDB with known contacts.
-
Simulates adding a batch of new and existing contacts.
To use the sample console application:
-
Create a new console application in Visual Studio.
-
Add the following NuGet packages from the Sitecore feed:
-
Sitecore.XConnect.Client
-
Sitecore.XConnect.Collection.Model
-
Sitecore.XConnect
-
-
Paste the following code into
Program.cs
:RequestResponsec#using Sitecore.XConnect; using Sitecore.XConnect.Client; using Sitecore.XConnect.Client.WebApi; using Sitecore.XConnect.Collection.Model; using Sitecore.XConnect.Schema; using Sitecore.Xdb.Common.Web; using System; using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; using Sitecore.XConnect.Operations; using Sitecore.Xdb.Common.Web; using Console = System.Console; using PersonalInformation = Sitecore.XConnect.Collection.Model.PersonalInformation; namespace Sitecore.Documentation { public class Program { private static int charcount = 20; private static void Main(string[] args) { MainAsync(args).ConfigureAwait(false).GetAwaiter().GetResult(); Console.ForegroundColor = ConsoleColor.DarkGreen; Console.WriteLine(""); Console.WriteLine("END OF PROGRAM."); Console.ReadKey(); } private static async Task MainAsync(string[] args) { CertificateHttpClientHandlerModifierOptions options = CertificateHttpClientHandlerModifierOptions.Parse( "StoreName=My;StoreLocation=LocalMachine;FindType=FindByThumbprint;FindValue=5496B9D84850AFE5A3554B7B2D23F3B5C79CA53A"); var certificateModifier = new CertificateHttpClientHandlerModifier(options); List<IHttpClientModifier> clientModifiers = new List<IHttpClientModifier>(); var timeoutClientModifier = new TimeoutHttpClientModifier(new TimeSpan(0, 0, 20)); clientModifiers.Add(timeoutClientModifier); var collectionClient = new CollectionWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers, new[] {certificateModifier}); var searchClient = new SearchWebApiClient(new Uri("https://tut4xconnect.dev.local/odata"), clientModifiers, new[] {certificateModifier}); var configurationClient = new ConfigurationWebApiClient( new Uri("https://tut4xconnect.dev.local/configuration"), clientModifiers, new[] {certificateModifier}); var cfg = new XConnectClientConfiguration( new XdbRuntimeModel(CollectionModel.Model), collectionClient, searchClient, configurationClient); try { await cfg.InitializeAsync(); // Print xConnect if configuration is valid var arr = new[] { @" ______ __ ", @" / \ | \ ", @" __ __ | $$$$$$\ ______ _______ _______ ______ _______ _| $$_ ", @"| \ / \| $$ \$$ / \ | \ | \ / \ / \| $$ \ ", @"\$$\/ $$| $$ | $$$$$$\| $$$$$$$\| $$$$$$$\| $$$$$$\| $$$$$$$ \$$$$$$ ", @" >$$ $$ | $$ __ | $$ | $$| $$ | $$| $$ | $$| $$ $$| $$ | $$ __ ", @" / $$$$\ | $$__/ \| $$__/ $$| $$ | $$| $$ | $$| $$$$$$$$| $$_____ | $$| \", @"| $$ \$$\ \$$ $$ \$$ $$| $$ | $$| $$ | $$ \$$ \ \$$ \ \$$ $$", @" \$$ \$$ \$$$$$$ \$$$$$$ \$$ \$$ \$$ \$$ \$$$$$$$ \$$$$$$$ \$$$$ " }; Console.WindowWidth = 160; foreach (string line in arr) Console.WriteLine(line); } catch (XdbModelConflictException ce) { Console.WriteLine("ERROR:" + ce.Message); return; } // Initialize a client using the validated configuration using (var client = new XConnectClient(cfg)) { try { Console.WriteLine(); Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine("#############################################################################"); Console.WriteLine("Prefill the xDB with some test data:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; var newContacts = GenerateListOfPeople(); Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Contacts created in xDB:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(new String('-', charcount * 4)); foreach (var p in newContacts) { var newContact = new Contact(); client.AddContactIdentifier(newContact, new ContactIdentifier(p.Source, p.Identifier, ContactIdentifierType.Known)); client.SetFacet(newContact, new PersonalInformation() {FirstName = p.Firstname, LastName = p.LastName}); client.AddContact(newContact); var row = p.Firstname.PadRight(charcount) + "| " + p.LastName.PadRight(charcount) + "| " + p.Identifier.PadRight(charcount) + "| " + p.Source.PadRight(charcount); Console.WriteLine(row); Console.WriteLine(new String('-', charcount * 4)); } Console.WriteLine(); try { client.Submit(); } catch (XdbExecutionException ex) { Console.WriteLine(ex.Message); } Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Create contact data that you want to import - mix new and existing contacts!:"); Console.WriteLine("#############################################################################"); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(); var newAndUpdatedContacts = GenerateListOfPeople(); Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("#############################################################################"); Console.WriteLine("Contact data to import:"); Console.WriteLine("#############################################################################"); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(new String('-', charcount * 4)); foreach (var p in newAndUpdatedContacts) { var row = p.Firstname.PadRight(charcount) + "| " + p.LastName.PadRight(charcount) + "| " + p.Identifier.PadRight(charcount) + "| " + p.Source.PadRight(charcount); Console.WriteLine(row); Console.WriteLine(new String('-', charcount * 4)); } // Create identifiers from Identifier/Source properties so that we can retrieve contacts var identifiers = newAndUpdatedContacts.Select(x => new IdentifiedContactReference(x.Source, x.Identifier)).ToArray(); /// Retrieve all contacts using list of identifiers - make sure you also retrieve the PersonalInfo facet in the expand options, /// as we want to write to these. var contactsTask = client.GetAsync<Contact>(identifiers, new ContactExpandOptions(PersonalInformation.DefaultFacetKey)); var contactResults = await contactsTask; if (contactResults != null && contactResults.Any()) { foreach (var result in contactResults) { if (result.Exists) { // Get existing contact's identifiers var contactIdentifiers = result.Entity.Identifiers.Select(i => i.Identifier); var csvEntry = newAndUpdatedContacts.Where(p => result.Entity.Identifiers.Select(c => p.Identifier == c.Identifier) .FirstOrDefault()) .FirstOrDefault(); Console.WriteLine(); Console.ForegroundColor = ConsoleColor.Cyan; Console.WriteLine("Updating existing contact: " + csvEntry.Identifier + "/" + csvEntry.Source); Console.ForegroundColor = ConsoleColor.White; if (csvEntry != null) { csvEntry.ContactExists = true; var personalInfoFacet = result.Entity.GetFacet<PersonalInformation>(PersonalInformation .DefaultFacetKey); if (personalInfoFacet != null) { Console.WriteLine("Contact has PersonalInformation facet."); /// Check if facet is identical to what's in the CSV - if so, ignore. No need to send unnecessarny facet updates. bool FacetHasChanged = false; if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.FirstName, csvEntry.Firstname)) { Console.WriteLine("First name was: " + personalInfoFacet.FirstName); personalInfoFacet.FirstName = csvEntry.Firstname; FacetHasChanged = true; Console.WriteLine("First name changed to: " + csvEntry.Firstname); } if (!Helpers.IsIdenticalOrEmptyOrNull(personalInfoFacet.LastName, csvEntry.LastName)) { Console.WriteLine("Last name was: " + personalInfoFacet.LastName); personalInfoFacet.LastName = csvEntry.LastName; FacetHasChanged = true; Console.WriteLine("Last name changed to: " + csvEntry.LastName); } if (FacetHasChanged) { client.SetFacet<PersonalInformation>(result.Entity, PersonalInformation.DefaultFacetKey, personalInfoFacet); } else { Console.WriteLine("No facet data has changed."); } } else { Console.WriteLine("Contact does not have PersonalInfo facet - creating new facet."); /// Set PersonalInfo facet data - note that we are not performing any validation; if first name and surname were not provided, those values will be null or empty. /// This method adds a SetFacetOperation for each contact and adds it to the batch. client.SetFacet<PersonalInformation>(result.Entity, PersonalInformation.DefaultFacetKey, new PersonalInformation() { FirstName = csvEntry.Firstname, LastName = csvEntry.LastName }); } } } } } // Now deal with new contacts foreach (var csvEntry in newAndUpdatedContacts.Where(x => !x.ContactExists)) { Console.ForegroundColor = ConsoleColor.Cyan; ; Console.WriteLine("Creating new contact: " + csvEntry.Identifier + "/" + csvEntry.Source); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine(); /// Create a new contact object for each row var contact = new Contact(); /// In this example, we have elected not to add contacts that do not have a Identifier naem if (!String.IsNullOrEmpty(csvEntry.Identifier)) { /// In this instance, Identifier names are the unique identifier. client.AddContactIdentifier(contact, new ContactIdentifier(csvEntry.Source, csvEntry.Identifier, ContactIdentifierType.Known)); client.SetFacet<PersonalInformation>(contact, PersonalInformation.DefaultFacetKey, new PersonalInformation() { FirstName = csvEntry.Firstname, LastName = csvEntry.LastName }); /// Add the contact - this method creates an AddContactOperation for each contact and adds it to the batch. client.AddContact(contact); } } await client.SubmitAsync(); } catch (XdbExecutionException ex) { // Handle conflicts that may have happened when udpating existing contacts var setPersonalOperations = ex.GetOperations(client) .OfType<SetFacetOperation<PersonalInformation>>() .Where(x => x.Result.Status == SaveResultStatus.Conflict); // Handle contacts that already exist var contactExists = ex.GetOperations(client).OfType<AddContactOperation>() .Where(x => x.Result.Status == SaveResultStatus.AlreadyExists); } } } private static List<Person> GenerateListOfPeople() { var contacts = new List<Person>(); var createMoreContacts = true; string source = String.Empty; string identifier = String.Empty; string firstname = String.Empty; string lastname = String.Empty; int count = 1; while (createMoreContacts == true) { Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine("Contact #" + count); Console.ForegroundColor = ConsoleColor.White; count++; Console.Write("Contact identifier source: "); source = Console.ReadLine(); Console.Write("Contact identifier: "); identifier = Console.ReadLine(); Console.Write("First name: "); firstname = Console.ReadLine(); Console.Write("Last name: "); lastname = Console.ReadLine(); var person = new Person() { Firstname = firstname, LastName = lastname, Identifier = identifier, Source = source }; contacts.Add(person); Console.WriteLine(); Console.Write("Create more contacts (Y/N)?: "); var createMore = Console.ReadLine(); Console.WriteLine(); if (createMore.ToLowerInvariant() == "n") { createMoreContacts = false; } } return contacts; } } public class Person { public string Firstname { get; set; } public string LastName { get; set; } public string Identifier { get; set; } public string Source { get; set; } public bool ContactExists { get; set; } } public static class Helpers { public static bool IsIdenticalOrEmptyOrNull(string current, string imported) { if (current.Equals(imported) || String.IsNullOrEmpty(imported)) { return true; } return false; } } }
-
Replace sample xConnect service URL (
https://xconnect.dev.local/
). -
Replace the sample certificate thumbprint. You can find your certificate thumbprint in the
<xconnect-root>\App_Config\AppSettings.config
file. -
Press F5 to run the application. The following example shows the console application's output:
RequestResponseshell______ __ / \ | \ __ __ | $$$$$$\ ______ _______ _______ ______ _______ _| $$_ | \ / \| $$ \$$ / \ | \ | \ / \ / \| $$ \ \$$\/ $$| $$ | $$$$$$\| $$$$$$$\| $$$$$$$\| $$$$$$\| $$$$$$$ \$$$$$$ >$$ $$ | $$ __ | $$ | $$| $$ | $$| $$ | $$| $$ $$| $$ | $$ __ / $$$$\ | $$__/ \| $$__/ $$| $$ | $$| $$ | $$| $$$$$$$$| $$_____ | $$| \ | $$ \$$\ \$$ $$ \$$ $$| $$ | $$| $$ | $$ \$$ \ \$$ \ \$$ $$ \$$ \$$ \$$$$$$ \$$$$$$ \$$ \$$ \$$ \$$ \$$$$$$$ \$$$$$$$ \$$$$ ############################################################################# Prefill the xDB with some test data: ############################################################################# Contact #1 Contact identifier source: source45 Contact identifier: mhwelander First name: Martina Last name: Welander Create more contacts (Y/N)?: Y Contact #2 Contact identifier source: source45 Contact identifier: myrtlesitecore First name: Myrtle Last name: Sitecore Create more contacts (Y/N)?: N ############################################################################# Contacts created in xDB: ############################################################################# -------------------------------------------------------------------------------- Martina | Welander | mhwelander | source45 -------------------------------------------------------------------------------- Myrtle | Sitecore | myrtlesitecore | source45 -------------------------------------------------------------------------------- ############################################################################# Create contact data that you want to import - mix new and existing contacts!: ############################################################################# Contact #1 Contact identifier source: source45 Contact identifier: mhwelander First name: Martina Updated Last name: Welander Updated Create more contacts (Y/N)?: Y Contact #2 Contact identifier source: source45 Contact identifier: bobmcbob First name: Bob Last name: McBob Create more contacts (Y/N)?: N ############################################################################# Contact data to import: ############################################################################# -------------------------------------------------------------------------------- Martina Updated | Welander Updated | mhwelander | source45 -------------------------------------------------------------------------------- Bob | McBob | bobmcbob | source45 -------------------------------------------------------------------------------- Updating existing contact: mhwelander/source45 Contact has PersonalInformation facet. First name was: Martina First name changed to: Martina Updated Last name was: Welander Last name changed to: Welander Updated Creating new contact: bobmcbob/source45 END OF PROGRAM.