Paginating xConnect search results

Abstract

How to paginate your results using the GetBatchEnumerator() and GetBatchEnumeratorSync() methods and how to use the Skip() and Take () methods.

You can paginate your results in one of two ways:

  • Using the recommended GetBatchEnumerator() / GetBatchEnumeratorSync() methods

  • Using the Skip() and Take() methods

If you do not paginate, the number of results is limited to the default batch size in order to preserve performance. To export data, you should use the data extraction feature.

Skipping a large number of results is just as inefficient as requesting a very large batch. In order to skip 999990 records and display record #999990 to record #1000000, the search provider must find and sort the first 999990 records. Even if you have skipped through those 999990 in batches of 10, each Skip() is treated as a new query with increasingly severe performance implications. This is true for Solr and Azure Search.

The .GetBatchEnumerator() and .GetBatchEnumeratorSync() methods return a cursor which is used to iterate through sequential batches. Each batch is associated with a cursor mark. A cursor mark is represented by a byte array, and bookmarks your location when iterating through a large result set. These methods improve performance by skipping results that you have already seen - in other words, when displaying record #999990 to record #1000000, the first 999990 records are ignored:

cursor-mark-11.png

Keep the following in mind when using cursor marks:

  • Cursor marks do not allow you to skip directly to record #999990 - you must iterate through all results until you get to this point. The performance gain comes from not having to find and sort previously seen results each time a new batch is returned.

  • Cursor marks are short-lived and are not resistant to model changes. Therefore, you should not rely on cursor marks in a pagination UI. For example, if you iterate through 30 batches, do not persist 30 cursor marks and treat them as page numbers.

Cursor marks can be used to resume a search operation if an error occurred. For example, if you are iterating through 1000 batches and an error occurs during batch 999, persisting the cursor mark at that point allows you to return to resume at batch 999.

If you must support deep pagination in a user interface, consider using infinite scrolling as this eliminates the need to support paging backwards and forwards through a result set.

For more information, see  Pagination of results in Solr - particularly the section under Performance Problems with “Deep Paging”.

Cursor marks become particularly important in a sharded environment, where each shard must return a batch of 100 results. Three batches of 100 results are then aggregated and the top 100 are returned to the client:

cursor-mark-21.png

It is acceptable to use the .Skip() and .Take() method if you want to interate through a small subset of contacts or interactions.

using Sitecore.XConnect.Collection.Model;
using System.Linq;
using Sitecore.XConnect;
using Sitecore.XConnect.Client;

namespace Documentation
{
    public class SearchInteractionSkipTake
    {
        // Async example
        // Example query string: searchresults?page=4&pageSize=10
        public async void ExampleAsync(int pageSize, int page)
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    var skip = pageSize * (page - 1);

                    IAsyncQueryable<Sitecore.XConnect.Interaction> queryable = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome")
                            .Skip(skip)
                            .Take(pageSize);

                    var results = await queryable.ToSearchResults();

                    var count = results.Count; // Actual count of all results

                    var interactions = await results.Results.Select(x => x.Item).ToList();

                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception
                }
            }
        }

        public void ExampleSync(int pageSize, int page)
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    var skip = pageSize * (page - 1);

                    IAsyncQueryable<Sitecore.XConnect.Interaction> queryable = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome")
                            .Skip(skip)
                            .Take(pageSize);

                    // NOTE: No sync version of ToSearchResults exists
                    // and it is the only way to get a count of actual results
                    var results = Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(queryable.ToSearchResults);

                    var items = results.Results.Select(x => x.Item); // Interactions
                    var count = results.Count; // Total count
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception
                }
            }
        }
    }
}

Note

Always call Skip().Take(), never Take().Skip().

The following example demonstrates how to use .GetBatchEnumerator() / .GetBatchEnumeratorSync() when iterating through a large set of results.

using Sitecore.XConnect.Collection.Model;
using System.Collections.Generic;
using System.Linq;
using Sitecore.XConnect;
using Sitecore.XConnect.Client;

namespace Documentation
{
    public class SearchInteractionCursorBasic
    {
        // Async example
        public async void ExampleAsync()
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    var results = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome");

                    var enumerator = await results.GetBatchEnumerator(200);

                    var numberOfBatches = 0;

                    // Enumerate through batches of 200
                    while (await enumerator.MoveNext())
                    {
                        numberOfBatches++;

                        // Loop through interactions in current batch
                        foreach (var interaction in enumerator.Current)
                        {
                            // Do something for each interaction
                        }
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception
                }
            }
        }

        // Sync example
        public void ExampleSync()
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                try
                {
                    var results = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome");

                    var numberOfBatches = 0;

                    var enumerator = results.GetBatchEnumeratorSync(200);

                    // Enumerate through batches of 200
                    while (enumerator.MoveNext())
                    {
                        numberOfBatches++;

                        // Loop through interactions in current batch
                        foreach (var interaction in enumerator.Current)
                        {
                            // Do something for each interaction
                        }
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception
                }
            }
        }
    }
}

The following example demonstrates how to persist a bookmark and use it to resume a search query in the event that an exception occurs:

using Sitecore.XConnect.Collection.Model;
using System.Collections.Generic;
using System.Linq;
using Sitecore.XConnect;
using Sitecore.XConnect.Client;

namespace Documentation
{
    public class SearchInteractionCursor
    {
        // Async example
        public async void ExampleAsync(byte[] bookmark)
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                byte[] bookmarkSave = null;

                try
                {
                    var results = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome");

                    var enumerator = await results.GetBatchEnumerator(200);

                    var numberOfBatches = 0;

                    // Enumerate through batches of 200
                    while (await enumerator.MoveNext())
                    {
                        bookmarkSave = enumerator.GetBookmark();
                        numberOfBatches++;

                        // Loop through interactions in current batch
                        foreach (var interaction in enumerator.Current)
                        {
                            // Do something for each interaction
                        }
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Retry using saved bookmark
                    ExampleAsync(bookmarkSave);
                }
            }
        }

        // Sync example
        public void ExampleSync(byte[] bookmark)
        {
            using (Sitecore.XConnect.Client.XConnectClient client = Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient())
            {
                byte[] bookmarkSave = null;

                try
                {
                    var results = client.Interactions
                        .Where(x => x.GetFacet<WebVisit>(CollectionModel.FacetKeys.WebVisit).Browser.BrowserMajorName == "Chrome");

                    var numberOfBatches = 0;

                    var enumerator = results.GetBatchEnumeratorSync(200);

                    // Enumerate through batches of 200
                    while (enumerator.MoveNext())
                    {
                        bookmarkSave = enumerator.GetBookmark();
                        numberOfBatches++;

                        // Loop through interactions in current batch
                        foreach (var interaction in enumerator.Current)
                        {
                            // Do something for each interaction
                        }
                    }
                }
                catch (XdbExecutionException ex)
                {
                    // Handle exception
                    ExampleSync(bookmarkSave);
                }
            }
        }
    }
}