1. Sitecore Content SDK

Lightweight tracking

Version:

The Sitecore Content SDK identifies bots and supports lightweight tracking for Next.js applications. The SDK identifies requests from web crawlers and other automated agents, tracks them separately from real users, and ensures that bot traffic does not skew analytics or trigger unnecessary processing like personalization.

Bot detection runs in the Next.js proxy and integrates with Sitecore events to record bot page views on the server.

Note

Lightweight tracking is only available in Content SDK versions 2.1.0 or later. See What's new in Content SDK 2.1 for more information on updating your app to version 2.1.0.

Flow of bot detection

When an incoming request is identified as a bot:

  • The SDK sends a single page view event to the Sitecore Events API during middleware execution.

  • The request is marked as bot traffic, allowing downstream code to recognise it.

  • Client-side tracking and other SDK features can skip work that is not relevant for crawlers.

This keeps analytics data clean and reduces unnecessary overhead for bot traffic. Analytics contain a single, server-side page view for the bot request. The browser-side pageView function automatically skips sending an additional event. Bot page views are tracked by the BotTrackingProxy.

For real visitors, the SDK tracks page views in the browser using the pageView function from the @sitecore-content-sdk/events package.

Using the BotTrackingProxy

To enable bot detection, add BotTrackingProxy to your Next.js proxy chain. It must be the first proxy so that bot information is available to all following proxies.

import {
  defineProxy,
  BotTrackingProxy,
  /* other proxies */
} from '@sitecore-content-sdk/nextjs/proxy';
import scConfig from 'sitecore.config';
import sites from '.sitecore/sites.json';

export default function proxy(req, event) {
  const botTracking = new BotTrackingProxy({
    ...scConfig.api.edge,
    sites,
    fetchEvent: event, // enables background tracking via waitUntil
  });

  return defineProxy(botTracking /*, locale, multisite, ... */).exec(req);
}

The proxy handles the following automatically:

  • Detecting bots from the incoming HTTP request.

  • Sending a page view event for bots to the Sitecore Events API.

  • Setting a cookie so that the client-side pageView function and other proxies can react accordingly.

Using fetchEvent to send events in the background

Sending a page view to Sitecore Events is a network call. Waiting for this call to complete before returning a response can slow down bot requests. Next.js provides a NextFetchEvent object with a waitUntil method that allows middleware to perform work in parallel with the response.

When you pass fetchEvent: event to BotTrackingProxy:

  • The response is returned immediately, without waiting for the Events API call.

  • The bot tracking request runs to completion in the background.

If you omit fetchEvent, the proxy falls back to awaiting the event sent inline, which adds latency to the request.

Bot traffic and personalization

BotTrackingProxy also enables other proxies to skip work for crawler requests. The PersonalizeProxy uses this out-of-the-box, meaning, personalization is disabled for bot traffic by default and doesn't trigger variant resolution.

If you need personalization to run for bot requests, you can opt in explicitly:

const personalize = new PersonalizeProxy({
  ...
  skipForBot: false, // default: true
});

In most scenarios, you should keep the default behaviour so that bot traffic stays inexpensive and personalization metrics remain accurate.

If you have suggestions for improving this article, let us know!