Robot detection overview

Current version: 10.1

Sitecore robot detection functionality filters out unwanted interactions from automated browsers and robots. This ensures that only genuine contacts are registered in the xDB. Robot detection classifies a contact based on an IP address, user agent, or verified human behavior.

There are two main ways to identify robots:

  • Use the request IP address or user agent.

  • Use the robot detection functionality to check for human behavior.

Filtering contacts by IP address or user agent

The Device Detection component detects user agents including well-known bots, crawlers, spiders, and robots. If a contact visits your website with a user agent that the Device Detection component recognizes as a robot, crawler, or spider, then the request to view the page is not tracked.

TheSitecore.Analytics.ExcludeRobots.config file contains an exclude list of IP addresses and custom user agents to filter unwanted contacts. If a contact visits your website from one of the excluded IP addresses or user agents on the exclude list, then the request to view the page is not tracked. You can manually add IP addresses and custom user agents to this list.

Identifying human contacts

The Sitecore robot detection component is enabled by default. It classifies all visitors on their first visit as robots, using classification code 925.

It next performs a series of tests to determine whether a visitor is a human or a robot. For example, the VisitorIdentification.js JavaScript file tests if a visitor is human by detecting computer mouse movements, or touch screen actions on a mobile phone or tablet, that the visitor performs. Human contacts are classified using classification code 0.

Important

Ensure that you include the robot detection component in your main layout so that it renders with every page view.

Contact classification

The Sitecore.Analytics.Core.ContactClassification class contains helper methods that allow the robot detection component to check the classification of a contact:

  • IsHuman()

  • IsRobot()

  • IsAutoDetectedRobot()

The robot detection component

The Sitecore robot detection component consists of a pipeline processor, an event handler, a JavaScript file, and several robot detection code classes.

The Sitecore.Analytics.Tracking.RobotDetection.config file contains several configuration settings in addition to the following pipeline and event handler:

  • Sitecore.Analytics.RobotDetection.Pipeline.InitializeTracker.Robots

  • Sitecore.Analytics.RobotDetection.Media.MediaRequestEventHandler

These components are triggered when a contact or robot begins to interact with your website, initiating the process that identifies and prevents robots from further accessing your website.

Do you have some feedback for us?

If you have suggestions for improving this article,