Configure robot detection functionality

Version: 9.0

You can identify and block robots by configuring the exclude robots configuration file to filter out unwanted visitors by user agent or IP address or by using the robot detection component to identify human behavior.

There are several settings related to robot detection that you can configure.

Excluding user agents and IP addresses

The Sitecore.Analytics.ExcludeRobots.config file contains a list of IP addresses and user agents to exclude. If a visitor comes to your website from one of the IP addresses in the exclude list, or is a user agent in the exclude list, then the request to view the page is ignored and not tracked.

You can edit this list manually by adding the user agents that you want to block under the <analyticsExcludeRobots> node and IP addresses under the <excludedIPAddresses> node.

To filter by user agent:

Navigate to the App_Config\Sitecore\Marketing.Tracking folder and open the Sitecore.Analytics.ExcludeRobots.config file.
Under the <analyticsUserAgents> node, enter each user agent that you want to block on a separate line. For example:
RequestResponse
```
<excludedUserAgents>
  UserAgent 1.0
  UserAgent 2.0
  UserAgent 2.0
</excludedUserAgents>
```

To filter by IP address:

Navigate to the App_Config\Sitecore\Marketing.Tracking folder and open the Sitecore.Analytics.ExcludeRobots.config file.
Under the <excludedIPAddresses> node, enter each IP address that you want to block on a separate line. For example:
RequestResponse
```
<excludedIPAddresses>
  10.1.2.3
  12.9.2.2
  35.2.5.4
</excludedIPAddresses>
```
Note

Ensure that IP addresses conform to the following supported formats:

IP address example: 10.2.3.4

IP range example: 10.1.2.3 - 10.1.2.30

Enabling robot detection

The robot detection component is enabled by default but to implement it fully you must ensure that you have added the visitor identification control to the layout of each page on your website.

Note

The visitor identification control is stored in the Websitelayoutssystem folder.

To use the robot detection component to identify human behavior:

Add the VisitorIdentification control to the layout of all the pages on your website.

The Sitecore Sample layout contains an example of how to include a reference to the VisitorIdentification control in your page layout.

RequestResponse

<head runat="server">
<title>Welcome to Sitecore</title>
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta name="CODE_LANGUAGE" content="C#" />
<meta name="vs_defaultClientScript" content="JavaScript" />
<meta name="vs_targetSchema" content="http://schemas.microsoft.com/intellisense/ie5" />
<link href="/default.css" rel="stylesheet" />
<sc:VisitorIdentification runat="server" />
</head>

Note

If you are using MVC layouts, use the @Html.Sitecore().VisitorIdentification() helper method to render the visitor identification control.

Disabling robot detection

Robot detection is enabled by default. However, you may need to disable robot detection, for example, if you are running automated performance tests.

To disable robot detection:

Navigate to the App_Config\Sitecore\Marketing.Tracking folder and open the Sitecore.Analytics.Tracking.config file.
In the Sitecore.Analytics.Tracking.config file, navigate to the Analytics.AutoDetectBots node.
Change the value of this setting from true to false.
RequestResponse
```
<setting name="Analytics.AutoDetectBots" value="false" />
```

Changing the default session time out

You can change the default session time out setting to minimize the time that interactions initiated by robots are stored in the session database. The default setting is one minute. For human contacts the default setting is 20 minutes.

To change the default session time out:

Navigate to the App_Config\Sitecore\Marketing.Tracking folder and open the Sitecore.Analytics.Tracking.config file.
In the Sitecore.Analytics.Tracking.config file, navigate to the Analytics.Robots.SessionTimeout setting.
To change the default setting, enter a different time value in minutes in the value attribute:
RequestResponse
```
<setting name=”Analytics.Robots.SessionTimeout” value=”1” />
```

Generating statistics on robot visits

By default, the IgnoreRobots setting is set to true so that robot visits are ignored and not saved to xDB. When you change this setting to false, then all visits by robots are saved.

To generate statistics on robot visits:

Navigate to the App_Config\Sitecore\Marketing.Tracking folder and open the Sitecore.Analytics.Tracking.config file.
In the Sitecore.Analytics.Tracking.config file, navigate to the Analytics.Robots.IgnoreRobots setting.

Change the value of this setting to false:

RequestResponse

<setting name="Analytics.Robots.IgnoreRobots" value="false" />

You can also generate statistics on the following performance counters:

Human requests
Robot requests
Malicious robot requests

Note

To change or customize the default robot detection logic you need to edit both the Sitecore.Analytics.RobotDetection.config file and the web.config file. The Sitecore.Analytics.RobotDetection.config file is the main configuration file for the robot detection component and the web.config file contains the media request session module for the robot detection component.

Do you have some feedback for us?

If you have suggestions for improving this article,