Scaling and configuring the Sitecore Cortex Processing Engine

Version: 10.4

The Sitecore Cortex™ Processing Engine (Processing Engine) is a worker role that performs data processing tasks at scale.

Installation

Use the Sitecore Installation Framework (SIF) or the Sitecore Azure Toolkit (SAT) to install the Processing Engine. Refer to the installation guide for your version of the platform for more information.

Scaling the Processing Engine

The following table describes the ways you can scale the Sitecore Cortex Processing Engine:

Scaling option

Supported

Notes

Combined instance

Partially

The Processing Engine is a worker role and cannot be combined with any other XP services. For example, you cannot combine the Processing Engine and the Marketing Automation Engine into a single Windows Service or WebJob.

Standalone instance

Yes

You can deploy the Processing Engine as:

  • A Windows Service that runs on the same server as the Sitecore Cortex Processing service role or any combination that includes these roles, such as the XP Single topology.

  • An Azure WebJob that runs in the same context as the combined Table Storage and Blob Storage web application or any combination that includes these roles, such as the XP Single topology.

High availability and failover

Yes

You can fail over to a passive instance of the Processing Engine or run multiple active instances.

Note

The Processing Engine does not have a health check API endpoint. You must use an external tool to monitor the WebJob or Windows Service health.

Horizontal scaling for load distribution

Yes

Multiple instances of the Processing Engine can process tasks in parallel. All instances must connect to one centrally located Processing Tasks database and one centrally located Processing Storage database.

The following general recommendations apply to all scaling scenarios:

  • If you scale up the host machine by adding more cores, the number of parallel task agents will scale automatically . You can add additional task agents manually.

    Note

    Only distributed tasks can be processed in parallel by multiple nodes; deferred tasks are processed by a single task agent.

Default topologies and packages

The following tables list the topologies that include the Processing Engine and how the role is packaged by default.

Sitecore Installation Framework

The Processing Engine is available in the following default topologies for the Sitecore Installation Framework:

Topology

Web Deploy Packages

Description

XP Single

Sitecore 9.X.X rev. XXXXXX (OnPrem)_xp0xconnect.scwdp.zip

The Processing Engine is bundled with all other XP service roles. The engine is installed as a Windows Service and is physically located in the App_data\jobs\continuous\ProcessingEngine folder.

XP Scaled

Sitecore 9.X.X rev. XXXXXX (OnPrem)_xp1cortexprocessing.scwdp.zip

The Processing Engine is bundled with the Table Storage and Blob Storage services. The engine is installed as a Windows Service and is physically located in the App_data\jobs\continuous\ProcessingEngine folder.

Sitecore Azure Toolkit

The Processing Engine is available in the following default topologies for the Sitecore Azure Toolkit:

Topology

Web Deploy Packages

Description

XP Single

Sitecore 9.X.X rev. XXXXXX (Cloud)_xp0xconnect.scwdp.zip

The Processing Engine is bundled with all other XP service roles. The engine runs as an Azure WebJob in the context of the combined application.

xDB Single

XP Scaled

Sitecore 9.X.X rev. XXXXXX (Cloud)_xp1cortexprocessing.scwdp.zip

The Processing Engine is bundled with the Sitecore Cortex Processing service. The engine runs as an Azure WebJob in the context of the Marketing Automation Operations application.

xDB Scaled

Do you have some feedback for us?

If you have suggestions for improving this article,