Processing service workflow
The processing service has three workflows:
Getting work
This is the first step in processing interactions. The processing engine runs on a schedule, and periodically checks the database to see if any interactions are closed and ready for processing. The process is:
-
The timer triggers the system to check for closed interactions.
-
The system checks the database for closed interactions and retrieves a batch of interactions with their events.
-
The system starts a parallel process that periodically sets the timeout (schedule) of the interactions.
-
The system sends the interactions (with linked events) to the worker pool for processing.
-
The worker pool distributes a single interaction to a single worker.
-
The worker processes the interaction.
-
The worker notifies the system that the interaction is complete.
-
The system marks the interaction to be completed so the SetTimeout worker can see it.
-
If the number of remaining interactions in a batch is less than a set threshold (for example, 10% of the original batch size), repeat from step 2.
-
If there are no more interactions, the timer is set to trigger the worker after a set timeout.
Process interaction
This details how a single interaction is processed to completion:
-
A single tracking interaction is received to be processed.
-
The worker checks if a pre-filter has been defined.
-
If the filter has been defined, run it.
-
If the filter excludes the interaction, stop and mark the interaction as complete.
-
The worker checks if an enrichment pipeline has been defined, and if it has, runs it.
-
The worker checks if a post-filter has been defined.
-
If the filter has been defined, run it.
-
If the filter excludes the interaction, mark the interaction as complete, and stop.
-
The worker adds the interaction to the xConnect context.
-
The worker adds the interaction facets to the xConnect context.
-
The worker sends the xConnect context.
-
The worker marks the interaction as complete.
Set timeout
When an interaction is checked out, the scheduled property of the interaction is set to some time in the future in order to provide recovery and continuity of processing. If the worker fails before completing the processing, another worker (either a single worker or a separate processing engine instance) may pick up the work after the scheduled DateTime has passed.
To prevent other workers from attempting to process the same interaction, the timeout must continually be pushed into the future by some small amount (such as 45 seconds).
This process runs in parallel with the Getting Work process:
-
The system starts a SetTimeout worker.
-
SetTimeout waits for a number of seconds. The number of seconds to wait is determined by system settings. Typically, the value is half of the timeout value that is set when checking out the interactions.
-
The worker checks if any interactions from the batch have been marked as completed, and deletes any completed interaction parts from the database.
-
For all non-completed interactions, the worker calls the database to set the timeout to the value that was used when the interactions were checked out.