Walkthrough: Creating and running a pipeline

Version: 8.0

This walkthrough is part three of Synchronizing data from a text file to Sitecore walkthrough series and describes how to create a pipeline that can run a data synchronization between a text file and Sitecore.

The walkthrough describes how to:

  • Add a pipeline that reads the source file.

  • Add a pipeline step to read data from a source file.

  • Add a pipeline step to iterate the data read from a source file.

  • Add a pipeline to sync a single row from the source file.

  • Add a pipeline step to resolve the target item.

  • Add a pipeline step to apply mappings.

  • Add a pipeline step to update the target item.

  • Add a pipeline batch

  • Test a pipeline batch

Add a pipeline to read the source file

You must implement a pipeline that reads data from the text file, iterates the data, and passes the data to the pipeline that handles a single row of data.

To add a pipeline to read the source file:

  1. In the Content Editor, navigate to /sitecore/System/Data Exchange/File System Provider Test Tenant/Pipelines.

    Tenant Pipelines folder in the content tree

  2. In the right-hand pane, click Pipeline. Name the new item Read City Info from File Pipeline.

  3. Drag and drop the new item to the first position in the Pipelines folder.

    Moving the Read City Info from File pipeline in the content tree
Note

The order of the pipelines in the folder is not important. However, when a pipeline calls another pipeline, it is helpful to place the pipeline that calls the other pipeline before the pipeline it calls.

Add a pipeline step to read data from a source file

You must configure a pipeline step that can read the data from the text file specified by the text file endpoint. The pipeline step must also put a pointer to the data in a location where subsequent pipeline steps can access it.

To add a pipeline step to read data from a source file:

  1. In the Content Editor, navigate to the pipeline item, for example, Read City Info from File Pipeline.

    Read City Info from File pipeline in the content tree

  2. In the right-hand pane, click Read Text File Pipeline Step. Name the new item Read Text File.

  3. In the content tree, select the new item.

    Read Text File pipeline step in the content tree

  4. In the right-hand pane, specify the following field values:

    Field

    Value

    Endpoint From

    File System/City Information File

  5. Save the item.

Add a pipeline step to iterate the data read from a source file

The first step in this pipeline adds a pointer to a location where subsequent pipeline steps can access it. You must configure the second step to access the data so it can iterate over the lines read from the text file, and pass the data to the other pipeline that handles the data.

To add a pipeline step to iterate data:

  1. In the Content Editor, navigate to the pipeline item City Info from File to City Info Item Sync Pipeline.

    Read City Info from File pipeline in the content tree

  2. In the right-hand pane, click Iterate Data and Run Pipelines Pipeline Step. Name the new item Iterate Rows from Text File and Run Pipeline.

  3. In the content tree, select the new item.

    Iterate Rows from Text File and Run Pipeline pipeline step in the content tree

  4. In the right-hand pane, in the Settings section, specify the following field values:

    Field

    Value

    Pipelines

    Pipelines/City Info from File to City Info Item Sync Pipeline

    Iterable Data Location

    Pipeline Context Iterable Data

    Data Location

    Pipeline Context Source

  5. Save the item.

  6. Drag and drop the new item to the second position in the Pipelines folder.

    Move the Iterate Rows from Text File and Run Pipeline pipeline step in the content tree

Add a pipeline to sync a single row from the source file

The synchronization process you are modeling consists of two pipelines. The first pipeline reads data from a source object, which is a text file. The second pipeline handles a single row from the text file.

Because the first pipeline calls the second pipeline, you must configure the second pipeline before you can configure the first.

To add the pipeline that handles a single row from the text file:

  1. In the Content Editor, in the content tree, select the tenant. Navigate to Pipelines.

    Tenant pipelines in the content tree

  2. In the right-hand pane, on the Folder tab, click Pipeline to add a new item and name it City Info from File to City Info Item Sync Pipeline.

Add a pipeline step to resolve the target item

The first pipeline step must check if there is an existing Sitecore item that matches the value from the text file row.

If the pipeline finds a matching item, it must set that item as the target object in the pipeline context. This means that during the mapping process, the pipeline updates the matching item with the new values.

If the pipeline does not find a matching item, it must create a new item, and set the new item as the target object in the pipeline context.

To set up these actions in the pipeline:

  1. In the Content Editor, navigate to the pipeline item you want to create a step for, for example, City Info from File to City Info Item Sync Pipeline.

    City Info From File to City Info Item Sync pipeline in the content tree

  2. In the right-hand pane, click Resolve Sitecore Item Pipeline Step. Name the new item Resolve City Info Item. Select the new item.

    Resolve City Info item in the content tree

  3. In the right-hand pane, specify the following field values:

    Field

    Value

    Template for New Item

    Templates/User Defined/City Information

    Item Name Value Accessor

    Data Access/Value Accessor Sets/Providers/File System/City Information File Fields/City from City Information File

    Endpoint From

    Sitecore/Sitecore Database Endpoint

    Identifier Value Accessor

    Data Access/Value Accessor Sets/Providers/File System/City Information File Fields/Identifier from City Information File

    Identifier Object Location

    Pipeline Context Source

    Resolved Object Location

    Pipeline Context Target

    Parent for Item

    sitecore/content/Cities

    Matching Field Value Accessor

    Data Access/Value Accessor Sets/Providers/Sitecore/City Information Item Fields/Identifier on City Information Item

  4. Save the item.

Add a pipeline step to apply mappings

The first step in this pipeline ensures that a target object is available, either by finding an existing item or creating a new one. In the second step, you must use the value mapping set you configured in Add Value Mappings to map values from the source object to the target object.

To add a pipeline step to apply mappings:

  1. Navigate to the pipeline item City Info from File to City Info Item Sync Pipeline.

    City Info from File to City Info Item Sync pipeline in the content tree

  2. In the right-hand pane, click Apply Mapping Pipeline Step. Name the new item Apply Mapping. Select the new item.

    Apply Mapping item in the content tree

  3. In the right-hand pane, set the following field value:

    Field

    Value

    Mapping Set

    Value Mapping Sets/Row from City Information File to City Information Item

  4. Save the item.

  5. Drag and drop the item to the second position in pipeline.

    Moving the Apply Mapping item in the content tree

Add a pipeline step to update the target item

After mappings are applied, the changes to the target item must be saved to the Sitecore database.

To add a pipeline step to update the target item:

  1. In the Content Editor, navigate to the pipeline item you want to add the step to, for example, City Info from File to City Info Item Sync Pipeline.

  2. In the right-hand pane, click Update Sitecore Item Pipeline Step. Name the new item, for example, Update Sitecore Item. Select the new item.

    Update Sitecore Item pipeline step in the content tree

  3. In the right-hand pane, specify the following field values:

    Field

    Value

    Item Location

    Pipeline Context Target

    Endpoint To

    Sitecore/Sitecore Database Endpoint

  4. Save the item.

  5. Drag and drop the new item to the third position in the pipeline.

    Move the Update Sitecore Item pipeline step in the content tree

Add a pipeline batch

To run a pipeline, you must configure a pipeline batch. This is basically the runtime for a pipeline.

To configure a pipeline batch:

  1. In the Content Editor, navigate to /sitecore/System/Data Exchange/File System Provder Test Tenant/Pipeline Batches.

    Tenant Pipeline Batches in the content tree

  2. In the right-hand pane, click Pipeline Batch. Name the new item, for example, City Info from File to City Info Item Sync Pipeline Batch.

  3. In the content tree, select the new item.

    City Info from File to City Info Item Sync pipeline batch in the content tree

  4. In the right-hand pane, specify the following field value:

    Field

    Value

    Pipelines

    Pipelines/Read City Info from File Pipeline

  5. Save the item.

Test a pipeline batch

You can run a pipeline batch in a number of ways. The easiest way is to run it manually.

To test a pipeline batch:

  1. In the Content Editor, navigate to /sitecore/System/Data Exchange/File System Provder Test Tenant/Pipeline Batches/City Info from File to City Info Item Sync Pipeline Batch.

    Pipeline batches in content tree

  2. On the ribbon, click Run Pipeline Batch.

    Run Pipeline Batch option in menu ribbon

  3. Click OK.

    Pipeline batch started message
    Note

    In this example, the text file only contains a few rows. In addition, the logic in the pipelines is very limited. This means the pipeline batch will finish almost instantly.

    You can tell if the pipeline batch is running by checking if the Stop Pipeline Batch button is enabled. This button does not automatically refresh, so you might have to click the pipeline batch to refresh the UI.

  4. In the content tree, navigate to /sitecore/content/Cities.

    Content in content tree

    You can see an item for each of the rows in the text files.

  5. Click the item you want to review, for example, the Copenhagen item. You can see the values from the row in the text file that corresponds to the item.

    Data item details

Do you have some feedback for us?

If you have suggestions for improving this article,