Create a JavaScript trigger to access a URL that requires pagination

A trigger is the starting point that Sitecore Search uses to look for content to index. Define a JavaScript trigger when you want to use a JavaScript function that returns a set of URLs.

To create a JavaScript trigger:

Log in to the Customer Engagement Console (CEC), click Sources , and select the crawler source you created.
On the Source Settings page, next to Triggers, click Edit.
In the Trigger Type drop-down menu, click JS.

In the Trigger Source field, paste the JavaScript function.

For example, paste:

RequestResponsejavascript

function extract(){
  var arr = [];
  var max = 1000;
  for (let i = 0; i < max; i+=100) {
    var url = "https://sampleDomain.com/api/v1/contents?last="+(i+100)+"&stFrom="+(i+1)";
    arr.push({"url": url});
  }
  return arr;
}

This sample function uses a for loop to iterate over a URL in increments of 100. Each iteration results in one URL. In this example, there will be 11 loops, resulting in 10 URLs. This is because, on the eleventh loop, i=1100, which is greater than the max of 1000, and the function stops.

This function gives the following output:

RequestResponseshell

"https://sampleDomain.com/api/v1/contents?last=100&stFrom=1"
"https://sampleDomain.com/api/v1/contents?last=110&stFrom=101"
"https://sampleDomain.com/api/v1/contents?last=210&stFrom=201"
...
"https://sampleDomain.com/api/v1/contents?last=1010&stFrom=901"

In the Timeout field, enter the time in milliseconds that the crawler waits for a response from the JS function.

For example, enter 5000.
Click Save.