Create a JavaScript trigger to get a set of URLs

On the menu bar, click Sources , and select the crawler source you created.

On the Source Settings page, next to Triggers, click Edit.

In the Trigger Type drop-down menu, click JS.

In the Trigger Source field, paste the JavaScript function.

For example, paste:

function extract(){
  var arr = [];
  var max = 1000;
  for (let i = 0; i < max; i+=100) {
    var url = "https://sampleDomain.com/api/v1/contents?last="+(i+100)+"&stFrom="+(i+1)";
    arr.push({"url": url});
  }
  return arr;
}

This sample function uses a for loop to iterate over a URL in increments of 100. Each iteration results in one URL. In this example, there will be 11 loops, resulting in 10 URLs. This is because, on the eleventh loop, i=1100, which is greater than the max of 1000, and the function stops.

This function gives the following output:

"https://sampleDomain.com/api/v1/contents?last=100&stFrom=1"
"https://sampleDomain.com/api/v1/contents?last=110&stFrom=101"
"https://sampleDomain.com/api/v1/contents?last=210&stFrom=201"
...
"https://sampleDomain.com/api/v1/contents?last=1010&stFrom=901"

In the Timeout field, enter the time in milliseconds that the crawler waits for a response from the JS function.

For example, enter 5000.

Click Save.