Configuring request extractors

A request extractor creates an additional list of URLs for the crawler to crawl. If you find that the crawler does not reach all the content you need to index by following the original starting point, the trigger , use a request extractor. The request extractor is a JavaScript function that uses the trigger output as its input.

For example, you have a sitemap that contains all HTML pages in your source content, and you use the sitemap URL as the trigger. But you also have PDF content embedded within some pages. The sitemap does not contain the URLs for the PDFs. In this case, configure request extractors to generate URLs to crawl only the PDF content.

Note

Request extractors are very important when you configure an API crawler. For an API crawler, triggers return JSON and not URLs. To handle this, configure a request extractor to use the output of the trigger and return URLs or API endpoints for the API crawler to crawl.

To configure a request extractor, you add a JavaScript (JS) function that returns a list of URLs or API endpoints to crawl.

Note

Configure the following settings to define request extractors for a crawler:

Setting	Description
Name	A meaningful name for the request extractor.
URLs to Match	The pattern that defines the URLs to which this extractor and its rules apply. You can use a regular expression or a glob expression. Use this field to create different extractors for different areas of your source content. This is an optional setting.
JS Source	JavaScript function that generates URLs or API endpoints for the crawler to crawl. The JavaScript function you define must: Use Cheerio syntax. That is, it must use this format: RequestResponse `function extract(request, response) { $ = response.body; }` Return an array of objects.

Do you have some feedback for us?

If you have suggestions for improving this article,