Crawler troubleshooting

If a crawl does not start and no URLs are processed, typically the crawler is unable to retrieve or process the sitemap within its configured limited. This usually happens when the trigger timeout is too low for the target site. To allow sufficient time for sitemap retrieval and processing, increase the trigger timeout from the default 1,000 ms (1 second) to 10–60 seconds.

If the crawler reports errors such as context deadline exceeded (Client.Timeout exceeded while awaiting headers), it indicates that the target site did not respond within the allowed time. The web crawler timeout is intentionally generous and should not normally be reached. Most pages should respond within a few seconds. Even in scenarios with large resources, such as large PDF downloads, responses should complete within minutes.

When pages approach this timeout repeatedly, the issue is usually not related to the crawler, but instead relates to server performance, network restrictions, or bot‑blocking by the site or Content Delivery Network (CDN). Increasing the crawler timeout does not resolve these issues, and leads to inefficient resource usage.

To fix the error:

When validating an individual URL works but a full crawl fails, the error is usually due to request volume. Validating a single URL is a lightweight request, whereas a crawl generates hundreds or thousands of requests over a short period, which is more likely to trigger security controls.

To fix the error:

Websites are not required to return detailed error responses when blocking automated traffic. Many websites intentionally delay or provide obscure errors as part of bot‑mitigation strategies. In these cases, the HTTP status code logged by the crawler provides the most reliable diagnostic message.

To fix the error:

If pages are marked as visited but show Indexed Documents = 0, or Visited > 0 but Crawled = 0, the crawler is able to load the pages but cannot extract content. This is often caused by missing or outdated extraction selectors, JavaScript rendering restrictions, or host/CDN anti‑bot rules. This issue is frequently observed on platforms such as Vercel, where headless crawlers may receive incomplete or restricted content.

For example, an extractor may attempt to read content such as:

$(".field-productshortdescription").html();

If the required element is not present at extraction time, the extractor returns no results, and the page is not crawled or indexed. To fix the error:

Optimized (delta) crawling is attempted based on the sitemap configuration. If the number of failed URLs exceeds the allowed threshold, the delta crawl fails and the next run automatically defaults to a full crawl. If fewer pages are indexed than expected, or a delta crawl defaults to a full crawl:

To avoid indexing incomplete or blocked pages, configure extractor validation logic to require the presence of a mandatory field. If the field is missing, extraction fails and the document is excluded from indexing. This ensures that only fully rendered and accessible pages are ingested.

When crawlers run frequently (for example, on an hourly schedule), repeated requests can increase the likelihood of blocking. If content changes infrequently, reducing the crawl schedule to a daily run can help lower request volume and improve stability. A sitemap index-based crawler with daily scheduling is appropriate for sites where content changes infrequently.

To keep crawls stable and efficient:

When troubleshooting failed crawls: