---
title: "Crawler specifications"
slug: "crawler-specifications"
abstract: "Detailed specifications of different crawlers available in Sitecore Search."
---
# Crawler specifications

The following table shows which feature each crawler type supports:

| Feature | | **Web crawler** | **Advanced web crawler** | **API crawler** |
| --- | --- | --- | --- | --- |
| Multiple entities | Extract attribute values from more than one entity | No | Yes | Yes |
| Content that can be crawled and parsed | HTML | Yes | Yes | Yes |
| | Microsoft Office formats | Yes | Yes | No |
| | PDF | Yes | Yes | No |
| | JSON | No | No | Yes |
| General Settings | Specify allowed domains | No | Yes | No |
| | Define a pattern to exclude some URLs | Yes | Yes | Yes |
| | Specify maximum crawler depth | Yes | Yes | Yes |
| | Specify maximum URLs that can be crawled | Yes | Yes | Yes |
| | Specify the number of workers to work in parallel | No | Yes | Yes |
| | Specify crawler timeouts | No | Yes | No |
| | Add headers to the crawler if original content expects a header | Yes | Yes | Yes |
| | Render JavaScript and crawl it in addition to the page source | No | Yes | n/a |
| Starting the crawl (Trigger) | Use a request URL | Yes | Yes | Yes |
| | Use a sitemap | Yes | Yes | n/a |
| | Use a sitemap index | Yes | Yes | n/a |
| | Use a JavaScript function | No | Yes | Yes |
| | Use an RSS feed | No | Yes | n/a |
| Extracting attributes (Document Extractor) | Use an XPath expression | Yes | Yes | n/a |
| | Use a CSS expression | No | Yes | n/a |
| | Use a JavaScript function | No | Yes | Yes |
| | Use JSONPath | No | No | Yes |
| | Match a specific URL pattern before extracting an attribute | No | Yes | Yes |
| | Create multiple rules to extract an attribute and then prioritize these rules | No | Yes | Yes |
| | Specify entity-based rules to extract attributes | No | Yes | Yes |
| Schedule scans to keep index documents up to date with your original content | | Yes | Yes | Yes |
| Handle original content with multiple locales and languages | | No | Yes | Yes |
| Handle original content that requires authentication | | No | Yes | Yes |
| Add additional starting points not covered by the trigger (Request Extractor) | | No | Yes | Yes |
| Use the Ingestion API to make incremental updates | | No | Yes | Yes |
| Use entity-based tags | | No | Yes | Yes |