Crawler specifications

Multiple entities

Extract attribute values from more than one entity

No

Yes

Yes

Content that can be crawled and parsed

HTML

Yes

Yes

Yes

Microsoft Office formats

Yes

Yes

No

PDF

Yes

Yes

No

JSON

No

No

Yes

General Settings

Specify allowed domains

No

Yes

No

Define a pattern to exclude some URLs

Yes

Yes

Yes

Specify maximum crawler depth

Yes

Yes

Yes

Specify maximum URLs that can be crawled

Yes

Yes

Yes

Specify the number of workers to work in parallel

No

Yes

Yes

Specify crawler timeouts

No

Yes

No

Add headers to the crawler if original content expects a header

Yes

Yes

Yes

Render JavaScript and crawl it in addition to the page source

No

Yes

n/a

Starting the crawl (Trigger)

Use a request URL

Yes

Yes

Yes

Use a sitemap

Yes

Yes

n/a

Use a sitemap index

Yes

Yes

n/a

Use a JavaScript function

No

Yes

Yes

Use an RSS feed

No

Yes

n/a

Extracting attributes (Document Extractor)

Use an XPath expression

Yes

Yes

n/a

Use a CSS expression

No

Yes

n/a

Use a JavaScript function

No

Yes

Yes

Use JSONPath

No

No

Yes

Match a specific URL pattern before extracting an attribute

No

Yes

Yes

Create multiple rules to extract an attribute and then prioritize these rules

No

Yes

Yes

Specify entity-based rules to extract attributes

No

Yes

Yes

Schedule scans to keep index documents up to date with your original content

Yes

Yes

Yes

Handle original content with multiple locales and languages

No

Yes

Yes

Handle original content that requires authentication

No

Yes

Yes

Add additional starting points not covered by the trigger (Request Extractor)

No

Yes

Yes

Use the Ingestion API to make incremental updates

No

Yes

Yes

Use entity-based tags

No

Yes

Yes