POST /1/crawlers

Default value: "application/json"

Name of the crawler.

Crawler configuration.

Function for extracting URLs from links on crawled pages.

For more information, see the linkExtractor documentation.

Possible values:

• "function"

Crawler index settings.

These index settings are only applied during the first crawl of an index. Any subsequent changes won't be applied to the index. Instead, make changes to your index settings in the Algolia dashboard.

Limits the number of URLs your crawler processes.

Change it to a low value, such as 100, for quick crawling tests. Change it to a higher explicit value for full crawls to prevent it from getting "lost" in complex site structures.

Because the Crawler works on many pages simultaneously, maxUrls doesn't guarantee finding the same pages each time it runs.

URLs from where to start crawling.

If true, use a Chrome headless browser to crawl pages.

Because crawling JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specific list of pages. Use micromatch to define URL patterns, including negations and wildcards.

References to external data sources for enriching the extracted records.

For more information, see Enrich extracted records with external data.

Algolia application ID where the crawler creates and updates indices. The Crawler add-on must be enabled for this application.

Sitemaps with URLs from where to start crawling.

Checks to ensure the crawl was successful.

For more information, see the Safety checks documentation.

Checks triggered after the crawl finishes but before the records are added to the Algolia index.

Stops the crawler if a specified number of pages fail to crawl.

Maximum difference in percent between the numbers of records between crawls.

Default value: 10

Query parameters to ignore while crawling.

All URLs with the matching query parameters will be treated as identical. This prevents indexing URLs that just differ by their query parameters.

You can use wildcard characters to pattern match.

Whether to back up your index before the crawler overwrites it with new records.

Whether to ignore the nofollow meta tag or link attribute.

For more information, see the ignoreNoFollowTo documentation.

Lets you add options to HTTP requests made by the crawler.

Timeout in milliseconds for the crawl.

Default value: 30000

Proxy for all crawler requests.

Maximum number of retries to crawl one URL.

Default value: 3

Headers to add to all requests.

Preferred natural language and locale.

Basic authentication header.

Cookie. The header will be replaced by the cookie retrieved when logging in.

Number of concurrent tasks per second.

If processing each URL takes n seconds, your crawler can process rateLimit / n URLs per second.

Higher numbers mean faster crawls but they also increase your bandwidth and server load.

A prefix for all indices created by this crawler. It's combined with the indexName for each action to form the complete index name.

Schedule for running the crawl.

For more information, see the schedule documentation.

URLs to exclude from crawling.

The Crawler treats extraUrls the same as startUrls. Specify extraUrls if you want to differentiate between URLs you manually added to fix site crawling from those you initially specified in startUrls.

A list of actions.

Key-value pairs to replace matching hostnames found in a sitemap, on a page, in canonical links, or redirects.

For more information, see the hostnameAliases documentation.

Unique identifier for the action. This option is required if schedule is set.

Whether to generate an objectID for records that don't have one.

Default value: true

Indicates intermediary pages that the crawler should visit.

For more information, see the discoveryPatterns documentation.

Whether the crawler should cache crawled pages.

For more information, see the cache documentation.

Whether the crawler cache is active.

Default value: true

Function for extracting information from a crawled page and transforming it into Algolia records for indexing. The Crawler has an editor with autocomplete and validation to help you update the recordExtractor property.

For details, consult the recordExtractor documentation.

A JavaScript function (as a string) that returns one or more Algolia records for each crawled page.

Possible values:

• "function"

File types for crawling non-HTML documents.

For more information, see Extract data from non-HTML documents.

Default value: [ "html" ]

Reference to the index used to store the action's extracted records. indexName is combined with the prefix you specified in indexPrefix.

Key-value pairs to replace matching paths with new values.

It doesn't replace:

• URLs in the startUrls, sitemaps, pathsToMatch, and other settings.
• Paths found in extracted text.

The crawl continues from the transformed URLs.

URLs to which this action should apply.

Uses micromatch for negation, wildcards, and more.

DOM selectors for nodes that must be present on the page to be processed. If the page doesn't match any of the selectors, it's ignored.

Whether to ignore the noindex robots meta tag. If true pages with this meta tag will be crawled.

Whether to ignore rules defined in your robots.txt file.

Algolia API key for indexing the records.

For more information, see the apiKey documentation.

Authorization method and credentials for crawling protected content.

Maximum path depth of crawled URLs. For example, if maxDepth is 2, https://example.com/foo/bar is crawled, but https://example.com/foo/bar/baz won't. Trailing slashes increase the URL depth.