Crawler mode
Crawler mode follows links instead of watching one page. Use it to find new pages on a site, such as press releases, job ads or product listings, or to track a list that spans several pages.
Two ways to crawl
| Option | What it reports |
|---|---|
| Discover pages across a site | Walks the site from the start URL and reports pages it hasn’t seen before. |
| Monitor a list across its pages | Walks every page of one list, merges them into a single set of records, and reports records that were added, removed or changed. |
Settings
Set Monitoring mode to Crawler on the task’s Settings tab; the Crawler Settings appear below it. Scope and notification options apply when discovering pages.
| Setting | What it does |
|---|---|
| Start URL | Optional. Where to start crawling, if different from the task’s address. |
| Keywords | Comma-separated words to look for in the pages it visits. |
| Max pages | The most pages visited in one run. |
| Max depth | How many links away from the start URL to go. |
| Scope options | Seed from the site’s sitemap.xml, Include subdomains and Capture external links (leave this off unless you need outbound links). |
| Include / exclude rules | Regular expressions or glob patterns for URLs to crawl or skip, for example https://example.com/news/**. |
| Crawler notifications | One summary alert after a run: Notify on new pages and/or Notify on new external links. |
Credits
A crawl is billed on the pages it actually visits: one check credit for every five pages, or part of five. Keep Max pages and the include rules tight.
Was this page helpful?