How often does the crawler crawl my content?
Answer
Sajari
Search.io handles crawling in a few different ways: instant updates triggered via a ping-back code snippet, regular crawls, and manually requested crawls of either entire Collections collections or single pages.
Regular crawls
We monitor your collection for any pages that have not been refreshed in the previous week. We then queue and update those pages.
To keep your collection in closest sync with your website content we recommend installing the ping-back code to enable Instant Indexing.
Instant indexing with ping-back
If you have the ping-back code installed, we crawl new content and add it to your Collection collection as soon as the page is visited. We will also automatically update popular pages on your site when their content and metadata changes.
For sites that don’t have the ping-back code installed, updates will rely on the regular crawl cycle.
To check if instant indexing is running on your collection visit the Instant Indexing page in your console.
Regular crawls
The regular crawling process runs on every Collection, whether you have the ping-back code installed or not. We monitor your collection for any pages that have not been refreshed in the previous week. We then queue and update those pages.To keep your Collection in closest sync with your website content we recommend in installing the ping-back code to enable Instant Indexing.
Immediately re-crawl a specific page
If you want to re-crawl a specific page of your Collection, use the “Diagnose page” button in Domains section of “Diagnose” functionality in Crawler crawl statuses section of your console to add the URL of the page. Once added, press Diagnose, and the current status of the page will be displayed. Click on “Index Page“ “Re-crawl page“ to request a re-crawl of the page and it will be re-indexed immediately.
...
You can request a re-crawl of an entire collection by visiting the Domains section in Crawler crawl statuses section of your console and selecting clicking “Re-index crawl all domainspages.” This will add all pages in all of your domains in the Collection collection to the crawling queue. The speed at which your Collection collection updates depends on a variety of factors including the size of your collection and speed of your website and can take from a few hours to a day or more.
...