How does crawling and indexing work?
Answer
Our crawler visits the webpages of the domains you add to your collection. Read “How the crawler works” for more information.
Once the collection is indexed, all the webpages in the index are then re-visited by the crawler periodically between 3-7 days.
If you have our Instant Indexing ping-back code installed on your website, any new or updated webpage is updated in the collection when:
any of the mentioned page meta field's are updated (title, description, canonical value, robots), and
the page is visited within 30 minutes of being updated.
Instant indexing does not remove records from the index if a webpage’s status code is changed to a 404, 403, 301, or a 302. However, the regular crawl cycle does take the status code into account and will remove the page from the index if the page returns a 404, 403, 301, or a 302.