How often does the crawler crawl my content?

Answer

If you have the ping-back code installed, we crawl new content to your Collection as soon as the page is visited. When a new page that has the ping-back code installed is visited, it triggers the ping-back code and a request is sent to the crawler which fetches the page and updates it in the index immediately.

For sites that don’t have the ping-back code installed, the crawler will crawl your site's content every 3-6 days.

How to request re-crawl of a specific page without delay:

If you want to request the crawler to crawl a specific page of your Collection without waiting for the normal crawl cycle, use the “Diagnose page” button in Domains section of your console to add the URL of the page. Once added, press Diagnose, and the current status of the page will be displayed. Click on “Index Page“ to request a re-crawl of the page and it will be re-indexed immediately.