How does crawling and indexing work?

Answer

Without our ping-back code installed, indexing is done by following links starting with the root of the domain.

For instance, for sajari.com the crawler starts with http://www.sajari.com and follows links from there. Here's some more information on how Sajari works with domains.

With the ping-back code installed, new or updated content is updated in the collection as soon as the page is visited rather than waiting for a periodic crawl cycle, which might take 3-7 days. Without our ping-back code installed, indexing is done by following links starting with the root of the domain.