...
You can setup a variety of rules for crawling your site.
From your Console, you can choose what domains are stored in your collection, and if crawling is active for each domain.
...
Page Comparison
General
Content
Integrations