This Confluence instance is now read-only, please head over to the Algolia Confluence instance for the same more up-to-date information
Can I index and crawl password protected sites?
- Former user (Deleted)
- Richard Davidson
- Jeremy Luscombe
Owned by Former user (Deleted)
Can Search.io index password protected sites?
Answer
Yes! It’s possible to index login-protected pages.
You can whitelist our crawler (via user agent) so that we can index your login-protected website pages. You can do this by allowing requests that have “Sajaribot -“ at the start of the user-agent.
Alternatively, you can also whitelist our crawler using IP addresses. Here is a list of the current IP addresses. The list is updated dynamically so IP’s will change over time albeit not frequently. To programmatically obtain the current IP’s we recommend using this tool.