This Confluence instance is now read-only, please head over to the Algolia Confluence instance for the same more up-to-date information

How do I prevent pages from being crawled?

Owned by Jens Schumacher (Deactivated)
Last updated: Oct 13, 2022 by Jessica Hong (Unlicensed)
1 min read

Problem

I want to prevent certain pages from being indexed and appearing in search results.

Solution

You can add an HTML element of data-sj-noindex anywhere in a page and it will not be indexed. Most commonly this will be defined in the <head> of an HTML page as follows:

  1. Locate the <head> tag of the page you want to prevent from being crawled.

  2. Add the following code within the <head>:
    <meta name="robots" content="noindex" data-sj-noindex />

  3. Save the changes. The crawler will ignore this page next time it comes across it.

Additionally you can use crawling rules to programmatically exclude sections or certain pages of your website.

Related articles

Documentation

Advanced crawler documentation