Streaming live at 10am (PST)

How hide specific pages from web crawlers?


#1

Hello -

I have a few test pages that I want to keep published but I prefer that they not be found by the public via search engines. Is there any simple way to do this?

Thanks,
Colin


Option to prevent search engines from indexing site
#2

Hi Colin,

In the dashboard, you can go to the SEO settings of your page and update your robots.txt file to achieve this:

Here are some examples for the format that crawlers expect: http://www.robotstxt.org/orig.html#examples

After you make a change, you need to republish your site to make sure the file gets copied over.

Important: Webflow has no control over how often different search engines crawl your site, or how quickly they restore your site to their index once you decide to stop hiding it. For example, if you set your /about page to be disallowed while you're working on it, then allow it again once it's ready, it might take a while for search engines to revisit it again.

If the goal is to hide a page temporarily, you could just give it some random slug like /about-abc123supersecret and not link to it from any other part of your site - that would effectively hide it from search engines.


#3

#4