Streaming live at 10am (PST)

Pages blocked in robots.txt showing in google index


#1

Hi

My robots file looks like this:
http://leicestermortgagebroker.co.uk/robots.txt

but when I run site://leicestermortgagebroker.co.uk in google, the contact and privacy pages are showing in google's index, which I don't want.

Any idea why this is? Is the only way around it to remove these urls in google webmaster tools?

Webflow Support, any idea if/when the ability to noindex individual pages will be available as a feature? It would be really useful for seo purposes.

Thanks a lot

Graham


Noindexing individual pages
#2

https://developers.google.com/webmasters/control-crawl-index/docs/faq?csw=1#h17

robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header.


#3

Hi Vincent

Thanks very much for your reply.

How do I use the noindex robots meta tag in webflow on an individual page please. I thought this wasn't possible in webflow yet?

Thanks

Graham


#4

I am not sure you can for individual pages as the custom header code (where Meta goes), impact the whole site. Maybe ask Webflow support about that.


#5

Was there any resolution to this question? I have the same need.


#6

I too, have the same question.


#7

Seems like you should be able to throw <META NAME="ROBOTS" CONTENT="NOINDEX"> into the Head tag section with no problem ... as noted in the UI "code included here will only apply to this page"


#8

I really don't know why I was under the impression that was not the case... must have been raelly tired :—) Thanks for clearing things up.


#9

No problem! Easy enough to do, especially when you're tired : )


#10