Streaming live at 10am (PST)

Best Practice for Disallow or NoIndex 404 page


#1

Based on another Webflow post...I started researching 404 Disallow of the 404 page vs NoIndex in meta data.

After a few hours, I came across sources that said disallow the 404 page isn't a guarantee it won't be indexed but that disallowing the 404 page could possible cause crawl issues for the entire site.

Adding meta name="robots" content="noindex" and meta name="googlebot" content="noindex" should accomplish no indexing. Since we can now add header data to each page would this method be safer/better than disallow? Adding that meta data to the 404 page shows towards the bottom of the header code when added in Webflow.


#2

Noindex and nofollow are fine for 404s. Google goes into detail and explains this here:

https://support.google.com/webmasters/answer/93710?hl=en

Regarding disallow/robots.txt, if you're using noindex and nofollow, this won't be necessary for your 404.

As for the placement in the header, Google will see it wherever you put it:

https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?hl=en

tl;dr: If you want to prevent a single page from appearing in Google results, add noindex and nofollow to that page's header:

<meta name="robots" content="noindex, nofollow" />

#3

@mbrannon47 Before using Webflow I was using "noindex" in the meta. Initially, Webflow didn't have per-page meta available if hosting with Webflow. I feel better using "noindex" over "disallow" (and I hate seeing the errors in webmaster tools)...so I'll be switching back to the "noindex" method.


#4

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.