Streaming live at 10am (PST)

Issues with SEO - Robots.txt


#1

https://preview.webflow.com/preview/marship?preview=a6e5f4afdb5b10d87fa5efa22f81838b

Hi, I recently submitted the above site (my first in Webflow) to a company called 'TotalSEO' it scored as I expected in most areas but they have raised a couple of issues that concern me. I'll post a topic for each!

They say:-

Robots.txt is missing

It is important to ensure your website has a robots.txt file.
A robots.txt file allows you to restrict the access of search engine robots that crawl the web and it can prevent these robots from accessing specific
directories and pages you may not want them to see. It also specifies where the XML sitemap file is located.
One of the first things we do when we start a new SEO campaign is ensure the robots.txt file is in place and correctly set up.

I thought this was built in to Webflows functionality? Could someone please let me know how to rectify this?

Thank you


#2

Are you exporting the site and using your own hosting solution or are you hosting on Webflow? Webflow will create a robots.txt file automatically if you have content typed in the SEO > Robots.txt section for hosted sites.

Under Site Settings, press SEO:

Scroll down to Robots.txt and enter your content. You can type Allow: / if you want a vanilla Robots.txt entry.

Upon publishing, you'll have a Robots.txt file published right inside the root directory of your site!

Edit: This was added to the Wishlist section a while back: http://forum.webflow.com/t/create-more-options-standard-text-for-robots-txt/9270


#3

Thanks @McGuire, I am using Webflow - so do I need to type Allow:/ or is that just for other hosts? Also, forgive my stupidity what do you mean by vanilla?


#4

I should have been more clear! :smile:

What I mean by vanilla is that typing that in will generate a plain robots.txt file that has no special properties. Here's a great article that has a cheat sheet: https://moz.com/learn/seo/robotstxt

There's a more important issue here: The robots.txt file is NOT required for the vast majority of sites. It sounds like this SEO tool is making you jump through extra hoops. There's tons of literature online about this, and even Google says it in plain English: https://developers.google.com/webmasters/control-crawl-index/docs/faq#h01 :+1:


#5

Thank you @McGuire amazingly helpful and I will read up on the resources provides. Really appreciated. I think most of these SEO companies try to make it more complicated to justify their fee; however when they offered a free review I used it as a learning tool and I've learnt some valuable things - Thank you :smile:


#6