▼
Streaming live at 10am (PST)

Basic robots.txt question

Publishing help SEO
#1

Normally, we host all of our sites on our own server, but a particular client needs features that we have decided to host them on webflow to provide.

We’re accustomed to creating our own robots.txt file and uploading it, but in Webflow, I see that on the SEO tab there is an “Indexing” section. I have pasted our robots.txt file there and published, however, Google Search Console is still showing it as non-existent. Can someone please explain if this is the proper place to upload the robots.txt file?

The sitemap worked and has been recognized/submitted. But I am just confused by the robots.txt portion. I’m sure it’s something basic I’m missing, and I’ve gone through the Webflow help documents to try to find my answer.

Thanks!

#2

Bumping this. Hoping for a little help soon.

#3

Under the root.
‎/robots.txt
This is also where webflow system creates this file.
so first check if you find this file under (maybe you not publish the changes to the “live site”)
www.mysite/robots.txt
https://support.google.com/webmasters/answer/6078399?hl=en

maybe this is hosting issue:

webflow docs:

#4

I’m hosting this site ON Webflow. The Search Console finds no robots.txt file. I thought Webflow generated one?

And, just in case, I inserted my own into the “indexing” portion on the site. Still nothing.

How is it a hosting issue? Again, I’m paying Webflow to host this site, just need to know how to get the robots.txt file recognizable by Google Search Console.

#5

Add url + Screenshot of your robots.txt area (In the UI)

First check if you have this file under
www.mysite/robots.txt

Example (webflow site from “showcase” area):
https://www.reine-lungau.at/robots.txt

#6

If you want the page not to get into the index, it is better to use noindex in the robots meta tag. To do this, add the following meta tag to the page in the section:

.

This will allow you

remove the page from the index when you next visit the search robot (and it will not be necessary to manually delete this page via webmasters)
allow you to transfer the reference page weight
robots.txt file is best to close from indexing:

admin site
site search results
login / login / password recovery page