▼
Streaming live at 10am (PST)

Basic robots.txt question

#1

Normally, we host all of our sites on our own server, but a particular client needs features that we have decided to host them on webflow to provide.

We’re accustomed to creating our own robots.txt file and uploading it, but in Webflow, I see that on the SEO tab there is an “Indexing” section. I have pasted our robots.txt file there and published, however, Google Search Console is still showing it as non-existent. Can someone please explain if this is the proper place to upload the robots.txt file?

The sitemap worked and has been recognized/submitted. But I am just confused by the robots.txt portion. I’m sure it’s something basic I’m missing, and I’ve gone through the Webflow help documents to try to find my answer.

Thanks!

#2

Bumping this. Hoping for a little help soon.

#3

Under the root.
‎/robots.txt
This is also where webflow system creates this file.
so first check if you find this file under (maybe you not publish the changes to the “live site”)
www.mysite/robots.txt
https://support.google.com/webmasters/answer/6078399?hl=en

maybe this is hosting issue:

webflow docs:

#4

I’m hosting this site ON Webflow. The Search Console finds no robots.txt file. I thought Webflow generated one?

And, just in case, I inserted my own into the “indexing” portion on the site. Still nothing.

How is it a hosting issue? Again, I’m paying Webflow to host this site, just need to know how to get the robots.txt file recognizable by Google Search Console.

#5

Add url + Screenshot of your robots.txt area (In the UI)

First check if you have this file under
www.mysite/robots.txt

Example (webflow site from “showcase” area):
https://www.reine-lungau.at/robots.txt