Streaming live at 10am (PST)

Multiple and conflicting robots.txt files

There are apparently 2 conflicting robot.txt files associated with my site. One that says

User-agent: *
Allow: /

and other says

User-agent: *
Disallow: /

Click the 2 urls below to see

https://www.atriumhq.com/robots.txt
https://atriumhq.com/robots.txt

I do not want to block Google from indexing my site. I’d be grateful for any help on how to fix this!

Thanks!
May

This was interesting. However, if you have set your site default to the www.mydomain it will not be a problem. Regardless what the user type as a domain it will be redirected to the www domain

Thanks for the response. The issue I’m having is that when someone searches for “atriumhq” the resulting google search result does not show the meta data description or title. The reason for this according to Google’s search console is that the page is being blocked by a robots.txt file. I’m assuming that Webflow is generating this file. However, I can’t seem to access it. I’ve already entered

User-agent: *
Allow: /

in the SEO / Indexing section of my project’s settings. Is there something else I can do?

Thanks!
May

1 Like

I am experiencing a similar issue - and the conflicting robots.txt file references “Sonar links”. If you refresh robots.txt it varies between the expected file set up in the Webflow backend, and the file of unknown origin which is causing Googlebot issues.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.