Streaming live at 10am (PST)

Robots.txt not found


#1

This post was flagged by the community and is temporarily hidden.


#2

Hi Simon,

Definitely a great question, and hopefully I can help :bowing_man:

You can use any of these rules to populate the robots.txt file.

  • User-agent: * means this section applies to all robots.
  • Disallow: tells the robot to not visit the site, page, or folder.

To hide your entire site -
User-agent: *
Disallow: /

To hide individual pages -
User-agent: *
Disallow: /page-name

To hide entire folder of pages -
User-agent: *
Disallow: /folder-name/

To include a sitemap -
Sitemap: https:yo

Helpful resources
Check out more useful robots.txt rules


#3

This post was flagged by the community and is temporarily hidden.


#4

Hi there,

I was able to take a look at your robots settings and noticed that it contains Allow: / which is currently not a rule.

You can use any of these rules to populate the robots.txt file.

User-agent: * means this section applies to all robots.
Disallow: tells the robot to not visit the site, page, or folder.

The robots.txt is usually used to list the URLs on a site that you don’t want search engines to crawl. You can also include the sitemap of your site in your robots.txt file to tell search engine crawlers which content they should crawl: https://university.webflow.com/article/disable-search-engine-indexing#generating-a-robots.txt-file


#5

Dear Mistercreate,

thank you very much for the fast response.
I changed the robots as you requested, but could you tell me, why I get an error 404 if I try to open this file?

/www.volunteer-vision.com / robots.txt

You did not answer to this.


#7

Hi there,

Thanks for completing the update as it was very helpful.

That does look like odd behavior as the update should allow you to access your robots.txt file.

Our Customer Success Team has notified our engineers, and they are currently looking into for you.

Thank you so much for your patience, and we will provide an update as soon as one is available.


#8

Hi @Simon_Fakir

Thanks again for posting about this. It looks like the robots.txt file is no longer showing as a 404.

Can you confirm this is working as expected on your end?