Streaming live at 10am (PST)

Robot.txt problem

Problem is Google representative cannot see the robot.txt file, neither I am.

I added it in here:

I’ve tested in here and, get an error: https://search.google.com/test/mobile-friendly?hl=en&utm_source=mft&utm_medium=redirect&utm_campaign=mft-redirect&id=kOlAYBYs6ovET29O3wascw

Why is this happening? I can see the robots.txt when I click https://www.try.adspert.net/robots.txt.

Do you have any idea @vincent?

I would be happy if you can help me :slight_smile:

Hi! I’m not an expert but:

  1. your code is missing a “/”, it should be:

    User-Agent: *
    Disallow: /

  2. the error you’re linking to says it can’t browse the pages, which is normal if you “Disallow” access to robots…

But if i add slash, it probably disallow any robots, see here: Robots.txt File [2020 Examples] - Moz

You’re right, your code isn’t improper.

image

But if you want to allow all robots everywhere, you don’t need a robots.txt file at all, do you?

1 Like