▼
Streaming live at 10am (PST)

Custom domain robot indexing disabling

#1

A bit of an unusual one. I have added a custom domain to my webflow website. I have set the webflow subdomain to prevent robot indexing for the entire site. If I then set the site live on the custom domain will I also need to add another custom robot file to the root location of the custom domain to prevent it being searched under that name? Or will the Webflow robot disabling be enough to cover all?

TIA

#2

That's easy to check. Browse the site on the url you want to check, look at the source, in the HEAD section, the metas... You should find a noindex,nofollow. If not, the site is crawled by search engines.

#3

Hi Vincent,

If the site is crawled, what would I need to do to fix?

TIA

#4

The custom domain is a subdomain of a different .com

#5

I think it won't be crawled but check with my method to be sure of this.

#6

Hi Vincent,

I checked the source of the webflow URL(The standard one given by webflow) and couldn't find a meta with the following "noindex,nofollow." I have set the webflow SEO setting to:

Disable Webflow Subdomain Indexing

robots.txt
User-agent: *
Disallow: /

In theory are you saying that once that has been set in webflow I should be able to find 'noindex,nofollow.' in the meta tags? As it isn't there? Does this mean the function within webflow isn't working correctly?

TIA

#7

this option is to disable indexing on the webflow.io domain when you have your site published both on .io and a custom domain, for various reasons.

Can you give me all the urls your site appears on?

#8

Hi Vincent,

I am unable to share the URL. But according to your previous post, in theory if I use the webflow subdomain indexing disable feature this should block the site and the custom domain from being indexed?

Thanks

#9

No, it will not block yourcustomdomain.com if it has been published there. It will only disallow indexing on yourcustomdomain.webflow.io (free webflow subdomain) from being indexed.

If your free webflow subdomain has already been indexed by search engines, it will take anywhere from four weeks to a year for it to be removed from the search engines' index automatically. Webflow has no control over third party services.

#10

Hi, this is what I originally thought. So do I also need to add a custom robot.txt to the root of the custom subdomain?

#11

I am unable to advise you further without knowing all the details.

closed #12

This topic was automatically closed after 60 days. New replies are no longer allowed.