Google’s John Mueller answered a question on Reddit about a commonly used robots meta tag and what would happen if it was missing. Mueller’s answer, while it makes sense and is documented, may still come as a surprise to many publishers and SEOs.
Robots Meta Tag
The HTML meta element communicates metadata. Metadata is machine readable information that a crawler like Googlebot can read.
There are many kinds of meta elements like the meta description element but the Robots Meta Element is different in that it can control the search engine crawlers.
The information communicated by the robots meta tag is called a directive, which means that robot crawlers are obligated to obey the instructions in the robots meta tag.
There are many directions to pass along through the robots meta but the following meta tag is one that is relevant to the question John Mueller answered.
The noindex, no follow meta tag:
<meta name="robots" content="noindex, nofollow">
The above meta tag tells the search engine crawlers to not index the content on the webpage and to not follow any links.
One of the most common meta tags is this one, which commands search engines to index the content and follow all the links:
<meta name="robots" content="index, follow">
While the above meta tag is common, there is a significant amount of misunderstanding about it. There’s a line of reasoning that because Google supports nofollow then it must imply that Google supports the follow directive.
I found a number of authoritative websites that say that Google uses the meta robots index, follow meta tag.
But that’s not actually how Google uses those directives, as John Mueller makes clear in his answer.
See also: Google Answers If Meta Robots Tags Affect Search Rankings
What’s The Effect Of Leaving Out The Meta Robots Index Tag?
The person on Reddit asked the following question:
“I’m a bit confused with a website I’m working on.
So, this is what the meta snippets on most of the websites I work on look like:
<meta name=’robots’ content=’index, follow ….
But, on the website at hand, it’s missing the ‘index’ tag.
My question is: What’s the effect of the site missing the ‘index’ tag.”
John Mueller answered:
“The “index” robots meta tag has no function (at least in Google) – it’s completely ignored. Also “follow”.
Google has https://developers.google.com/search/docs/crawling-indexing/special-tags & https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag to document the meta tags that have functions. You can use anything else, it’ll be ignored. <meta name=”robots” content=”topranking bestcheese”> is an option, if you want to throw people off.”
Why Google Ignores Robots Index & Follow
The simple reason why Google ignores the robots index and follow meta tag is because indexing and follow are the defaults.
Indexing and following links is what search engine robots do, they don’t have to be told to index content and follow links because that’s their purpose.
Google’s documentation on robots tags advises:
“The default values are index, follow and don’t need to be specified.”
The full list of valid directives for Google is here.
If the robots meta you want to use isn’t’ listed there then Googlebot is going to ignore it.
Related: How To Control Googlebot’s Interaction With Your Website
Is Index, Follow Completely Useless?
It’s true, documented and official that when it comes to Googlebot, <meta name=”robots” content=”index, follow”> is a waste of HTML space and is ignored by Googlebot.
Bing treats index, follow in a similar way but with a slight difference, as described in the official Bing documentation for meta tags.
This is what Bing’s documentation says about the index directive:
“By default we assume “index”, but if needed you can use <meta name=”robots” content=”index”> to explicitly state that we may index the page.”
And this is what it says about the follow directive:
“By default we assume “follow”, but you can explicitly state “follow” if so desired.”
In my 20+ years of working in SEO creating websites, optimizing them and ranking them, I’ve always considered it a good policy to give the bots what they expect and try not to give them anything that’s unexpected. So if a meta description is not necessary then my impulse would be to leave it out because the whole point of optimizing is to make it as easy as possible for the search engines to index and understand the content, which means to get rid of anything that might work against that goal.
In this case, it’s highly likely it’s not going to have an effect one way or the other.
But… There’s another way that follow and index trip people up.
Some publishers use this robots meta tag:
<meta name="robots" content="noindex, follow">
Some sites advise that if the page isn’t indexed, the use of the “follow” directive compels the search engine to follow the links.
But that’s not true if there’s a “noindex” directive for the simple reasons that Google cannot follow a link on a page that it is not indexed. If it’s not in the index then the links on those pages are not in the index.
Read: How To Get Google To Index Your Site (Quickly)
Featured Image by Shutterstock/Bangun Stock Productions
