Google On Phantom Noindex Errors In Search Console

Google’s John Mueller just lately answered a query about phantom noindex errors reported in Google Search Console. Mueller asserted that these stories could also be actual.

Noindex In Google Search Console

A noindex robots directive is certainly one of the few instructions that Google should obey, certainly one of the few ways in which a website proprietor can train management over Googlebot, Google’s indexer.

And but it’s not completely unusual for search console to report being unable to index a web page due to a noindex directive that seemingly does not have a noindex directive on it, at the least none that is seen in the HTML code.

When Google Search Console (GSC) stories “Submitted URL marked ‘noindex’,” it is reporting a seemingly contradictory scenario:

• The positioning requested Google to index the web page by way of an entry in a Sitemap.
• The web page despatched Google a sign not to index it (by way of a noindex directive).

It’s a complicated message from Search Console {that a} web page is stopping Google from indexing it when that’s not one thing the writer or web optimization can observe is taking place at the code stage.

The individual asking the query posted on Bluesky:

“For the previous 4 months, the web site has been experiencing a noindex error (in ‘robots’ meta tag) that refuses to disappear from Search Console. There is no noindex anyplace on the web site nor robots.txt. We’ve already seemed into this… What may very well be inflicting this error?”

Noindex Reveals Solely For Google

Google’s John Mueller answered the query, sharing that there have been at all times a noindex displaying to Google on the pages he’s examined the place this sort of factor was taking place.

Mueller responded:

“The circumstances I’ve seen in the previous had been the place there was truly a noindex, simply generally solely proven to Google (which may nonetheless be very laborious to debug). That mentioned, be at liberty to DM me some instance URLs.”

Whereas Mueller didn’t elaborate on what may be going on, there are methods to troubleshoot this difficulty to discover out what’s going on.

How To Troubleshoot Phantom Noindex Errors

It’s attainable that there is a code someplace that is inflicting a noindex to present only for Google. For instance, it could have occurred {that a} web page at one time had a noindex on it and a server-side cache (like a caching plugin) or a CDN (like Cloudflare) has cached the HTTP headers from that point, which in flip would trigger the previous noindex header to be proven to Googlebot (as a result of it ceaselessly visits the website) whereas serving a contemporary model to the website proprietor.

Checking the HTTP Header is straightforward, there are many HTTP header checkers like this one at KeyCDN or this one at SecurityHeaders.com.

A 520 server header response code is one which’s despatched by Cloudflare when it’s blocking a person agent.

Screenshot: 520 Cloudflare Response Code

Under is a screenshot of a 200 server response code generated by cloudflare:

Screenshot: 200 Server Response Code

I checked the identical URL utilizing two completely different header checkers, with one header checker returning a a 520 (blocked) server response code and the different header checker sending a 200 (OK) response code. That reveals how in a different way Cloudflare can reply to one thing like a header checker. Ideally, strive checking with a number of header checkers to see if there’s a constant 520 response from Cloudflare.

In the scenario the place an internet web page is displaying one thing completely to Google that is in any other case not seen to somebody taking a look at the code, what you want to do is to get Google to have a look at the web page for you utilizing an precise Google crawler and from a Google IP handle. The way in which to do that is by dropping the URL into Google’s Wealthy Outcomes Check. Google will dispatch a crawler from a Google IP handle and if there’s one thing on the server (or a CDN) that’s displaying a noindex, this can catch it. As well as to the structured knowledge, the Wealthy Outcomes take a look at may even present the HTTP response and a snapshot of the internet web page displaying precisely what the server reveals to Google.

Whenever you run a URL by the Google Wealthy Outcomes Check, the request:

• Originates from Google’s Knowledge Facilities: The bot makes use of an precise Google IP handle.
• Passes Reverse DNS Checks: If the server, safety plugin, or CDN checks the IP, it would resolve again to googlebot.com or google.com.

If the web page is blocked by noindex, the instrument will probably be unable to present any structured knowledge outcomes. It ought to present a standing saying “Web page not eligible” or “Crawl failed”. For those who see that, click on a hyperlink for “View Particulars” or increase the error part. It ought to present one thing like “Robots meta tag: noindex” or ‘noindex’ detected in ‘robots’ meta tag”.

This method does not ship the GoogleBot person agent, it makes use of the Google-InspectionTool/1.0 person agent string. Meaning if the server block is by IP handle then this technique will catch it.

One other angle to verify is for the scenario the place a rogue noindex tag is particularly written to block GoogleBot, you may nonetheless spoof (mimic) the GoogleBot person agent string with Google’s personal User Agent Switcher extension for Chrome or configure an app like Screaming Frog set to determine itself with the GoogleBot person agent and that ought to catch it.

Screenshot: Chrome Consumer Agent Switcher

Phantom Noindex Errors In Search Console

These sorts of errors can really feel like a ache to diagnose however before you throw your arms up in the air take a while to see if any of the steps outlined right here will assist determine the hidden purpose that’s chargeable for this difficulty.

Featured Picture by Shutterstock/AYO Manufacturing