Google could increase the listing of unsupported robots.txt guidelines in its documentation primarily based on evaluation of real-world robots.txt information collected by HTTP Archive.
Gary Illyes and Martin Splitt described the undertaking on the newest episode of Search Off the Record. The work began after a neighborhood member submitted a pull request to Google’s robots.txt repository proposing two new tags be added to the unsupported listing.
Illyes defined why the workforce broadened the scope past the two tags in the PR:
“We tried to not do issues arbitrarily, however slightly gather information.”
Reasonably than add solely the two tags proposed, the workforce determined to have a look at the prime 10 or 15 most-used unsupported guidelines. Illyes mentioned the objective was “a good place to begin, a good baseline” for documenting the most typical unsupported tags in the wild.
How The Analysis Labored
The workforce used HTTP Archive to examine what guidelines web sites use of their robots.txt recordsdata. HTTP Archive runs month-to-month crawls throughout hundreds of thousands of URLs utilizing WebPageTest and shops the ends in Google BigQuery.
The primary try hit a wall. The workforce “rapidly found out that nobody is really requesting robots.txt recordsdata” throughout the default crawl, that means the HTTP Archive datasets don’t sometimes embrace robots.txt content material.
After consulting with Barry Pollard and the HTTP Archive neighborhood, the workforce wrote a customized JavaScript parser that extracts robots.txt guidelines line by line. The custom metric was merged before the February crawl, and the ensuing information is now out there in the custom_metrics dataset in BigQuery.
What The Knowledge Exhibits
The parser extracted each line that matched a field-colon-value sample. Illyes described the ensuing distribution:
“After enable and disallow and consumer agent, the drop is extraordinarily drastic.”
Past these three fields, rule utilization falls into a protracted tail of much less widespread directives, plus junk information from damaged recordsdata that return HTML as an alternative of plain textual content.
Google at the moment supports four fields in robots.txt. These fields are user-agent, enable, disallow, and sitemap. The documentation says different fields “aren’t supported” with out itemizing which unsupported fields are most typical in the wild.
Google has clarified that unsupported fields are ignored. The present undertaking extends that work by figuring out particular guidelines Google plans to doc.
The highest 10 to 15 most-used guidelines past the 4 supported fields are anticipated to be added to Google’s unsupported guidelines listing. Illyes did not identify particular guidelines that might be included.
Typo Tolerance Could Develop
Illyes mentioned the evaluation additionally surfaced widespread misspellings of the disallow rule:
“I’m most likely going to increase the typos that we settle for.”
His phrasing implies the parser already accepts some misspellings. Illyes didn’t commit to a timeline or identify particular typos.
Why This Issues
Search Console already surfaces some unrecognized robots.txt tags. If Google paperwork extra unsupported directives, that might make its public documentation extra carefully mirror the unrecognized tags individuals already see surfaced in Search Console.
Trying Forward
The deliberate replace would have an effect on Google’s public documentation and the way disallow typos are dealt with. Anybody sustaining a robots.txt file with guidelines past user-agent, enable, disallow, and sitemap ought to audit for directives which have by no means labored for Google.
The HTTP Archive information is publicly queryable on BigQuery for anybody who desires to study the distribution straight.
Featured Picture: Screenshot from: YouTube.com/GoogleSearchCentral, April 2026.
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
