John Mueller from Google said that the new Content Signals robots.txt directive invented by Cloudflare last year has “no effects whatsoever for any crawler or LLM.” He added that it “just adds bloat and future maintenance to your robots.txt file.” As far as he knows, he said, none of the crawlers or LLMs use the content-signal robots.txt directives.
Related Posts
Daily Search Forum Recap: September 25, 2025
Here is a recap of what happened in the search forums today…
Daily Search Forum Recap: April 25, 2025
Here is a recap of what happened in the search forums today, through the eyes of the Search Engine Roundtable and other search forums on the web. Google reported earnings last night and their ad revenue continues to grow…
Google Search Console Page Indexing Report Missing A Chunk Of Data
The Google Search Console page indexing report seems to be missing a chunk of data. The missing data is from December 15th and earlier, but we do have data from December 15th (14th) through today.
