John Mueller from Google said that the new Content Signals robots.txt directive invented by Cloudflare last year has “no effects whatsoever for any crawler or LLM.” He added that it “just adds bloat and future maintenance to your robots.txt file.” As far as he knows, he said, none of the crawlers or LLMs use the content-signal robots.txt directives.
Related Posts
Report: Small Google Business Profile Updates May Trigger Re-Verification
Krystal Taing, a Google local product expert in the Google Business Profiles forum and Vice President at Uberall, said she is seeing a rise in re-verifications from Google Business Profiles even after just making small changes or additions to your Google Business Profile.
Google: Don’t Spend Too Much Time On Redirects Analysis For SEO
Google’s John Mueller said he would “caution against assuming that you need to do this level of analysis for all URLs on a website in order to achieve optimal SEO” when it comes to reviewing bad redirects or CSP settings. Why, because bad redirects or CSP settings are often simply visible when doing normal browsing, and if you see it, then that is enough.
Daily Search Forum Recap: May 20, 2025
Here is a recap of what happened in the search forums today…
