{"id":8239,"date":"2023-02-17T17:46:27","date_gmt":"2023-02-17T17:46:27","guid":{"rendered":"https:\/\/craighordlow.com\/index.php\/2023\/02\/17\/google-warns-against-using-403-or-404-status-codes-for-googlebot-crawl-rate-limiting\/"},"modified":"2023-02-17T17:46:27","modified_gmt":"2023-02-17T17:46:27","slug":"google-warns-against-using-403-or-404-status-codes-for-googlebot-crawl-rate-limiting","status":"publish","type":"post","link":"https:\/\/craighordlow.com\/index.php\/2023\/02\/17\/google-warns-against-using-403-or-404-status-codes-for-googlebot-crawl-rate-limiting\/","title":{"rendered":"Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting"},"content":{"rendered":"

Google is warning against using 404 and other 4xx client server status errors, such as 403s, for the purpose of trying to set a crawl rate limit for Googlebot. \u201cPlease don\u2019t do that,\u201d Gary Illyes from the Google Search Relations team wrote<\/a>.<\/p>\n

Why the notice. <\/strong>There has been a recent increase in the number of sites and CDNs using these techniques to try to limit Googlebot crawling. \u201cOver the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use\u00a0404\u00a0and other\u00a04xx\u00a0client errors (but not\u00a0429) to attempt to reduce Googlebot\u2019s crawl rate,\u201d Gary Illyes wrote.<\/p>\n

What to do instead. <\/strong>Google has a detailed help document<\/a> just on the topic of reducing Googlebot crawling on your site. The recommended approach is to use the Google Search Console<\/a> crawl rate settings to adjust your crawl rate.<\/p>\n

Google explained, \u201cTo quickly reduce the crawl rate, you can\u00a0change the Googlebot crawl rate in Search Console<\/a>. Changes made to this setting are generally reflected within days. To use this setting, first\u00a0verify your site ownership<\/a>. Make sure that you avoid setting the crawl rate to a value that\u2019s too low for your site\u2019s needs. Learn more about\u00a0what crawl budget means for Googlebot<\/a>. If the\u00a0Crawl Rate Settings<\/a>\u00a0is unavailable for your site,\u00a0file a special request<\/a>\u00a0to reduce the crawl rate. You cannot request an increase in crawl rate.\u201d<\/p>\n

If you can\u2019t do that, Google then says \u201creduce the crawl rate for short period of time (for example, a couple of hours, or 1-2 days), then return an informational error page with a 500, 503, or 429 HTTP response status code.\u201d<\/p>\n

Why we care. <\/strong>If you noticed crawling issues, maybe your hosting provider or CDN recently deployed these techniques. You may want to submit a support request with them to show them Google\u2019s blog post on this topic to ensure they are not using 404s or 403s to reduce crawl rates.<\/p>\n

The post Google warns against using 403 or 404 status codes for Googlebot crawl rate limiting<\/a> appeared first on Search Engine Land<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"

Google is warning against using 404 and other 4xx client server status errors, such as 403s, for the purpose of trying to set a crawl rate limit for Googlebot. \u201cPlease don\u2019t do that,\u201d Gary Illyes from the Google Search Relations team wrote. Why the notice. There has been a recent increase in the number of sites and CDNs using these techniques to try to limit Googlebot crawling. \u201cOver the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting […]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"h5ap_radio_sources":[],"cybocfi_hide_featured_image":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-8239","post","type-post","status-publish","format-standard","hentry","category-best-seo-blogs"],"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":false,"author_link":"https:\/\/craighordlow.com\/author\/"},"_links":{"self":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts\/8239","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/comments?post=8239"}],"version-history":[{"count":0,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts\/8239\/revisions"}],"wp:attachment":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/media?parent=8239"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/categories?post=8239"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/tags?post=8239"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}