{"id":7964,"date":"2023-02-01T13:59:52","date_gmt":"2023-02-01T13:59:52","guid":{"rendered":"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/"},"modified":"2023-02-01T13:59:52","modified_gmt":"2023-02-01T13:59:52","slug":"is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites","status":"publish","type":"post","link":"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/","title":{"rendered":"Is Google using a ChatGPT-like system for spam and AI content detection and ranking websites?"},"content":{"rendered":"<p>The headline is intentionally misleading \u2013 but only insofar as using the term \u201cChatGPT\u201d is concerned.<\/p>\n<p>\u201cChatGPT-like\u201d immediately lets you, the reader, know the type of technology I\u2019m referring to, instead of describing the system as \u201ca text-generation model like GPT-2 or GPT-3.\u201d (Also, the latter really wouldn\u2019t be as clickable\u2026)<\/p>\n<p>What we will be looking at in this article is an older, but highly relevant Google paper from 2020, \u201c<a href=\"https:\/\/arxiv.org\/pdf\/2008.13533.pdf\" target=\"_blank\" rel=\"noopener noreferrer\">Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study<\/a>.\u201d<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_81 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/#What_is_the_paper_about\" >What is the paper about?<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/#The_setup\" >The setup<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/#An_aside_about_spam_prevalence\" >An aside about spam prevalence<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/#Their_findings\" >Their findings<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/craighordlow.com\/index.php\/2023\/02\/01\/is-google-using-a-chatgpt-like-system-for-spam-and-ai-content-detection-and-ranking-websites\/#A_note_about_AI-generated_content\" >A note about AI-generated content<\/a><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"What_is_the_paper_about\"><\/span>What is the paper about?<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Let\u2019s start with the description of the authors. They introduce the topic thusly:<\/p>\n<p><em>\u201cMany have raised concerns about the potential dangers of neural text generators in the wild, owing largely to their ability to produce human-looking text at scale.<\/em><\/p>\n<p><strong><em>Classifiers trained to discriminate between human and machine-generated text have recently been employed to monitor the presence of machine-generated text on the web<\/em><\/strong><em> [29]. Little work, however, has been done in applying these classifiers for other uses, despite their attractive property of requiring no labels \u2013 only a corpus of human text and a generative model. In this work, we show through rigorous human evaluation that <\/em><strong><em>off-the-shelf human vs. machine discriminators serve as powerful classifiers of page quality<\/em><\/strong><em>. That is, texts that appear machine-generated tend to be incoherent or unintelligible. To understand the presence of low page quality in the wild, we apply the classifiers to a sample of half a billion English webpages.\u201d<\/em><\/p>\n<p>What they\u2019re essentially saying is that they have found that the same classifiers developed to detect AI-based copy, using the same models to generate it, can be successfully used to detect low-quality content.<\/p>\n<p>Of course, this leaves us with an important question:<\/p>\n<p>Is this <strong>causation<\/strong> (i.e., is the system picking it up because it\u2019s genuinely good at it) or <strong>correlation<\/strong> (i.e., is a lot of current spam created in a way that is easy to get around with better tools)?<\/p>\n<p>Before we explore that however, let\u2019s look at some of the authors\u2019 work and their findings.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_setup\"><\/span>The setup<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>For reference, they used the following in their experiment:<\/p>\n<p><strong>Two text-generation models<\/strong>, OpenAI\u2019s <a href=\"https:\/\/github.com\/openai\/gpt-2-output-dataset\/tree\/master\/detectorv\" target=\"_blank\" rel=\"noopener noreferrer\">RoBERTa<\/a>-based GPT-2 detector (a detector that uses the RoBERTa model with GPT-2 output and predicts whether it is likely AI-generated or not) and the <a href=\"http:\/\/gltr.io\/\" target=\"_blank\" rel=\"noopener noreferrer\">GLTR<\/a> model, which also has access to top GPT-2 output and operates similarly. <\/p>\n<p>We can see an example of the output of this model on the content I copied from the paper above:<\/p>\n<div class=\"wp-block-image\">\nThe prevalence of purple and red is indicative of non-AI-generated content. I\u2019m happy to report that the authors of this paper didn\u2019t use GPT to generate it.<\/div>\n<p><strong>Three datasets<\/strong> Web500M (a random sampling of 500 million English webpages), <a href=\"https:\/\/github.com\/openai\/gpt-2-output-dataset\" target=\"_blank\" rel=\"noopener noreferrer\">GPT-2 Output<\/a> (250k GPT-2 text generations) and Grover-Output (they internally generated 1.2M articles using the pre-trained <a href=\"https:\/\/github.com\/tencent-ailab\/grover\/blob\/main\/README.md\" target=\"_blank\" rel=\"noopener noreferrer\">Grover-Base model<\/a>, which was designed to detect fake news).<\/p>\n<p><strong>The Spam Baseline<\/strong>, a classifier trained on the <a href=\"http:\/\/nlp.cs.aueb.gr\/software_and_datasets\/Enron-Spam\/index.html\" target=\"_blank\" rel=\"noopener noreferrer\">Enron Spam Email Dataset<\/a>. They used this classifier to establish the Language Quality number they would assign, so if the model determined that a document is not spam with a probability of 0.2, the Langage Quality (LQ) score assigned was 0.2.<\/p>\n<p><!-- START INLINE FORM --><\/p>\n<div class=\"nl-inline-form border py-2 px-1 my-2\">\n<div class=\"row align-items-center\">\n<div class=\"col-12 col-lg-3 col-xl-4 pe-md-0 pb-2 pb-lg-0\">\n<p class=\"inline-form-text text-center mb-0\">Get the daily newsletter search marketers rely on.<\/p>\n<\/div>\n<div class=\"col-12 col-lg-7 col-xl-6  pe-lg-0\">\n<div class=\"form-nl-inline\">\n<div class=\"row g-0\">\n<div class=\"form-group col-12 col-lg-8 mb-0\"><\/div>\n<div class=\"col-12 col-lg-4 col-xl-4 mb-0\">\n<div class=\"d-none\">\n<p>Processing\u2026Please wait.<\/p>\n<\/div>\n<p>\t\t\t\t\t\t\t<button class=\"btn btn-dark inlineButton w-100 rounded-0 font-weight-bold\">SUBSCRIBE<\/button>\n\t\t\t\t\t\t<\/p><\/div>\n<\/div>\n<\/div>\n<\/div>\n<div class=\"col-12 col-lg-2 col-xl-2 ps-1\">\n<p class=\"text-center mb-0\"><a class=\"nl-terms\" href=\"https:\/\/searchengineland.com\/termsofuse\" target=\"_blank\" rel=\"noopener noreferrer\">See terms.<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!-- END INLINE FORM --><\/p>\n<h2><span class=\"ez-toc-section\" id=\"An_aside_about_spam_prevalence\"><\/span>An aside about spam prevalence<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>I wanted to take a quick aside to discuss some interesting findings the authors stumbled upon. One is illustrated in the following figure (Figure 3 from the paper):<\/p>\n<div class=\"wp-block-image\">\n<\/div>\n<p>It&#8217;s important to notice the score below each graph. A number toward 1.0 is moving to a confidence that the content is spam. What we&#8217;re seeing then is that from 2017 onward \u2013 and spiking in 2019 \u2013 there was a prevalence of low-quality documents.<\/p>\n<p>Additionally, they found the impact of low-quality content was higher in some sectors than others (remembering that a higher score reflects a higher probability of spam).<\/p>\n<div class=\"wp-block-image\">\n<\/div>\n<p>I scratched my head on a couple of these. Adult made sense, obviously. <\/p>\n<p>But books and literature were a bit of a surprise. And so was health \u2013\u00a0until the authors brought up Viagra and other &#8220;adult health product&#8221; sites as \u201chealth&#8221; and essay farms as &#8220;literature&#8221; \u2013 that is.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"Their_findings\"><\/span>Their findings<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Aside from what we discussed about sectors and the spike in 2019, the authors also found a number of interesting things that SEOs can learn from and must keep in mind, especially as we start to lean on tools like <a href=\"https:\/\/searchengineland.com\/what-is-chatgpt-and-why-seos-should-care-392165\" target=\"_blank\" rel=\"noopener\">ChatGPT<\/a>.<\/p>\n<p>Low-quality content tends to be lower in length (peaking at 3,000 characters).<\/p>\n<p>Detection systems trained to determine whether text was written by a machine or not are also good at classifying low vs. high-level content.<\/p>\n<p>They call our content designed for rankings as a specific culprit, though I suspect they&#8217;re referring to the trash we all know shouldn&#8217;t be there.<\/p>\n<p>The authors do not claim that this is an end-all-be-all solution, but rather a starting point and I&#8217;m sure they&#8217;ve moved the bar forward in the past couple of years.<\/p>\n<h2><span class=\"ez-toc-section\" id=\"A_note_about_AI-generated_content\"><\/span>A note about AI-generated content<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Language models have likewise developed over the years. While GPT-3 existed when this paper was written, the detectors they were using were based on GPT-2 which is a significantly inferior model.<\/p>\n<p>GPT-4 is likely just around the corner and <a href=\"https:\/\/in.mashable.com\/tech\/45824\/google-to-unveil-its-chatgpt-killer-ai-sparrow-soon-all-you-need-to-know\" target=\"_blank\" rel=\"noopener noreferrer\">Google&#8217;s Sparrow<\/a> is set for release later this year. This means that not only is the tech getting better on both sides of the battleground (content generators vs. search engines), combinations will be easier to pull into play.\u00a0<\/p>\n<p>Can Google detect content created by either Sparrow or GPT-4? Maybe. <\/p>\n<p>But how about if it was generated with Sparrow and then sent to GPT-4 with a rewrite prompt?<\/p>\n<p>Another factor that needs to be remembered is that the techniques used in this paper are based on auto-regressive models. Simply put, they predict a score for a word based on what they would predict that word to be given those that preceded it.<\/p>\n<p>As models develop a higher degree of sophistication and start creating full ideas at a time rather than a word followed by another, the AI detection may slip.<\/p>\n<p>On the other hand, the detection of simply crap content should escalate \u2013 which may mean that the only &#8220;low quality&#8221; content that will win, is AI-generated.<\/p>\n<p>The post <a href=\"https:\/\/searchengineland.com\/google-study-generative-models-redictors-page-quality-392460\" target=\"_blank\" rel=\"noopener\">Is Google using a ChatGPT-like system for spam and AI content detection and ranking websites?<\/a> appeared first on <a href=\"https:\/\/searchengineland.com\/\" target=\"_blank\" rel=\"noopener\">Search Engine Land<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"<p>The headline is intentionally misleading \u2013 but only insofar as using the term \u201cChatGPT\u201d is concerned. \u201cChatGPT-like\u201d immediately lets you, the reader, know the type of technology I\u2019m referring to, instead of describing the system as \u201ca text-generation model like GPT-2 or GPT-3.\u201d (Also, the latter really wouldn\u2019t be as clickable\u2026) What we will be looking at in this article is an older, but highly relevant Google paper from 2020, \u201cGenerative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study.\u201d What is the paper about? [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"h5ap_radio_sources":[],"cybocfi_hide_featured_image":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-7964","post","type-post","status-publish","format-standard","hentry","category-best-seo-blogs"],"featured_image_src":null,"featured_image_src_square":null,"author_info":{"display_name":false,"author_link":"https:\/\/craighordlow.com\/author\/"},"_links":{"self":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts\/7964","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/comments?post=7964"}],"version-history":[{"count":0,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/posts\/7964\/revisions"}],"wp:attachment":[{"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/media?parent=7964"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/categories?post=7964"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/craighordlow.com\/index.php\/wp-json\/wp\/v2\/tags?post=7964"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}