Gary Illyes from Google said on LinkedIn that if your server returns a 500/503 HTTP status code for an extended period of time for your robots.txt file, then Google may remove your site completely from Google Search.
Related Posts
Google: Internal Linking Gives Google A Sense Of Page Level Importance
Google’s John Mueller was asked how someone can help Google understand which pages of a site it really wants Google to index. A webmaster said that Google was only indexing 10% or so of the site’s pages in the sitemap file. He asked how can they notate which pages are more important if Google doesn’t look at the priority field in XML Sitemaps.
