Google Adds More Details To Robots.txt Specifications Help Document

Google has added a bunch of new supporting details to the robots.txt specification document. Jamie Indigo noticed these changes and explained what was added include “explicit details on the files for IDNs, IP-addresses, and port numbered hostnames.”

SearchEngineRoundtable1?d=UCg8P nGCaA | Craig Hordlow SearchEngineRoundtable1?d=yIl2AUoC8zA | Craig Hordlow SearchEngineRoundtable1?d=hl8ANDcrVaY | Craig Hordlow SearchEngineRoundtable1?d=V t1I SPZMU | Craig Hordlow SearchEngineRoundtable1?d=Gu391qSwH A | Craig Hordlow SearchEngineRoundtable1?d=cGdyc7Q 1BI | Craig Hordlow SearchEngineRoundtable1?i=NjXVVmAmBC0:nZ7nM FG2oE:V sGLiPBpWU | Craig Hordlow SearchEngineRoundtable1?i=NjXVVmAmBC0:nZ7nM FG2oE:F7zBnMyn0Lo | Craig Hordlow SearchEngineRoundtable1?d=MbsSfiz sEw | Craig Hordlow SearchEngineRoundtable1?d=qj6IDK7rITs | Craig Hordlow

NjXVVmAmBC0 | Craig Hordlow