AI
AI labs agree to read one robots.txt training opt-out
Google, Microsoft, OpenAI and Anthropic said on Friday they will honour a single shared policy token in robots.txt, letting a publisher refuse generative-AI training without writing a rule for every crawler. The signal comes from an IETF draft by Anthropic's Kevin Kelley; Anthropic has begun implementing it, OpenAI aims to follow in the coming months, Microsoft on 1 February 2027 and Google in 2027. The token states a preference about training, not a block on crawling.