AI giants pledge standardized protocol for website scraping blocks

OpenAI, Google, Microsoft, and Anthropic pledged to introduce a single standardized instruction allowing website publishers to block content scraping for AI training.

Publishers currently must implement multiple blocking commands for different web crawlers, though the new unified setting will permit scraping by default until changed.

Microsoft will implement the feature on February 1, 2027, while Google and OpenAI plan rollouts in 2027 and coming months respectively amid ongoing legal disputes.