With the rise of AI, web crawlers are suddenly controversial - The Verge

the state of robots.txt is a testament to the value of social contracts.

future proofing and strong gaurantees aren't a true reality in most domains. with that said, LLMs definitely have changed the price of serving information freely on the web and new contracts are needed.

unfortunately those are being replaced by costly contracts that are creating walled gardens.

in the end, the future is inevitable and so is the spread of information (and misinformation).

The article discusses the increasing controversy surrounding web crawlers in the age of AI. With AI companies using websites' data for training models, the implicit agreement behind robots.txt—a file that allows website owners to control which crawlers can access their sites—is breaking down. The article traces the history of robots.txt, explores different types of web crawlers, and examines the responses of various publishers and platforms to AI data scraping. It also discusses the legal and ethical implications of this issue and potential future solutions.