I am learning from the AI Insights updates on Cloudflare Radar. I have long been a champion of how we’ll be automating the onboarding of clients, bots, agents, and other non-human users of the web. I consider my work on data.json for the Obama administration and my ongoing investment into apis.json a PHD in this area. Cloudflare has a unique position as the DNS edge for much of the web when it comes to understanding this realm, and I find their breakdown of standards used to produce their AI insights as part of Cloudflare radar to be educational. robots.txt — https://www.rfc-editor.org/rfc/rfc9309.html The Robots Exclusion Protocol, originally from 1994 and formally standardized as RFC 9309 in September 2022 IETF. A plain-text file at a site’s root that tells crawlers which paths they may or may not access. Sitemap — https://www.sitemaps.org/protocol.html An XML protocol introduced by Google in 2005 (with Yahoo and Microsoft adopting in 2006) Wikipedia that lists URLs on a site along with metadata like last-modified date and change frequency to help search engines crawl more efficiently. AI rules in robots.txt — https://datatracker.ietf.org/doc/draft-ietf-aipref-attach/ The IETF AIPREF working group’s draft extension to robots.txt that adds a Content-Usage directive GitHub so sites can express AI-specific preferences (training, search, inference) alongside traditional Allow/Disallow rules. Link headers — https://www.rfc-editor.org/rfc/rfc8288.html RFC 8288 (Web Linking) defines a model for relationships between web resources and serializes those links in HTTP headers via the Link header field IETF. Used for pagination, alternate representations, and resource relationships. OAuth discovery — https://datatracker.ietf.org/doc/html/rfc8414 RFC 8414 defines a metadata format at /.well-known/oauth-authorization-server that OAuth 2.0 clients use to discover an authorization server’s endpoints and capabilities IETF. Markdown negotiation — https://vercel.com/blog/making-agent-friendly-pages-with-content-negotiation Using the standard HTTP Accept: text/markdown header so servers can return clean markdown to agents and HTML to browsers from the same URL. RFC 7763 registered the text/markdown MIME type in March 2016 Ekamoira. Universal Commerce Protocol (UCP) — https://ucp.dev/ An open-source standard developed by Google with Shopify, Etsy, Wayfair, Target, Walmart, and 20+ partners Medium that lets AI agents discover merchant capabilities and complete commerce flows (checkout, identity linking, order management) through a standardized manifest at /.well-known/ucp. Content signals — https://blog.cloudflare.com/content-signals-policy/ Cloudflare’s Content Signals Policy extends robots.txt with three categories — search, ai-input, and ai-train — so operators can express how crawlers may use content after it’s been fetched Cloudflare. Released under CC0. OAuth Protected Resource — https://www.rfc-editor.org/rfc/rfc9728 RFC 9728, published April 2025, defines a metadata format at /.well-known/oauth-protected-resource that OAuth 2.0 clients use to learn what scopes, token types, and authorization servers an API requires RFC Editor. Widely used for MCP authorization discovery. Agent Skills — https://agentskills.io/home An open standard originally developed by Anthropic for filesystem-based SKILL.md files — folders of instructions, scripts, and resources that agents load on demand Agent Skills to specialize for domain tasks. Web Bot Auth — https://blog.cloudflare.com/web-bot-auth/ A Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture) that uses HTTP Message Signatures (RFC 9421) Anchor Browser Docs so bots can cryptographically sign requests and sites can verify their identity without…