AI crawlers and their robots.txt tokens

Each operator below names its agents with their own robots.txt tokens, though some say their user-triggered fetchers may not follow robots.txt. Most operators listed here run more than one agent, typically some mix of a crawler that puts sites into answers, an agent that fetches a page when someone asks, and a crawler or token for model training. Each page covers one of them, cites the operator's own documentation, and gives the robots.txt lines to allow or block it.

How often real sites block them is a separate question, with its own sample and method: what the robots.txt files of 45 audited sites allowed.