Last December, a small text file called llms.txt appeared on Google’s own developer website. An internal system had created it, nobody had switched that system off, and the file was gone within hours.
It was a fitting cameo, because Google doesn’t use the file at all. Yet more than a quarter of the sites in one large study still publish it. Some advice refuses to stay buried. 🧟
The promise was simple. Put llms.txt in your site’s root folder, point it at your best pages, and AI tools would start citing you. It was free, it took twenty minutes, and it felt like progress. Nobody ever showed it moving a single number.
What the evidence says
In June 2026, Google added a mythbusting section to its AI optimisation guide. It says llms.txt neither helps nor hurts visibility in Search, because Google’s AI answers run on the same index as regular results.
Other AI tools barely touch it. Ahrefs checked around 38,000 sites that publish the file. In May 2026, 97% of those files got zero requests, and AI search crawlers made up about one percent of the visits that did arrive.
Google’s John Mueller compared llms.txt to the old keywords meta tag. A site describing itself is only a claim, and the engine still reads the page to check it.
Who it’s for
The format was built in 2024 for coding assistants that need to read technical documentation quickly, and it still does that job well. If developers build software on your API, keep it. If you run review or comparison pages, you can let it go.
A better use of the twenty minutes
Here’s the good news. A more common problem is also easier to fix. A 2026 report from Otterly, based on over a million AI citations, found that about 73% of sites had technical barriers keeping AI crawlers out. Most of your competitors are probably in that group.
Three checks to get ahead of them:
- Open robots.txt and look for AI crawlers by name, such as GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. One blanket rule often blocks the search bots along with the training bots.
- Ask your CDN provider what it does with AI crawlers by default. Some block them automatically.
- Check the raw HTML of your key pages. Tables that load only through JavaScript can look empty to a crawler.
Halloween is a good night to bury a few habits for good. At Joker Partners, we’re always happy to compare notes on what moves the numbers. Happy Halloween. 🎃

