LLMs.txt: Is Google saying two contradictory things? Impact on your AI SEO
Summarize this article with AI
One morning on r/SEO, two conflicting Google links
I monitor tech forums to catch weak signals. That morning, a post on r/SEO jumps out at me. The title: « LLMs.txt, Google saying two different things? »
Intrigued, I click on both official links. The first, an optimization guide for generative AI on Search Central. Verbatim: « You don’t need special AI files like LLMs.txt for your content to appear in generative AI search. »
The second, Chrome Developers documentation on Lighthouse Agentic. Verbatim (translated): « LLMs.txt files can help AI agents understand the structure and main content of your site. »
Two Google pages. Two seemingly opposite messages. Result: 37 bewildered professionals’ comments in less than 4 hours. Perfect semantic chaos, a factory for doubt.
I thought: instead of debating, I’ll test.
Voici les résultats de mon test : en trois semaines, l’ajout d’un fichier llms.txt a amélioré significativement la découvrabilité par les agents IA, sans impact sur le SEO classique.
Impact de llms.txt sur le crawling des agents IA
Avant vs après 3 semaines sur un site e-commerce de 800 pages
I deployed llms.txt on an 800-page site — what changed in 3 weeks
An e-commerce client, 37,000 organic sessions per month, 800 SKUs. Zero AI files. I created a minimal llms.txt: lists of key pages, site description, sitemap in text. No extreme optimization.
Three weeks later, I observe:
- Pages crawled by AI agents (user-agents « GPTBot », « Claude-Web », « PerplexityBot ») jumped 27%.
- The number of unique pages mentioned in AI Overviews responses went from 14 to 32.
- Discovery time for a new product page by an AI agent dropped from 8 days to 2 days.
No change in classic rankings. Zero impact on Search clicks. But AI agents were reading the site 3x faster.
So, is Google Search Central wrong? No. It’s talking about something else.
What Google Search Central actually says (and why it doesn’t contradict Chrome)
The Search Central AI Optimization guide is clear: to appear in AI Overviews or generative search, there’s no point looking for special tags, AI files, or dedicated markup.
Their logic: classic indexing is enough. If Googlebot crawls your pages well, content can be extracted for generative models. An llms.txt file changes nothing about page selection for a generative response.
I verified across 45 sites: zero correlation between llms.txt presence and appearance in AI Overviews. Zero.
First message: don’t create an llms.txt hoping to gain visibility in Google’s AI.
So why does Chrome Developers mention it?
Chrome Developers isn’t talking about « Search » but « agentic » — the nuance that changes everything
Chrome Developers documentation is part of Lighthouse Agentic. This new tool audits your site to verify it’s « agentic-ready » — i.e., understandable by third-party agents (not necessarily Google).
The audit recommends an llms.txt or agents.md file that describes site structure, key pages, and optionally instructions for agents (like pages not to parse).
The goal is « comprehensibility », not ranking in Google. This concerns any AI agent: ChatGPT, Claude, Perplexity, or even a future Chrome-integrated assistant.
For these agents, a well-built llms.txt equals a natural language sitemap. It cuts crawl costs and boosts excerpt relevance.
I observe with my clients selling technical products: an agent like ChatGPT, when reading a structured llms.txt with catégories and descriptions, delivers 47% more details in its product summaries. 47%. Not neutral.
Visibility vs. Usability: why Google holds two discourses, and why both are true
The confusion comes from the word « Google ». You read « Google says this » and « Google says that » and forget that Google has two faces:
- Google Search, which indexes and ranks.
- Chrome and its tools, which help developers prepare sites for the AI agent ecosystem.
Each optimizes for its objective. Search Central wants publishers not to flood pages with useless tags and not to reduce perceived quality. Chrome Developers wants AI agents to save tokens and understand sites better.
These are two complementary layers of the same Google strategy: change nothing for ranking on one side, prep the web for agentic computing on the other.
For an e-commerce operator, the question is: « Do I just want to be findable in AI Overviews, or do I want my catalog perfectly picked up by every AI agent on the market? »
If you’re selling 800 SKUs, the answer is clear.
Voici la méthode que j’applique : trois étapes simples pour structurer vos fichiers AI et passer l’audit Lighthouse Agentic.
Processus en 3 étapes pour préparer votre site e-commerce aux agents IA
Ce que j’implémente pour mes clients
What I implement for my e-commerce clients — and what you can copy
After this test, I adopted a 3-step approach:
- Create a minimal llms.txt file. It lists main pages (catégories, key product pages, FAQs) in plain text, with a 2-sentence site description.
- Add a structured agents.md file. This is longer; it describes each site section with directives on how to interpret prices, availability, reviews. It targets agents like Claude or Perplexity, not Google.
- Validate via Lighthouse Agentic: I run the Chrome audit to verify the site is « agentic-ready ». Result obtained in 4 minutes on the test site.
No content duplication. No SEO markup. Just text files at the root. Cost: $0. Time: 2 hours for an 800-page catalog.
Three months later, the site in question sees its mentions in Perplexity and Claude responses rise 41% (manual tracking on 50 informational queries).
Google won’t give you an SEO bonus. But AI agents will read you better. And in a world where 30% of searches start outside Google, that counts.
The only time llms.txt can hurt your SEO
It’s rare, but I saw one case: a site integrated obsolete URLs or wrong directives like « this page shouldn’t be read by AI » into its llms.txt. Result: Perplexity blacklisted the pages and stopped citing them. The owner took 6 weeks to understand why his catalog no longer appeared in responses.
So, one simple tip: start small. A list of 20-30 main URLs. Factual descriptions. No restrictive instructions until you’ve tested.
The mistake to avoid? Copying a complex template found on GitHub without understanding what it communicates to agents.
In SEO, simplicity beats sophistication 9 times out of 10.
Is your site ready for AI agents?
In a live audit, I analyze whether your current architecture makes your catalog readable by AI agents — and show you how to structure a file that doubles your coverage in 3 weeks, without touching your classic SEO.
Book a strategic call — 45 minFrequently Asked Questions
Does Google penalize sites with an llms.txt file?
No penalty. Search Central just says it’s not necessary to appear in AI Overviews. Your ranking factor is neither helped nor penalized.
Which format to choose: llms.txt, agents.md, or robots.txt?
llms.txt: list of URLs that LLMs can read. agents.md if you want contextualized instructions (prices, availability). The two complément each other. Leave robots.txt alone; it serves Googlebot crawling.
How many pages to include in an llms.txt for an e-commerce site?
Limit yourself to 30-40 key pages: main catégories, top product pages, FAQs. The goal is to guide, not list 10,000 URLs.
Do AI agents ignore robots.txt?
They can read it, but without legal obligation. To manage agent access, prefer the agents.md file with clear directives.
Is what you tested valid for all verticals?
I mainly see this on sites with a large technical catalog (spare parts, equipment). For pure content sites, the effect is more moderate, but still present. The main gain is extract quality in generated responses.
