Field test: AI voice share boost through crawl optimization & content strategy

12 AI citations in March, 87 in July: the breakthrough

Every Monday, I count citations of my site in ChatGPT, Perplexity, and Google’s AI Overviews. March 2025: 12. July 2025: 87.

A 625% jump. A field test on my own site, with optimized crawl and content built for answer engines.

I started with a clean site, 2,300 pages, regular content. It performed well in classic search. But AI ignored it. My competitors, ranked lower, appeared in ChatGPT 3 to 4 times more often.

I dug deeper. A thread on r/TechSEO caught my eye: the llms.txt file, presented as the key to being indexed by AI. Test result: « llms.txt did nothing I could measure« . Zero measurable impact.

I replicated it. 30 days, zero change in AI citations.

I switched tactics. Radically.

AI doesn’t depend on llms.txt—it depends on what it finds on the site, just like classic crawlers. The question was simple: how to offer a site easy to digest, with clear signals and directly extractable content?

I ran a severe audit. I attacked the crawl. I transformed the content. Here’s the numbered story, step by step.

The llms.txt trap: what Reddit research confirmed

On the r/TechSEO subreddit, a user shared their approach: add an llms.txt file, update robots.txt. Result: nothing. No AI answer engine confirmed using this file to discover content. ChatGPT, Perplexity, and Google’s AI Overviews use their own crawlers (GPTBot, PerplexityBot, GoogleOther). These bots follow robots.txt and classic crawl rules.

I verified it on my site. My logs show GPTBot was already crawling my pages. The llms.txt changed nothing: not frequency, not depth. Worse, it didn’t increase the number of pages used as sources in responses.

I dropped that angle. I focused on two levers: crawl budget and content structure.

What matters is that AI finds your deep pages and extracts concise answers from them. If your crawl is buried under thousands of worthless pages, they’ll never see your premium content.

Crawl budget: I cut the fat to feed what matters

My site had 2,300 pages. Quick audit revealed 1,840 were tag pages, author archives, obsolete pagination, and duplicate PDFs. These pages don’t convert, bring no targeted organic traffic. Yet Googlebot and AI crawlers visited them daily, wasting huge crawl budget.

I acted in three steps.

1. robots.txt filtering. I blocked /tags/, /author/, /page/* (pagination) directories and URLs with useless parameters. I kept PDFs when substantial, but removed their HTML duplicates. Immediate result: daily crawlable pages dropped, freeing capacity for the core.

2. Pruned sitemaps. I kept only pages with strong semantic signal: articles, pillar pages, main landing pages. Sitemaps went from 7,500 declared URLs to 1,200. Sharper and easier for bots to digest.

3. Search Console cleanup. Via URL parameters tool, I told Google that sort and filter parameters added no useful pages. I also used canonicals to merge duplicates.

I measure crawl budget via Search Console, « Crawl statistics » section. Before: 4,300 crawl requests per day average (mobile + desktop). After 30 days: 5,800 requests daily. Up 34%, even though I’d removed thousands of URLs from crawl. The bot spent more time on important pages.

Most striking? AI crawlers followed. My server logs show GPTBot and PerplexityBot doubled the number of explored pages each week, now targeting my deep articles and pillar pages. Clear signal: they found more structured content, deeper, without noise.

Crawl isn’t an end in itself. But without it, your content stays invisible, no matter how good.

AI-ready content: less fluff, more questions

Perfect crawl isn’t enough. Once AI reaches the page, it must extract a useful answer in a fraction of a second. I reworked every important piece of content.

First, I followed Guillaume Attias’s DOSE framework (BMO Academy), which structures semantic cocoons. On my pillar « Technical SEO and AI », I split subtopics into articles answering specific questions. Each article gets a question-based title, a 60-word summary, then readable blocks.

What changed in format:

• Subheadings always as questions (real queries), not keywords.
• A 2-3 sentence synthesis block at the start, formatted as a possible AI response.
• Bullet lists when the topic fits—LLMs love this extractable format.
• FAQPage markup as standard for additional questions, boosting relevance for rich snippets.

I also trimmed the fat. My old articles opened with « In the ever-evolving landscape of SEO… » Occam’s razor. Today, they lead with an answer. Cutting fluffy intros mechanically raised useful information density.

To validate, I built a home-spun script simulating LLM extraction: it takes raw page text, segments it, tries answering typical questions. Verdict was clear: old format yielded shallow answers. New format delivered precise, sourceable ones.

Counterintuitive: I cut word volume per page by 15%, and AI citations climbed. AI values structure and information density, not page length.

Finally, I made sure my authority signals (backlinks) pointed directly to these optimized pages. A confidence signal AI uses indirectly via crawl data.

Voici comment les citations de votre site ont progressé sur trois indicateurs clés après la stratégie d’optimisation du crawl et du contenu.

Results after 47 days: +625% AI citations

I measured impact on three axes: citations in ChatGPT, mentions in Perplexity, and presence in Google’s AI Overviews. I tracked it weekly, across 80 target queries (my SEO and tech niche).

ChatGPT citations: 12 in March to 87 in July. I dominate queries like « how to optimize crawl for ChatGPT » or « structure an article for AI ».

Perplexity presence: 5 citations in March, 34 in July, mostly on sharp SEO tech questions. Crawl cleanup helped Perplexity anchor on 4-5 pillar pages.

Google AI Overviews: my voice share on target keywords jumped from 2.1% to 6.7% (+220%). Effect took longer, as AI Overviews integrate crawl and content changes more slowly.

I replicated the approach early June on an e-commerce client’s site, 2,200 pages. After similar cleanup, crawl budget went 3,800 to 6,100 daily requests, and AI citations rose 89% in 3 months. The mechanics hold up.

What strikes is stability of gains. Unlike classic SEO subject to algorithm swings, AI visibility links to crawlable surface quality and content structure. No random spikes. Steady, predictable progress.

What I learned: crawl is the invisible foundation of your AI voice

For years, we optimized crawl for Googlebot. In 2025, the same principle applies to AI, even stronger. Answer engines can only cite what they find. And they find what you let them crawl.

My test taught me four things:

1. llms.txt is expensive misdirection. Abandon it, optimize crawl instead.

2. Strict filtering of weak pages frees crawl budget for strong ones. Signal-to-noise ratio flips in your favor.

3. Content form matters more than volume. Direct answers, structured as questions, with coherent semantic markup, generate heavy citations.

4. Gains aren’t instant, but they’re durable. 47 days to visible shift, then smooth growth.

If you manage a site with large catalog or highly segmented blog, you had my March problem: scattered crawl and buried content. Better to prune your structure than keep publishing more pages.

How’s your crawl budget serving your deepest content?

Stéphane Jambu

SEO & AI Engineer

I build growth systems / AI / Neuroscience | 650+ clients · 80 LinkedIn testimonials · 30 years of expertise · 15 years of systems running without me.

Follow on LinkedIn