AI Overviews: the true cost of misconfigured robots.txt (NewsRoom data)

The phone call that dismantled a myth

A client calls me on a Tuesday morning. He invested $8,000 in overhauling his semantic architecture. His organic traffic just dropped 34% in three weeks. He doesn’t understand.

« I blocked Google-Extended in robots.txt. Normally, AI can’t scrape my content anymore. »

Normally.

What I discovered inspecting his site confirms data that John Shehata, CEO of NewzDash, revealed in July 2026: 15.5% of trending US queries now display Top Stories directly inside AI Overviews. And robots.txt blocking doesn’t change that.

My client wasn’t protected. He’d just made himself invisible to a massive traffic source.

Les données de NewzDash révèlent des disparités fortes dans l’intégration des Top Stories au sein des AI Overviews. Les régions et secteurs d’actualité n’égalent pas le taux US de 15.5% : certains domaines comme l’Entertainment dépassent largement cette moyenne, tandis que Health et Science restent sous 5%.

15.5%: the figure John Shehata released in July 2026

NewzDash gives us concrete numbers. When Google displays Top Stories on trending queries, 15.5% now incorporate them directly into the AI Overview in the US. 17.46% in the United Kingdom.

Entertainment: 35%. World News: nearly 32%. Health and Science stay under 5%. The impact concentrates on breaking news and newsrooms.

Google doesn’t hide it. It’s the same strategy as 2007 with Universal Search: inject vertical content directly into results. But this time, instead of displaying a list of blue links, you get an AI-generated answer.

Why robots.txt doesn’t protect you from AI Overviews

Many editorial teams confuse two distinct crawlers.

Googlebot: the classic robot that indexes your pages for search. It scans, ranks, feeds the SERP.

Google-Extended: a crawler launched in 2023, designed solely for training generative AI models (Bard, Vertex AI).

When you add a line Disallow: / for Google-Extended in your robots.txt, you’re simply saying: « Don’t train on my content. »

You’re not saying: « Don’t show me in AI Overviews. »

AI Overviews come from the normal index. Googlebot read everything. Blocking Google-Extended changes nothing. Zero impact on your presence in those carousels.

John Shehata repeated on LinkedIn: « robots.txt alone doesn’t protect from AI. » He’s right. Publishers who believe they’re protected are mistaken.

Pour un groupe média générant 2.3 millions de sessions mensuelles, les chiffres deviennent tangibles. Ce waterfall illustre le chemin du trafic potentiel jusqu’à la perte de revenus due à l’absence de contrôle des AI Overviews.

What misconfiguration truly costs

Take a concrete case. A media group I track generates 2.3 million organic sessions per month. Based on NewzDash’s proportions, 15.5% of its trending queries could appear in Top Stories within AI Overviews.

2.3 million × 15.5% = 356,500 potential sessions.

With an average advertising RPM of $9.50 observed at this publisher type, monthly revenue loss reaches $3,386.

Per month.

This scénario costs $40,632 per year. Not because Google penalizes the site. But because competitors control the carousels instead.

The worst part: the robots.txt file didn’t even prevent appearance in AI Overviews. The content shows up anyway, because blocking only concerns the training crawler. Except the traffic doesn’t follow. The carousel displays other, better-optimized sources.

My client hadn’t secured his revenue. He’d just handed it to the competition.

What actually works to control your AI presence

To prevent your pages from appearing in AI Overviews, there’s one solution: the nosnippet or max-snippet:0 meta tag.

Except that setting also removes all your snippets from normal results. No description. No rich title. No anchored link.

Plainly: to exit AI Overviews, you become a ghost link in the SERP. Lost clicks typically exceed expected gains.

There’s another path. It’s the one I apply with the publishers I work with.

1. Accept the AI Overview and structure it. Google cites multiple sources. Being among them keeps you visible. For that, your content must be modeled as entities. A news article without semantic markup is a dispatch lost in a swamp.

2. Build a semantic cocoon. I organize pages into thematic silos with strong pillar pages. Each article reinforces the parent entity. The Googlebot index understands you’re the primary source. Your odds of being cited alone in AI Overviews increase.

3. Track citations. NewzDash or Search Console exports show which queries cite you in AI Overviews. We adjust content to restart clicks (compelling title, proper markup).

4. Don’t touch Google-Extended. Blocking this crawler accomplishes nothing. It doesn’t reduce appearance. Worse, it might signal to Google that you’re hiding something. I don’t take that risk.

Avez-vous configuré votre robots.txt sans penser aux conséquences ? Ce radar vous aide à auto-évaluer vos restrictions : chaque critère représente un risque d’invisibilité accrue dans les résultats de recherche et les découvertes algorithmiques.

The anti-pattern I still see too often

In audits, I regularly find robots.txt files stacking restrictions: Disallow: / for Google-Extended, but also for GPTBot, CCBot, sometimes even Googlebot-News!

That's a mistake. Blocking Googlebot-News excludes your site from Google News. Yet Top Stories in AI Overviews come mostly from the Google News index. By thinking you're protecting yourself, you're cutting off access to your traffic source itself.

Result: a locked-down site that generates nothing. Not in AI, not in News, not in Discover.

One extra line. That's enough to lose thousands of dollars monthly.

Stéphane Jambu

SEO & AI Engineer

I build growth systems / AI / Neuroscience | 650+ clients · 80 LinkedIn testimonials · 30 years of expertise · 15 years of systems running without me.

Follow on LinkedIn