AI Search: The Crawlability Checklist to Complete Before Optimizing for AI Agents

A Client, an Audit, a Page That Doesn’t Exist

I’m going to tell you something agencies hate to hear.

A client calls me on a Tuesday morning. He invested $7,500 in an “AI search” optimization layer. Eight weeks later, zero citations. Not one mention in ChatGPT, Perplexity, or Google’s AI Overview.

The problem wasn’t the content. It was the architecture.

The robots.txt file was blocking the most common AI bots. GPTBot, ClaudeBot, PerplexityBot. All blocked. One line was enough. One server configuration line.

The Ahrefs guide published by Mateusz Makosiewicz on September 25, 2026 reminds us. Google says its AI features still rely on its core search systems. Your content must always be crawlable, indexable, and useful. Microsoft adds: Bing’s index is evolving from a list of pages into a source of facts that AIs can use to build answers.

Put these two ideas side by side. AI search still depends on traditional search. Ranking is no longer the only job. Your content must also provide facts that AIs understand, trust, and cite.

But everything starts with access. No access, no citation. No citation, no visibility.

Nothing.

This guide has already been picked up by 57 external sites. It’s not an opinion. It’s a convergence.

With this client, we fixed robots.txt in 20 minutes. We resubmitted the URLs in Search Console. We checked the 404, 499, and 5xx statuses in the AI bot logs. Four weeks later, the first GPTBot crawls were recorded. Citations started to appear.

The lesson is simple. Before writing content for AI agents, ask whether these agents can even read it.

That’s uncomfortable. Because it doesn’t sell a new miracle offer. It sells foundational work. Work that demands rigor, not magic.

Clear.

Crawlability: The First Step. Not the Most Spectacular.

An AI bot reads your page like Googlebot. If it can’t reach it, it won’t cite it. It’s mechanical.

The Ahrefs guide puts this check first. If AIs can’t access your pages, nothing else matters. Before citing your content, they need to be able to read it.

The errors to track are precise. 404, 499, 5xx. Three codes. A 404 is a missing URL. A 499 is an interrupted request. A 5xx is a server crashing. Each has its fix.

In Bot Analytics, I watch the failed requests from AI bots. I filter by status code. I click on the number of affected pages. I identify the URLs to fix.

In Web Analytics, I check 404 pages receiving traffic from AI Search. If the same broken URL keeps coming back, I redirect to the most relevant live page. Sometimes a recurring 404 hides a content opportunity. The AI sends users to a topic you don’t cover. The right answer isn’t always a redirect. Sometimes it’s a page to create.

Also check server settings, firewall, CDN. Cloudflare, for example, controls access for AI crawlers. Its AI Crawl Control documentation explains how the firewall and bot management can block crawlers without you knowing. One overly strict Cloudflare rule is enough. The bot gets a 403. Your content is invisible.

Simple.

Crawlability isn’t glamorous. It doesn’t generate likes. It generates citations. Without it, everything else collapses.

An e-commerce site with 1,200 product references lost four months on enriched content for AI agents. The block came from an obsolete security rule blocking GPTBot. We lifted the block. The first cited excerpts appeared within three weeks.

No magic. Access.

The first step is not spectacular. It remains vital.

Avant d’ajouter la moindre couche de contenu pour les agents IA, il y a cinq contrôles à passer. Voici la séquence exacte, dans l’ordre.

The 5-Point Checklist Before Touching AI Content

Here is the checklist I use in audits. Five points. Five checks. To do before adding any layer of content for AI agents.

• Check robots.txt. GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended. Each bot must be allowed if you want to be cited. A User-agent: GPTBot directive with Disallow: / is enough to block everything.
• Check 404, 499, and 5xx statuses. In the AI bot logs. Don’t guess. Measure.
• Audit Cloudflare and the CDN. AI Crawl Control shows active blocks. The firewall can block ClaudeBot even when robots.txt is clean.
• Track 404s receiving AI Search traffic. Redirect or create the page. It’s a signal of real intent.
• Check noindex, canonical, and visible content. A noindex page can be crawled, never cited. A misplaced canonical dilutes the signal.

The Ahrefs guide says it: an AI agent can audit these points for you. With the right access, it checks robots.txt, noindex rules, status codes, redirects, and visible content. But you decide what you publish. The editorial decision remains human.

“Optimizing for AI search means improving your pages so they are more likely to be cited by AI assistants when people ask relevant questions.” — Mateusz Makosiewicz, Ahrefs

This list is not theoretical. It unfolds in 72 hours. It doesn’t require extra budget. It requires order.

Ce n’est pas un cas isolé, c’est la norme : sur 15 sites audités chaque semaine, 12 présentent au moins un blocage de bot IA. Le détail parle de lui-même.

The Real Blockage Isn’t the Content. It’s the Pipe.

I see it every week. Teams put thousands of euros into enriched content for AI agents. And GPTBot gets a 403. The server refuses the connection. The bot reads nothing.

Out of 15 sites audited each week, 12 have at least one AI bot block. Twelve. It’s not an exception. It’s the norm.

That’s the paradox. Budgets go into writing optimized answers, snippets, data schemas. And the firewall blocks access. Noindex pages prevent extraction. 404s break the path.

In my view, most “GEO” tactics stack layers before having a crawlable foundation. That’s noise. The foundation is access architecture.

Content isn’t the problem. The pipe is. An inaccessible page is a nonexistent page for AI agents.

The Ahrefs guide says it plainly: “Your content must always be crawlable, indexable, and useful.” The order matters. Access first. Indexing next. Usefulness last.

Many teams confuse optimizing for AI with generating content by AI. It’s not the same thing. Optimizing for AI means making your facts extractable. It means clarifying important passages. It means keeping facts fresh and consistent.

But no extraction happens if the bot can’t reach the page. It’s as simple as that.

A deliberate counterintuitive point: in AI visibility, you don’t win by adding layers. You win by removing blocks. Clean before adding.

Order of magnitude: on a catalog of 4,200 product pages, 300 5xx errors are enough to break AI agents’ trust. They stop coming back. In AI search, trust is built on technical reliability.

Verified.

The DOSE Effect of Clarification: Returning to Fundamentals Calms Teams

The DOSE framework, taught by Guillaume Attias at BMO Academy, explains what happens. Less uncertainty, more serotonin. A reassured team acts faster and more accurately.

AI Search has created enormous hype. Entire teams wonder whether all their SEO is obsolete. Some buy training, tools, and content without checking the foundation. Clarifying is what gets them back on track.

The Ahrefs guide does exactly that. It turns fuzzy hype into a 5-point checklist. No more fog, time for action.

Guillaume Attias describes the clarification effect as a serotonin lever. When uncertainty drops, the brain releases serotonin. Teams regain confidence. They approach technical tasks more calmly.

Why panic when the foundation is known? Crawlability, we’ve known it for a long time. Googlebot has checked it for 25 years. AI bots follow the same principles.

Reframing does a service. It avoids investing in superficial tactics. It brings you back to fundamentals that produce measurable results.

Serotonin fuels decision-making. An anxious team multiplies useless actions. A clarified team executes the right ones.

Clarify before tooling up. Check before creating. Build on solid ground.

That’s the clarification effect.

72 Hours to Complete Crawlability: The Action Plan

Three days. That’s enough to clean up access. Here is the plan.

• Day 1, access audit. Check robots.txt for AI crawlers. Open Cloudflare AI Crawl Control. Log 404, 499, and 5xx codes in Bot Analytics. List blocked URLs.
• Day 2, correction. Allow useful AI bots. Fix 5xx errors. Redirect or remove 404s. Check noindex and canonical. Test one page with an AI agent.
• Day 3, monitoring. Track AI bot crawls in logs. Compare before/after statuses. Measure first citations in ChatGPT, Perplexity, AI Overview. Note recurring 404s pointing to content to create.

An AI agent can perform these checks for you. The Ahrefs guide offers adapted prompts for Letaido, Claude Code, ChatGPT Agents, and Codex. With the right access, the agent checks rules, statuses, redirects, and visible content. You keep the decision to publish.

How much time do you lose on invisible content? Every week spent on inaccessible content is a lost week.

Nothing complicated. Three days, five checks, and a roadmap.

Sites that complete this checklist before optimizing for AI agents start with an advantage. Nothing to do with intelligence: they do the basics.

The foundation is access. It’s also technical trust and the right to be cited.

Why 72 hours? AI bot blocks are rarely complex. A robots.txt directive is fixed in minutes. A Cloudflare rule is lifted in minutes. 5xx errors require a bit more, rarely weeks. Slowness comes from the absence of method, not from complexity.

72 hours.

The Real Question Isn’t “How to Optimize for AI”

The question is simple. Are your pages readable by AI agents?

If the answer is no, everything else waits. Content stratégies, schemas, entities, topical authority. Nothing compensates for a closed door.

The Ahrefs guide says it plainly. It’s not a new discipline, it’s the continuation of SEO. Crawlability first. Indexability next. Usefulness last.

This observation goes against the hype. No magic results promised. Solid foundations. Serious sites need to hear this.

We win by returning to fundamentals. By checking before creating. By treating AI agents as demanding readers, not mysterious algorithms.

I’m not selling you a method. I’m showing the pages.

And you, are your pages ready to be read?

Stéphane Jambu

SEO & AI Engineer

I build growth systems / AI / Neuroscience | 650+ clients · 80 LinkedIn testimonials · 30 years of expertise · 15 years of systems running without me.

Follow on LinkedIn