Is Your Site Invisible to AI Crawlers? Access and Rendering

Can an AI crawler actually fetch your page?

Start here, because it's the most basic failure mode and the easiest to check. A page can be fully allowed in robots.txt and still never get a successful fetch, for reasons that have nothing to do with robots.txt at all:

• The server returns a non-200 status to the bot specifically (some bot-management tools serve a different response to flagged user-agents than to a browser).
• A CDN or WAF rule silently drops the request before your origin server ever sees it.
• The bot's published IP range is blocked at the network level, regardless of what robots.txt says.

Checking this requires looking at what actually happened on a real request, not re-reading your robots.txt file and assuming it's being honored.

Does your page need JavaScript to show its text?

This is the second, and probably more common, failure mode for sites that otherwise look fine. Crawlers fetch raw HTML; most AI crawlers, like most traditional search crawlers, don't run a full browser engine to execute JavaScript the way a person's browser does. If your page's actual content — the words a reader would read — only appears after client-side JavaScript runs, a crawler sees whatever's in the raw HTML response and nothing more.

A real test: what's actually in the raw HTML

Rather than describe this abstractly, here's what a real raw-HTML fetch returns for two different kinds of page, run live for this post:

A server-rendered page. Fetching contextowl.com's own homepage with no JavaScript execution — a plain HTTP request, exactly what a crawler does — returns a complete, readable page in the raw HTML: the title tag reads "Free AI visibility score for your brand — ContextOwl," and the extracted body text includes full sentences like "See what AI engines are saying about your brand, find gaps, get fix recommendations & outrank your competitors" and "Does ChatGPT recommend you or your competitor?" — all present and readable with zero JavaScript executed. This is what server-side rendering (used by most modern frameworks, including Next.js) looks like to a crawler: the same text a person sees in their browser is already sitting in the HTML response.

Evidence: Plain HTTP fetch (curl, no JavaScript execution, no browser), target: contextowl.com homepage, 10 October 2026. Result: full page text present in the raw HTML response — title, headline, and body copy all readable without running any script.

A client-side-only page, by contrast, typically returns something close to this in its raw HTML, with the actual content assembled only after a script runs in a browser:

<!DOCTYPE html>
<html>
  <head><title>App</title></head>
  <body>
    <div id="root"></div>
    <script src="/bundle.js"></script>
  </body>
</html>

A crawler fetching that page gets an empty <div> and a script tag — no headline, no product description, nothing to index or cite, regardless of what robots.txt allows.

How to check your own site

1. Fetch your own page's raw HTML with a plain request that doesn't execute JavaScript (a command-line fetch, or "view page source" in a browser, which shows the server's raw response, not the rendered DOM).
2. Search that raw output for a sentence or phrase you know is on the page.
3. If it's there, your core content is crawler-visible regardless of JavaScript. If it's missing — if all you find is an empty container and script tags — your content depends on client-side rendering, and most AI crawlers won't see it.
4. The fix, if you find this gap, is server-side rendering or static generation for at least your primary content — a framework-level decision, not a quick patch.

Can a CDN or firewall block bots even with a clean robots.txt?

Yes, and this is the failure mode a clean robots.txt file gives no visibility into at all. Content delivery networks and web application firewalls increasingly ship purpose-built "block AI bots" or "block AI scrapers" toggles, separate from and enforced before robots.txt is ever consulted. These settings typically work by matching known AI-bot user-agents or IP ranges and returning a block page or a non-200 status — invisibly, from the bot's point of view, and with no connection to what your robots.txt file says is allowed. Cloudflare's version of this, for example, sits under Security settings as a "Block AI Bots" toggle — check whichever CDN or WAF you actually use for the equivalent setting, since the name and location vary by vendor.

Why this matters more than it looks: a site owner who only checks robots.txt can conclude, wrongly, that every AI crawler is welcome, while a CDN setting enabled by a previous team member, a default security preset, or a vendor's own "AI protection" feature silently blocks the exact bots the robots.txt file was written to allow.

A related caution from Anthropic applies here too: the company explicitly warns that IP-based blocking "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." A firewall rule that blocks a bot's IP range before it can even request robots.txt creates exactly this kind of unintended, unverifiable block.

How do you confirm any of this in your logs?

Server or CDN access logs are the only source of truth for what actually happened, as opposed to what your configuration intends:

1. Filter logs by the bot's user-agent string (OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, and the rest from the robots.txt crawler guide).
2. Check the HTTP status code returned, not just whether a log line exists. A 403 or a redirect to a CAPTCHA page means the bot was seen and blocked, not successfully served.
3. Cross-reference the source IP against the bot's published range (each company publishes one — OpenAI's openai.com/searchbot.json, Anthropic's claude.com/crawling/bots.json, Perplexity's perplexity.com/perplexitybot.json) to confirm a logged request claiming to be a given bot is actually from that company's published range, not a spoofed user-agent.
4. Look for the absence of a bot entirely, not just blocked requests — if a bot that should be crawling regularly hasn't appeared in logs at all, that's its own signal, pointing back to a network-level block upstream of your server.

What's a safe order to check and fix these in?

1. Robots.txt first — confirm the bots you want are explicitly allowed (see the full crawler guide). Cheapest to check, cheapest to fix.
2. Raw HTML content next — fetch your own key pages without executing JavaScript and confirm your actual content is present. If it isn't, this is a rendering-architecture fix, not a quick setting change.
3. CDN/WAF bot-management settings third — audit every "AI bot protection" or "scraper blocking" toggle across your CDN, firewall and any bot-management vendor, since these operate independently of and can silently override robots.txt.
4. Logs last, to confirm — once the first three are addressed, check real server logs to confirm the bots you expect to see are actually succeeding, not just configured to be allowed.

Checklist

• Robots.txt explicitly allows the search/citation bots you want (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot).
• Your page's core content confirmed present in the raw HTML response, independent of JavaScript execution.
• Every CDN, firewall and bot-management "AI bot" toggle audited and set deliberately, not left on a vendor default.
• No IP-range blocking used as a robots.txt substitute.
• Server logs checked for actual bot requests and their status codes, not just configuration review.

Tools to use

• The AI Bot Access Checker — a free check of whether named AI bots can currently reach your site.
• A command-line HTTP client (curl) or a browser's "view page source" to see raw HTML without JavaScript execution.
• Your CDN or hosting provider's access logs, filtered by user-agent and status code.
• Your CDN/WAF dashboard's security or bot-management settings page.

Common mistakes

• Checking only robots.txt and assuming that settles the question of crawler access.
• Assuming a modern-looking, fast site must be server-rendered — speed and rendering method are unrelated; a fast client-side app can still be invisible to a non-JavaScript crawler.
• Relying on IP blocking instead of robots.txt, against Anthropic's own stated caution that doing so can be unreliable and can interfere with the bot reading robots.txt at all.
• Leaving a CDN's default "AI bot protection" setting untouched without checking what it actually blocks.

FAQ

If robots.txt allows a bot, is my site definitely crawlable by it? Not necessarily. A CDN or firewall setting, or a JavaScript-only rendering architecture, can each block a bot that robots.txt explicitly allows. Check all three, not just robots.txt.

How do I know if my site needs JavaScript to show its content? Fetch the raw HTML with no script execution (a command-line request or "view page source") and search for text you know is on the page. If it's missing, your content is JavaScript-dependent.

Should I block AI bots by IP address to be safe? No. Anthropic specifically warns this "may not work correctly or persistently guarantee an opt-out," and it can interfere with a bot's ability to read your robots.txt file in the first place. Use robots.txt as the primary control.

Can a CDN block AI crawlers even if I never configured it to? Yes. Some CDN and WAF products ship AI-bot-blocking features enabled by default or as part of a broader "bot protection" preset — check your dashboard directly rather than assuming nothing is blocking by default.

What to do next

1. Fetch your own homepage's raw HTML (no JavaScript) and confirm your real content is present.
2. Audit every CDN, firewall and bot-management setting for an "AI bot" or "scraper" toggle, and set it deliberately — and don't rely on blocking GPTBot alone, which doesn't control the same thing as OAI-SearchBot.
3. Run the AI Bot Access Checker and cross-check the result against your own server logs, with the full bot-by-bot robots.txt guide as reference.

ContextOwl sells one-time AI visibility reports. The AI Bot Access Checker is a free check of whether named AI bots can currently reach your site.