Technical AEO

Why your website is invisible to AI search, and 7 fixes

Last updated

8 min read

A website that never shows up in AI answers usually has a specific, fixable cause. These are the seven I check, in the order I check them, with the official source for each.

Short answer

A website is usually invisible to AI search for one of seven reasons: AI crawlers are blocked by robots.txt or a firewall, key content only appears after JavaScript runs, pages aren't indexed by Google or Bing, snippet controls opt pages out, pages don't answer questions directly, the brand isn't a clear entity, or no one else mentions the brand. Each cause has a test and a fix.

Key facts

  • To appear as a link in AI Overviews or AI Mode, a page must be "indexed and eligible to be shown in Google Search with a snippet" (Google).
  • Since September 15, 2026, Cloudflare's Block setting for training crawlers also blocks Googlebot, Bingbot and Applebot (Cloudflare).
  • In a December 2024 analysis, none of the major AI crawlers from OpenAI, Anthropic or Perplexity rendered JavaScript (Vercel).
  • Google's nosnippet rule also stops content being used as "a direct input for AI Overviews and AI Mode" (Google).

Are AI crawlers blocked by robots.txt or your firewall?

AI search tools can't cite pages their crawlers can't fetch, and the block can sit in robots.txt or the CDN rather than on the page. A crawler is the program a company uses to fetch web pages.

robots.txt

Open yoursite.com/robots.txt and look for Disallow: / under the AI search crawlers: OAI-SearchBot (ChatGPT), Claude-SearchBot (Claude), PerplexityBot (Perplexity), Googlebot and Bingbot. As of November 2026, OpenAI says sites that opt out of OAI-SearchBot "will not be shown in ChatGPT search answers", and Anthropic says disabling Claude-SearchBot "may reduce your site's visibility and accuracy in user search results".

Check the status code too. Under RFC 9309, if robots.txt is unreachable because of a server error, crawlers "MUST assume complete disallow", so a robots.txt that returns a 5xx error can block the whole site for crawlers that follow the standard. robots.txt for AI crawlers lists every bot and its token.

CDN and firewall

A CDN (content delivery network) is a service such as Cloudflare that sits in front of your server, and its bot protection can stop crawlers that robots.txt allows. In November 2026, Google wrote that when a CDN or firewall challenges automated requests with something like a CAPTCHA, "Google's crawlers fail the test", and that if the block stays, pages "will begin to drop out of Search".

Cloudflare's AI crawler settings have changed several times since mid-2025, and two changes matter most. In July 2025, it started asking every new domain at sign-up whether to allow AI crawlers. On September 15, 2026, its Block and "Block on pages with ads" settings started applying to mixed-use crawlers including Applebot, Bingbot and Googlebot, so either setting now affects search as well as training.

To fix it in Cloudflare, open Security, then Settings, then AI Crawler and Agent access policies, and set Search and Agent to Allow. To opt out of training while staying in search, choose Disallow AI Training instead of Block. Then check Bot Fight Mode: Perplexity notes it "can't be skipped with WAF custom rules" and can still challenge its bots.

Does your content only appear after JavaScript runs?

Content that appears only after JavaScript runs may be invisible to most AI crawlers. In a December 2024 analysis of traffic on Vercel's network, Vercel and MERJ found that "none of the major AI crawlers currently render JavaScript", including OpenAI's OAI-SearchBot, ChatGPT-User and GPTBot, Anthropic's ClaudeBot and PerplexityBot. Google's Gemini and Applebot did render pages.

Places to check on stores and course sites: prices, stock and reviews loaded by a client-side widget, a syllabus that loads only when a tab is clicked, and FAQ answers injected by a script. Microsoft's guidance adds: "Don't hide important answers in tabs or expandable menus: AI systems may not render hidden content".

To test, fetch the raw HTML and search it for a fact that should be there:

curl -s https://www.example.com/products/spf-50-sunscreen | grep -i "549"

If the price isn't in the output, a crawler that doesn't run JavaScript can't see it. Render main content on the server, with server-side rendering or static generation, and keep client-side code for extras. The Vercel data is from December 2024, so retest your own pages.

Are your pages indexed where AI search looks?

Google's and Microsoft's AI answers draw on their search indexes, so a page that isn't indexed there can't be cited. Google says a page "must be indexed and eligible to be shown in Google Search with a snippet" to appear as a supporting link in AI Overviews or AI Mode. Check important pages with the URL Inspection tool in Search Console.

Bing matters too. Microsoft says experiences like Microsoft Copilot are "powered by Bing's search index" (Microsoft). Verify your site in Bing Webmaster Tools, submit your sitemap, and turn on IndexNow, a protocol that pings search engines when a URL changes.

IndexNow is supported by Microsoft Bing, Naver, Seznam.cz, Yandex and Yep, and Bing says it "helps keep information fresh across search and AI experiences".

Have you opted out of AI answers by accident?

Snippet controls and opt-out settings can remove pages from AI answers while leaving them in regular results. Check these four:

  • Google nosnippet and max-snippet. Google says nosnippet also prevents content being used as "a direct input for AI Overviews and AI Mode", and max-snippet limits how much can be used (Google).
  • Search Console's Search generative AI control. Since August 31, 2026, every site can exclude itself from AI Overviews, AI Mode and generative AI features in Discover. Include is the default; check Settings, then Search generative AI (Google).
  • Bing data-nosnippet and NOARCHIVE. Bing excludes data-nosnippet content from "snippets and AI summaries" (Bing), and said in 2023 that NOARCHIVE content would not be included in Bing Chat answers (Bing).
  • Apple nosnippet. Apple says it won't use content tagged nosnippet as context when its AI models generate output (Apple).

To test, run curl -sI on a URL and look for an X-Robots-Tag header, then search the page source for nosnippet, max-snippet and noarchive. Test a product page, a course page and a blog post, because templates differ.

Do your pages answer the question directly?

A page that never states the answer plainly gives AI systems nothing clean to quote. Bing's February 2026 guidance says "clear headings, tables, and FAQ sections help surface key information and make content easier for AI systems to reference accurately."

Compare these openings:

  • Store, before: "Our bestselling sunscreen, loved by thousands."
  • Store, after: "This sunscreen is SPF 50 and PA++++, comes in 50 ml for ₹549, and is made for oily, acne-prone skin."
  • Course, before: "Transform your career with our flagship program."
  • Course, after: "This 12-week, part-time SQL course costs ₹18,000, runs on weekends and ends with three graded projects."

Specifics have research behind them. A KDD 2024 study found that adding statistics, quotations and citations raised a source's visibility in generative engine answers by 30 to 40 percent on one measure. Google adds that you don't need to "chunk" pages or write in a special way for AI (Google), so write for buyers first.

Is your brand a clear entity?

An entity is a distinct thing, such as a company, that search systems can identify and describe. Microsoft's guidance tells publishers to "reduce ambiguity" so that text, images and video "consistently represent the same entities, products, or concepts" (Bing).

Use one name and one spelling everywhere, and keep your Merchant Center and Business Profile details current, which Google lists among its best practices for AI features. Add Organization structured data to your home page; Google says it "can help Google better understand your organization's administrative details and disambiguate your organization in search results" (Google).

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Example Skin Co",
  "url": "https://www.example.com/",
  "logo": "https://www.example.com/logo.png",
  "sameAs": ["https://www.instagram.com/exampleskinco", "https://www.linkedin.com/company/exampleskinco"]
}

Google also says structured data "isn't required for generative AI search", so treat it as clarity, not a trick.

Does anyone else mention your brand?

AI search engines lean on third-party sources, so a brand that only describes itself is easy to skip. A September 2025 study comparing AI search with Google found "a systematic and overwhelming bias towards Earned media", and Google says its AI features "can show what's being said about products and services across the web, including in blogs, videos, and forum discussions" (Google).

Run your buyer questions, list the domains that get cited, and earn real coverage there: reviews, "best of" lists, community answers and press. Google warns that chasing inauthentic mentions "isn't as helpful as it might seem". How to get your brand mentioned in ChatGPT answers covers the process.

I check all seven for small and mid-size stores and course businesses. I'm a quant developer, and I implement the fixes rather than handing over a report.

Find out how AI engines see your brand

Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.

Sources

Frequently asked questions

Does Cloudflare block AI crawlers by default?

Not on most new domains as of November 2026. Cloudflare's docs say new domains allow Search, Agent and Training crawlers by default, while sites that monetize with ads are offered stricter presets. Domains added after July 2025 were asked about AI crawlers at sign-up, so check your current settings.

Do AI crawlers run JavaScript?

Most major ones did not in a December 2024 analysis by Vercel and MERJ, which found no JavaScript rendering by OpenAI, Anthropic, Meta, ByteDance or Perplexity crawlers. Google's Gemini and Applebot did render pages. Put important content in the initial HTML to be safe.

Will nosnippet remove my page from Google AI Overviews?

Google says nosnippet prevents a page's content from being used as a direct input for AI Overviews and AI Mode, and it also removes the text snippet from regular results. To exclude a whole site instead, Google offers the Search generative AI control in Search Console.

Do I need an llms.txt file to appear in AI search?

Not for Google. Google says Google Search ignores llms.txt files, so creating one neither helps nor harms visibility there. Other services may use these files, which is why this site still publishes one.

Written by

TutorTechWiz AEO

An independent AEO and SEO practice run by a quant developer who audits how AI engines see a brand and then builds the fixes. About the consultant.

Find out how AI engines see your brand

Start with an AEO audit: a fixed-price report on how ChatGPT, Google’s AI features, Gemini and Perplexity see your brand today, with a prioritized fix list. I can implement the fixes too.