How to optimize your website for AI search

This is Chapter 9 of The SEO Playbook for 2027, published here in full as a free sample. Research cut-off: 8 October 2026. Every claim carries a date and a link to its source.

How evidence is labelled. [Official] is a platform’s own documentation or blog. [Googler] is something a Google employee said on a podcast, at an event or online. [Study] is an industry or academic study with published data. [Practitioner] is the analysis or opinion of a named SEO practitioner. [News] is a trade publication’s report. [Unverified] marks a claim we could not confirm from a primary source. [Our reasoning] marks a recommendation that follows from the evidence but that no study tests directly. Boxes headed What Google says appear only where Google’s stated position contradicts the experts.

People now get answers from Google’s AI Overviews and AI Mode, from ChatGPT, Perplexity, Microsoft Copilot, Gemini and Claude, and from assistants built into phones and browsers. Each of them reads the web, picks a handful of sources, and sometimes links to them. Being one of those sources has become a goal in its own right, and a small industry has grown up around it under names like GEO (generative engine optimisation) and AEO (answer engine optimisation). This chapter separates what the evidence supports from what is being sold.

What the experts say

The consensus: AI visibility is built on search visibility. Most practitioners who measured AI answers in 2026 reached the same conclusion: what gets a page cited is mostly what gets it ranked. It has to be crawlable and indexed, and it has to be the clearest available answer to a specific question.

• Lily Ray argued in March 2026 that much of what is sold as GEO is SEO under a new name, and that Google’s index is the main retrieval layer for most AI search, ChatGPT included. Her summary: “GEO Gets the Credit. SEO Did the Work.” (Substack, 2026-03-18) [Practitioner]. In January she wrote that AEO and GEO are “not an overhaul or abandonment of SEO” (Substack, 2026-01-19).
• Kris Jones of LSEO put the share of AI search optimisation that is fundamental SEO at 70–80% in a Search Engine Journal interview (SEJ, 2026-08-03) [Practitioner].
• Kevin Indig described AEO and GEO as “a brand channel disguised as a performance channel” (Growth Memo, 2026-07-27) [Practitioner]. His point is that AI answers shape which brands people consider far more than they send measurable visits.
• Tom Capper of Moz made the same point from the data he has seen: for those sites, traffic from Gemini and ChatGPT peaked below 2%. He suggested judging this work by mentions or impressions, not traffic (Moz Whiteboard Friday, 2026-07-24) [Practitioner].

Disagreement: is this a separate discipline? Mike King of iPullRank is the most prominent voice saying yes. AI systems retrieve passages, not whole pages, so he argues content has to be engineered passage by passage and tested with the same vector methods the systems use. He published a point-by-point rebuttal of Danny Sullivan’s remarks on “chunking” in January 2026, writing that “Chunking and creating content for users are not mutually exclusive” (iPullRank, 2026-01-15). In May he called Google’s new AI guidance “naive and self-serving” (iPullRank, 2026-05-18) [Practitioner]. Conductor’s report on AI search benchmarks treats it as a new channel in its own right (Conductor, 2025-11-12, updated 2026-07-06). Microsoft uses the term too: Bing describes its AI Performance report as an early step towards “Generative Engine Optimization (GEO) tooling” (Bing, 2026-02-10) [Official].

Disagreement: should content be “chunked”? King, a 2025 Semrush study that recommended presenting information in a “chunkable” way (Semrush, 2025-07-21) and Microsoft’s own advertising blog all favour structuring pages into short, self-contained sections. Microsoft says Copilot parses pages into pieces, and that long walls of text “make it harder for AI to separate content into usable chunks” (Microsoft Advertising, 2025-10-08) [Official]. Lily Ray counters that bullet points, summaries and FAQ formats are SEO basics with a new label (2026-03-18).

The two positions are closer than the argument suggests. Clear headings, a direct answer under each, and sections that make sense on their own are good writing whichever side is right. What Google is warning against is rewriting pages into fragments for machines, or keeping a second machine-only version [Our reasoning]. King himself goes further than most and has proposed serving Markdown versions to AI crawlers, explicitly not to Google (iPullRank, 2026-05-14); few practitioners follow him there.

Disagreement: how to earn mentions. Everyone agrees that being mentioned across the web matters (see “What gets cited” below). They disagree about how far to go to get mentioned.

• Charlie Marchant of Exposure Ninja suggested finding the pages AI systems cite for your target prompts and pitching their authors for a mention (Moz Whiteboard Friday, 2026-07-17) [Practitioner].
• Capper included sponsored publisher partnerships in his playbook (2026-07-24).
• Jones advised pursuing industry awards, even pay-to-play ones, while calling it spam to rank yourself first in your own list (2026-08-03).
• Lily Ray documented steep visibility drops from late January 2026 at sites that relied on self-promotional “best” lists, a tactic she called “the most popular ‘GEO’ tactic” (Substack, 2026-02-03). In October she predicted a crackdown on undisclosed paid mentions and placements (Substack, 2026-10-04) [Practitioner].

Disagreement: can AI visibility be measured? Vendors sell AI-visibility scores, and practitioners doubt them. King reported two tools giving the same client 41% and 23% visibility, and argued that in AI search “accuracy” is a category error: the sampling method is the measurement (iPullRank, 2026-09-17) [Practitioner]. Indig suggested treating prompt tracking like opinion polling, with samples and margins of error, rather than like rank tracking (2026-07-27).

AI search runs on search indexes

The most useful fact in this chapter is also the least exciting: almost every AI answer engine finds its sources through a search index, and for most of them that index belongs to Google or Bing.

Sources, all [Official] unless marked: Google AI optimisation guide, 2026-05-15; Google common crawlers, updated 2026-07-14; OpenAI crawlers, undated; Perplexity crawlers, modified 2026-01-29; Anthropic crawlers, updated 2026-04-07; About Applebot, 2026-09-04; DuckAssistBot, undated; the Bing–ChatGPT overlap is from Seer Interactive, 2025-02-06 [Study, small sample]. In March 2025 Claude’s web search was reported to use Brave Search (Simon Willison, 2025-03-21); we could not confirm the provider in 2026 [Unverified].

Two consequences follow.

First, being properly indexed by both Google and Bing is most of the work. If Googlebot and Bingbot can crawl and index your pages, you are eligible for AI Overviews, AI Mode, Gemini, Copilot, DuckDuckGo’s answers and, through the Bing overlap, probably much of ChatGPT search. Microsoft said in February 2026 that Bing’s grounding powers nearly every major AI assistant (Bing Search blog, Jordi Ribas, 2026-02-12) [Official]. Getting properly crawled and indexed is therefore the foundation of AI visibility.

Second, blocking the wrong crawler removes you from an answer engine. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links; Anthropic says blocking Claude-SearchBot may reduce a site’s visibility in Claude’s search results (OpenAI; Anthropic) [Official]. Training crawlers are different: blocking GPTBot, ClaudeBot, Google-Extended or Applebot-Extended is a business decision about model training, and the companies say it does not affect search. Decide the two questions separately.

Google’s own controls

Google gives you two switches, and they do different things [Official]:

• Google-Extended (a robots.txt token) governs whether your content is used to train and ground Gemini models. Google says it does not affect inclusion in Google Search or AI Overviews (Google common crawlers).
• The Search generative AI control in Search Console, rolled out to all sites on 2026-08-31, lets you exclude your site’s links and content from AI Overviews, AI Mode and generative AI features in Discover. The default is to include. Google says the control is not used as a ranking or inclusion signal elsewhere in Search, and that excluding your site means you receive no traffic or impressions from those features (Search generative AI control).

For most sites that want search traffic, the right setting is the default. If you change it, measure what happens.

What gets cited

The studies of AI citations are young, mostly correlational, and often run by companies that sell AI-visibility tools. Read them as signposts, not laws. With that caveat, here is what the better ones found [Study]:

• Ranking still helps, but less than it did. Louise Linehan at Ahrefs found in July 2025 that 76.1% of URLs cited in AI Overviews also ranked in Google’s top 10 for the query (1.9 million citations from a million AI Overviews). A re-run published on 2026-03-02 and updated on 2026-05-31 found 37.9% in the top 10, 31.2% in positions 11–100 and 31.0% beyond position 100 (Ahrefs, 2025-07-21; Ahrefs, 2026-03-02). Ahrefs puts the change down to query fan-out: the Overview draws on results for related sub-questions, not only the query typed. A page that precisely answers one sub-question can be cited without ranking for the head term.
• Brand mentions correlate with AI visibility; site size does not. In an Ahrefs study of 75,000 brands, branded web mentions correlated with AI visibility at about 0.66–0.71 and YouTube mentions at about 0.74, while the number of pages on a site correlated at only about 0.19 (Ahrefs, 2025-12-12, updated 2026-08-12). Correlation is not cause: well-known brands get mentioned and cited for the same underlying reason.
• Titles that match the question. In an Ahrefs analysis of 1.4 million ChatGPT prompts, cited pages had titles that were more semantically similar to the underlying search queries than pages that were retrieved but not cited (a similarity score of 0.602 against 0.484) (Ahrefs, 2026-04-15).
• Fresher pages, on some engines. Ahrefs found that pages cited by AI assistants were on average about 26% “fresher” than pages in organic results, with ChatGPT favouring newer pages most and AI Overviews the least (Ahrefs, 2025-07-28, updated 2026-04-27). That is not a reason to fake dates, which Google’s guidance names as a practice to avoid; it is a reason to keep important pages genuinely current.
• Statistics, quotations and cited sources. The academic paper that coined “GEO” found that adding citations, quotations and statistics to a page improved its visibility in generative answers by up to 40% in its test set (Aggarwal et al., KDD 2024). The experiment used its own test engine, so treat the size of the effect with caution.
• The studies disagree with each other. Semrush reported that ChatGPT cited pages ranking beyond position 20 almost 90% of the time (Semrush, 2025-07-21), while Seer found that 87% of ChatGPT search citations matched Bing’s top 20 results for the same question (Seer, 2025-02-06). The methods differ, and so do the engines from month to month. Optimising for one engine’s current quirks is a poor bet.

What to do on the page

Everything below is ordinary good practice for search, which is the point (sources as cited):

1. Use normal HTML pages. John Mueller said in June 2026 that having normal HTML pages is basically the main thing for AI systems (Search Off the Record, 2026-06-15) [Googler]. Do not create separate Markdown or “LLM-only” versions of pages; Danny Sullivan said in January 2026 that Google doesn’t want two versions of content (Search Off the Record, 2026-01-08) [Googler].
2. Put the answer and the numbers first. A short opening that answers the question, followed by a table and then the method. Microsoft’s guidance for inclusion in AI answers recommends titles, headings and descriptions that agree with each other, question-and-answer pairs, lists and tables (Microsoft Advertising, 2025-10-08) [Official].
3. Write headings that match how people ask. Question-shaped H2s where the section answers a question; a title that states what the page answers.
4. Use readable URL slugs. /project-management-software-prices/ rather than /p?id=4821. The Ahrefs title finding above suggests the words that describe the page matter to how it is matched [Our reasoning].
5. Show your sources and your statistics. Name where each figure comes from and when it was checked. Original numbers are the most citable thing you can publish.
6. Keep the content in the HTML. A 2024 analysis of crawler traffic by Vercel and MERJ found that “none of the major AI crawlers currently render JavaScript”, apart from those run by Google and Apple (Vercel, 2024-12-17) [Study]. Answers that appear only after scripts run may never be seen.
7. Cover each topic on one strong page. Bing says it clusters near-duplicate pages and lets one represent them in AI answers (Bing, 2025-12-19) [Official].

What is not needed, according to the evidence

Three things are widely sold as AI optimisation and have little or no evidence behind them for search.

llms.txt. An llms.txt file is a Markdown summary of a site, intended as a guide for language models. Adoption is growing: Semrush counted 951 domains with one in July 2025 and 16,670 in September 2026 (Semrush, updated 2026-09-28) [Study]. Use is not. Ahrefs analysed 137,210 domains in May 2026 and found that 97% of the llms.txt files it saw received no requests at all in its data; it called the file “largely decoration” for anyone hoping to appear in ChatGPT, Perplexity or AI Overviews (Ahrefs, 2026-06-15) [Study]. Semrush’s Tushar Pol admits there is no proof it improves visibility but frames early adoption as a low-cost edge; Mike King says it is useful for Claude (iPullRank, 2026-05-18), a claim we could not confirm in Anthropic’s crawler documentation [Unverified]. Marie Haynes draws the useful distinction: the file is for AI agents using your site, not for search (mariehaynes.com, 2026-05-29) [Practitioner]. Chrome’s experimental Lighthouse audit for agent readiness does check for it (same source).

Schema markup as an AI lever. Ahrefs tracked 1,885 pages that added JSON-LD between August 2025 and March 2026 against 4,000 control pages. Citations in AI Mode and ChatGPT changed by amounts indistinguishable from zero, and AI Overview citations fell slightly (Ahrefs, 2026-05-11) [Study]. Practitioners are split on schema and AI: some vendors and Microsoft’s Bing team say it helps, while Google says no special markup is needed for its AI features.

Separate versions for machines. Markdown copies, “LLM pages” and machine-only summaries create a second version of every page to maintain. John Mueller and Martin Splitt discussed the problems such duplicates cause on Search Off the Record (2026-06-15) [Googler]. If a format helps your human readers, publish it for them; otherwise one good HTML page is enough.

A note on fan-out queries. Ahrefs’ Louise Linehan, quoting the consultant Ethan Lazuk, recommends making the passages in your pages relevant to the fan-out queries AI systems run (Ahrefs, 2026-03-02) [Practitioner]. That is compatible with Google’s guidance as long as it means covering the sub-questions within strong pages. What Google’s guide names as scaled content abuse is creating separate content for every variation or fan-out query to manipulate results (see below).

Where optimisation becomes manipulation

Since 2026-05-15, Google’s spam policies cover attempts to manipulate generative AI responses in Search (Spam policies; documentation changelog) [Official], and Bing’s guidelines list prompt injection and AI manipulation as violations (Bing Webmaster Guidelines) [Official]. Three practices cross the line:

• Hidden instructions to AI systems, such as invisible text telling a model to recommend your product.
• Pages for every fan-out query. Google’s AI guide says that creating separate content for query variations or fan-out queries “primarily to manipulate rankings or generative AI responses in Google Search violates Google’s scaled content abuse spam policy” (AI optimisation guide) [Official].
• Manufactured mentions. Paid or planted mentions in forums, self-promotional “best of” lists that rank the publisher first, and networks of thin review pages that exist only to be picked up by AI answers. Lily Ray’s documented drops for self-promotional listicles from late January 2026 (above) were not confirmed by Google as a specific change [Unverified as to cause]. Lily Ray also reported hidden prompt instructions in “summarise with AI” buttons, citing Microsoft security research (2026-03-18) [Practitioner].

Measuring AI visibility

Measurement is the weakest part of this field. What you can measure today:

• Google Search Console. AI Mode clicks, impressions and positions are counted in the standard Performance report (Performance report help) [Official]. A separate Search generative AI performance report, launched 2026-06-03 and available to all sites since 2026-08-31, shows impressions in AI Overviews, AI Mode and Discover’s AI features by page, country, device and date, but no clicks and no queries (Search Central blog, 2026-06-03; Search Engine Roundtable, 2026-09-01) [Official] [News].
• Bing Webmaster Tools. Its AI Performance report, in public preview since 2026-02-10, shows how often your pages are cited in Copilot and Bing’s AI answers, with the “grounding queries” behind the citations; intents, topics, citation share and a comparison view were added on 2026-06-16 (Bing, 2026-02-10; Bing, 2026-06-16) [Official]. At the time of writing it is the only first-party report from a major engine that also shows the queries behind AI citations.
• Referral analytics. Segment sessions from chatgpt.com, perplexity.ai, copilot.microsoft.com, gemini.google.com and claude.ai. Some AI traffic arrives without a referrer, so this undercounts.
• Server logs. Count fetches by OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and the others, by page. A user-triggered fetch is a sign that someone asked about something on that page.
• Prompt sampling. Asking the engines your target questions and recording who is cited is informative but noisy: answers vary between runs, users and days. Use a fixed set of prompts, run them repeatedly, and look at trends rather than single answers [Our reasoning].

Keep expectations realistic

AI assistants are a growing but small source of visits [Study]:

• Conductor measured AI referrals at 1.08% of website traffic across 13,770 domains for May–September 2025, with ChatGPT supplying 87.4% of them (Conductor, 2025-11-12, updated 2026-07-06).
• BrightEdge reported that ChatGPT referral traffic roughly doubled from January to August 2026 (+101%) and made up 95.1% of AI referrals in its data (BrightEdge, 2026-09-24).
• Patrick Stox at Ahrefs estimated in February 2026, from traffic data on 76,000 sites, that Google sends about 190 times more traffic to websites than ChatGPT does (Ahrefs, 2026-02-11).
• On the other side, a Similarweb study described by Rand Fishkin found that being recommended by an AI assistant was followed by more direct visits to the recommended brand within a week, visits that would not show up as AI referrals in analytics (SparkToro, 2026-06-28) [Study, second-hand].

For most sites in 2026, AI answer engines are a secondary channel that comes with being well indexed, and a reason to make pages more citable. They are not yet a replacement for search traffic.

Follow these steps

1. Allow the search crawlers of every answer engine you want to appear in. At minimum Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot and DuckAssistBot.

   Done when: your robots.txt, fetched from outside your network, allows each of them to crawl your content, and your CDN or firewall does not challenge them.

2. Decide on training crawlers as a separate question. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and similar tokens.

   Done when: the decision for each is written down with its reason, and robots.txt and your CDN settings match it.

3. Get fully set up in Bing. Verify the site in Bing Webmaster Tools, submit your sitemap, and send IndexNow pings for URLs as they change.

   Done when: Bing reports your sitemap as processed and its index count for your site is within the range you expect.

4. Leave Google’s Search generative AI control on “Include” unless you have decided otherwise.

   Done when: you have checked the setting in Search Console and recorded its value and the date.

5. Rewrite the opening of your ten most important pages to answer first. The question, the answer and the key number in the first lines; a table below; the method after.

   Done when: for each of the ten pages, someone who reads only the first three sentences can state the answer.

6. Check titles, headings and slugs against how people ask. Titles that state what the page answers; question-shaped headings where a section answers a question; readable slugs.

   Done when: every indexable template produces a descriptive title and a readable slug, and the ten key pages have headings that mirror real questions.

7. Source every figure. Name the source and the date next to each important number.

   Done when: a spot check of the ten key pages finds no figure without a visible source or date.

8. Remove anything that could look like AI manipulation. Hidden prompts, pages created for each query variation, and self-promotional lists that rank you first.

   Done when: a review of every template finds none of these.

9. Set up AI measurement. Search Console’s generative AI report, Bing’s AI Performance report, a referral segment for AI assistants in analytics, and AI crawler counts from server logs.

   Done when: a monthly report pulls all four, and the first month’s baseline is recorded.

10. Run a fixed prompt set monthly. Twenty to fifty questions your pages answer, asked in the main engines, recording whether and how you are cited.

    Done when: the prompt set is saved, the first run is recorded with the date and engine, and a repeat date is scheduled.

This chapter is one of twelve in The SEO Playbook for 2027. The others follow the same pattern: what experienced practitioners are saying, with names, dates and links; a What Google says box wherever Google’s position contradicts theirs; and numbered steps, each with a “Done when:” check. They cover how search changed in 2025–2026 and what is announced for 2027, getting a new site crawled and indexed, what “helpful” now means for comparison, affiliate and publisher sites, trust signals for small and anonymous publishers, programmatic pages, internal linking, structured data, technical essentials, links without outreach, Google’s spam policies, and a 13-week plan that puts all the steps in order.

Like this chapter, the book does not promise results. It sets out what the evidence and the experts point to, and how to measure what happens on your own site.