Answer Engine Optimization: What Actually Decides Whether You Get Quoted

What is answer engine optimization?

Answer engine optimization is the work of making a page's content easy for an AI assistant to find, extract and attribute. It differs from ordinary SEO in one specific way: a search engine ranks whole pages, while an answer engine lifts a passage out of a page and reproduces it inside its own answer. So the unit that matters stops being the page and becomes the paragraph that can stand on its own.

Everything else follows from that. A sentence beginning "As we saw above, it does this by…" cannot be quoted, because the thing it points at will not be in the extract. A page whose answer only appears after JavaScript runs may never be read at all. A claim attributed to "we" gives a model nobody to cite.

This guide is published by SiteBuilderStack and was written on 25 September 2026. Every crawler behaviour below is quoted from the vendor's own documentation, with the link next to the claim, because the field is full of confident assertions that were true for about a month.

Which crawlers actually matter, and what each one does

Four vendors, ten user agents, and they do not all do the same job. Blocking the wrong one costs you citations; blocking the right one may be exactly what you want.

Sources, each stating the behaviour quoted above: OpenAI's bots documentation, Anthropic's crawler support article, Perplexity's bot guide and Google's common crawlers reference, all read on 25 September 2026.

The split worth noticing

Look at the last column. The three vendors handle user-initiated fetching differently, and the difference is documented rather than inferred.

OpenAI says of ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply." Perplexity is blunter about Perplexity-User: "this fetcher generally ignores robots.txt rules." Anthropic states that all three of its agents, Claude-User included, honour robots.txt.

The practical consequence: a robots.txt disallow is not a reliable way to stop your content appearing in an AI answer. It stops the systematic crawlers. It does not necessarily stop a fetch that a person triggered by asking a question. If your requirement is genuinely "this must not be readable by an assistant", robots.txt is the wrong control and you need authentication.

Should you allow AI crawlers?

It is a real decision with a real trade-off, and there is no answer that is correct for everyone.

Allow them if you want to be cited. A blocked crawler cannot quote you, and citation is how an answer engine sends a reader to your site at all.

Block the training agents, allow the search ones if you want the traffic without contributing to model training. The vendors have deliberately split these into separate tokens so that this is expressible: block GPTBot and ClaudeBot, allow OAI-SearchBot, Claude-SearchBot and PerplexityBot.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

One thing to check before you write any of that: no directive at all is the same as allowed. A crawler with no matching User-agent group falls through to the User-agent: * group, and on most sites that group permits everything. If your robots.txt mentions none of these agents, they are already allowed, and adding an explicit Allow changes nothing except making the decision visible to the next person who reads the file. That is a decent reason to add it anyway.

What makes a passage quotable

This is the part that actually moves, and it is writing rather than configuration.

Put a complete answer in the opening. Not a preamble about why the topic matters. The first hundred words are what gets read and lifted; treat them as the whole page if you have to.

Make sentences survive extraction. Read your opening and ask of each sentence: pulled out on its own, does it still mean anything? Sentences that fail usually start with a dependent word — this, that, it, they, however, therefore, as mentioned — because those point at something the extract will not contain.

Name the entity. "We rebuilt the workflow" attributes to nobody. "SiteBuilderStack rebuilt the workflow" gives a model something to cite. Use the name in the opening at least once, even where it reads slightly formally.

Be specific enough to be wrong. "Considerably faster" is not quotable; "cut the redirect chain from four hops to one" is. Counts, versions, dates and named things are what make your passage a better answer than the generic one three competitors also wrote.

Shape some headings as questions. Not all of them — a heading becomes a question when a reader would genuinely ask it in those words. But a page whose H2s are all noun phrases gives a model no obvious anchor for a specific query.

Keep the answer in the HTML. If the content only exists after JavaScript executes, assume it may not be read. This is the oldest technical-SEO rule and it applies with more force here, not less.

The Google-Extended misunderstanding

This one is worth stating plainly because it is widely got wrong, including by people selling AEO services.

Google-Extended is not a crawler. Google's documentation describes it as "a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models", and adds that it "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings."

Two consequences, both from the same source:

• Blocking it does not stop Google fetching your pages. Googlebot still crawls; the token only governs what happens to the content afterwards, in Gemini training and grounding.
• Blocking it costs you nothing in Search. Google states that "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

So if your reason for allowing Google-Extended is "I do not want to hurt my rankings", that reason does not hold. Decide it on whether you want your content used for Gemini training and grounding, which is a different question and a legitimate one to answer either way.

Is llms.txt worth publishing?

Probably yes, cheaply, without expecting much on its own.

llms.txt is a Markdown file at the site root that lists and describes a site's important pages for language models. It was proposed by Jeremy Howard in September 2024, with a v2 revision in August 2026. Adoption is real: thousands of sites publish one, documentation platforms generate them automatically, Chrome's Lighthouse audits for one, and OpenAI, Anthropic and Google all publish llms.txt files for their own developer documentation.

What it is not is a ratified standard. The specification is described as "open for community input", and no search or answer engine documents using it as a ranking or citation input. Publish it because it is a genuinely useful index of your own site that costs an afternoon, not because you have been told it wins citations.

The thing that makes one useful rather than decorative is the description under each link. A bare list of URLs tells a model nothing it could not get from a sitemap. One or two sentences per entry saying what that page answers is the whole value.

Does structured data get you cited?

No, and it is worth being clear about why it is still worth having.

Structured data does not cause a citation. What it does is remove ambiguity about the things a model would otherwise have to infer: who wrote this, when, what the headline is, what kind of page it is. Article or BlogPosting markup that matches the visible content is cheap insurance against being described wrongly.

Two failure modes to avoid. Do not add FAQPage markup to a page that is not a list of questions and answers, and do not add aggregateRating or review markup for reviews that do not exist — that is fabricated structured data, and the risk is a manual action rather than a citation. And keep the dates in the markup consistent with the dates a reader can see; a datePublished that disagrees with the page is worse than none.

How to check a page you have already published

Four checks, in this order, each of which you can run in a few minutes.

1. Fetch it as a crawler would. A plain HTTP request, no browser. If the answer is not in the response body, nothing else on this list matters.

   curl -s -A "OAI-SearchBot" https://example.com/your-page | head -c 3000

2. Read your own robots.txt for each agent you care about. Not the whole file — the group that actually applies to that user agent, remembering that no group means the * group applies.
3. Extract your own opening. Copy the first hundred words into a blank document and read them cold. If you cannot tell what the page answers, neither can a model.
4. Test each opening sentence for self-containment. Take them one at a time, out of order. The ones that stop making sense are the ones that cannot be quoted.

If you would rather not do the fourth check by hand, there is a free browser tool that does it: the Answer Engine Optimization Checker. Paste an opening and it reports which sentences survive extraction, shows the passage it would lift, and writes the llms.txt entry. It runs entirely in your browser, sends nothing anywhere and needs no account.

It is worth noting what that tool cannot do, because it is the same point this guide makes. It is hosted on claude.ai, and claude.ai/robots.txt returns Disallow: / for GPTBot, OAI-SearchBot, ChatGPT-User and Google-Extended. So the tool itself can never be cited by an answer engine, however well written it is. Anything you want quoted has to live on a domain you control. That is why this guide is here and not there.

A prompt that does the fourth check, and only that check:

Assess whether this page's opening can be quoted by an answer engine.
Do not rewrite it yet. Report only.

I will paste the first three paragraphs of a page, and the question the page
is meant to answer.

FOR EACH SENTENCE, report:
- Could it be lifted out and still mean something on its own? Yes or no.
- If no, name the exact word or phrase that depends on something outside the
  sentence — a pronoun with no antecedent in the extract, a connective like
  "however" or "therefore", a reference like "as described above".

THEN report:
- The single passage you would extract if you had to answer the question
  using only this text. Quote it exactly. If there is no such passage, say
  so plainly rather than assembling one from fragments.
- Whether the passage names who is making the claim, or attributes it to
  "we" / "our team" / nobody.
- Which words from the question appear in the first hundred words.
- Every specific, checkable detail in the opening: counts, versions, dates,
  named products. List them. If there are none, say so.

DO NOT
- Do not score the page out of ten. There is no scale that means anything.
- Do not tell me this will improve my rankings or get me cited. You cannot
  know that, and no answer engine publishes its extraction rules.
- Do not rewrite the text. I want the diagnosis first.

Finish with the single change that would most improve extractability, and
why.

What nobody can promise you

No answer engine publishes its extraction or citation rules. There is no equivalent of a ranking-factors list, no tool that can verify you will be quoted, and no agency that can guarantee it. Anyone who tells you otherwise is describing a mechanism they cannot see.

What is knowable is narrower and more useful: whether your page can be fetched, whether the answer is in the HTML, whether a passage survives extraction, and whether your claims are specific enough to be worth quoting over someone else's. Those are all checkable today, by you, without a tool. Everything past that is inference.

Treat AEO as ordinary editorial discipline with a particular constraint — write so that one paragraph can be removed and still be true — rather than as a new channel with its own tricks. The pages that get quoted are, so far, mostly the pages that were clearly written.

Download the readiness kit

Two files. A commented robots.txt with all ten AI crawler user agents, each vendor's own description quoted and sourced, and three positions to choose between — allow everything, allow search but block training, or block what you can. Plus the per-page checklist from this article as a tick-list.

Every crawler behaviour in it carries its source and the date it was read. This part of the web moves quickly; check the source rather than trusting a file you downloaded six months ago.

Questions

Is AEO different from SEO?

It is a subset with a different unit. Technical SEO decides whether your page can be crawled, indexed and rendered; if it fails, AEO is irrelevant. AEO decides whether a passage inside a crawlable page can be lifted and attributed. Do the technical work first.

Will blocking AI crawlers hurt my search rankings?

Blocking Google-Extended does not, and Google says so explicitly. Blocking GPTBot, ClaudeBot or PerplexityBot has no bearing on Google Search either, because those are different companies' crawlers. What blocking costs you is visibility inside those assistants.

Can I stop AI assistants reading my content entirely?

Not with robots.txt alone. Two of the user-initiated fetchers documented above either may not apply robots.txt rules or generally ignore them, because a person triggered the request. If the requirement is absolute, the content needs authentication.

Does publishing llms.txt get me cited?

There is no published evidence that it does. It is widely adopted, Lighthouse audits for it, and it costs little — but no engine documents using it as a citation input, and a guide that tells you otherwise is guessing.

How long until I see an effect?

Unknown, and specific to each engine's refresh cycle. Nobody can give you a number here, and a number offered without evidence is worth exactly what it cost to produce.

Next action

Open the page you most want quoted and copy its first hundred words into a blank document. Read them with no context. If you cannot tell what question the page answers, or a sentence starts with a word pointing at something that is no longer there, you have found the work — and it is an editing job, not a technical one.

Related reading: How to Run a Technical SEO Audit With Claude Code covers the crawlability and rendering work that has to be right before any of this matters, and Google Search Console With Claude Code covers measuring what conventional search is already doing with your pages.