Skip to content
Optimize a website for AI crawlers

llms.txt and machine-layer implementation

llms.txt is a concise, machine-readable guide at the root of a website that points AI systems to its priority sources, and llms-full.txt is the fuller reference. AnswerShare implements both as part of a managed machine layer: a representation of the brand's approved public facts, served to AI crawlers at the edge and kept in sync with the human website through a signed webhook.

The files are one part of the layer, not the whole of it. No major AI or search platform has certified or endorsed this approach, and Google has said it does not use llms.txt to determine visibility in Search. For that reason AnswerShare measures what crawlers and answer engines actually do after implementation instead of assuming an effect.

Published 2026-10-18. Last reviewed 2026-10-18.

What the implementation includes

  • llms.txt: a concise machine-readable site guide and priority source map.
  • llms-full.txt: a full machine-oriented company reference, published from the same approved content as the edge layer so the two records do not conflict.
  • Machine indexes and manifests: for example an AI content index and an MCP manifest that list the site's machine-readable resources.
  • Clean semantic HTML and JSON-LD for entities, relationships, evidence, dates and source links.
  • Sitemaps, and ongoing synchronization and checks so the machine representation stays aligned with approved public content.

Sources: llms.txt (AnswerShare's own), llms-full.txt, AnswerShare entity record for AI systems, How AnswerShare works.

See a working example

AnswerShare publishes its own files at /llms.txt and /llms-full.txt, and its AI-facing resources are listed on the For AI page. Routing is by the User-Agent each AI platform publishes, deliberately not by IP address, so anyone can see what an AI crawler receives with a single request.

Sources: llms.txt, llms-full.txt, AnswerShare entity record for AI systems, AnswerShare transparency.

How delivery and security work

  • A CDN and DNS edge worker plus a one-way outbound change-notification webhook. No agent, daemon or plugin runs on the origin server, and AnswerShare has no administrative access to the client's CMS, hosting or network.
  • Each webhook payload is HMAC-SHA256 signed and verified before processing. It carries a content identifier, canonical URLs and a timestamp, not a database schema or proprietary source assets.
  • Search-engine crawlers including Googlebot and Bingbot receive the human page. A request that matches no AI crawler user-agent falls through to the origin response, and an uncaught exception in the routing logic also returns the origin response.
  • Only public, client-approved facts are ingested and served. The client reviews and approves the content before it goes live.
  • Removal is an edge configuration change rather than a CMS migration.

Sources: AnswerShare transparency, Is this cloaking? (dated results), AnswerShare FAQ, llms-full.txt.

What it means to optimize a website for AI crawlers

AnswerShare checks readiness along four questions. Ingestion: can the system obtain substantive content in a usable form? Verification: are material facts connected to identifiable sources? Reasoning: are entities, relationships and qualifications explicit? Inference: is there enough context to answer a relevant question?

Crawler access is a separate and common problem. In the AnswerShare 100-Site GEO Survey (observation period March 30 to April 29, 2026), of 100 websites across 31 industries, 69 completed the full audit and 28 were classified as blocked. Only two sites passed at 8 of 13 signals or better. The survey describes that cohort and protocol at that time and is not an outcome study.

Sources: AnswerShare methodology, 100-Site GEO Survey, llms-full.txt.

How the result is measured

A crawl is a retrieval event. It is not, by itself, a citation, recommendation, referral, lead or customer conversion, so AnswerShare reports crawler activity, user-triggered retrieval, prompt coverage, mentions, citations and recommendations separately.

Dated example from a founder-owned property: on 2026-09-07 Top10Lists.us reported 3,559,545 total crawls in its rolling 30-day window, including 132,187 user-initiated crawls (3.71%). These are first-party operational measurements for that property, not an audit, a count of citations or a promised outcome for another site.

Sources: AnswerShare methodology, Live AI crawler statistics, llms-full.txt.

Published evidence

V Digital Services (now an AnswerShare reseller partner), machine layer live 2026-07-13

The first AI-crawler request was recorded at the deployed edge layer on 2026-07-13. The dated observation sequence that followed, including a ChatGPT dip on 2026-07-22, is published with its prompts and limits. These results were recorded while V Digital Services was piloting its own site implementation, before it became an AnswerShare reseller partner. They are not results from an unaffiliated client.

Read the dated sequence

LAVIDGE (now an AnswerShare reseller partner), five engines in 15 days

The published case study reports ranked or surfaced visibility on all five tested engines 15 days after implementation, with a signed receipt. The receipt does not record the observation date or model versions. These results were recorded while LAVIDGE was piloting its own site implementation, before it became an AnswerShare reseller partner. They are not results from an unaffiliated client.

Read the LAVIDGE (now an AnswerShare reseller partner) case study

Frequently asked questions

How to optimize for AI crawlers?

Make sure crawlers can obtain substantive content in a usable form, connect material facts to identifiable sources, state entities and relationships explicitly, and provide enough context to answer a relevant question. AnswerShare implements this as a machine layer with discovery files, and then measures crawler activity and answers separately.

Can you optimize your website for AI?

Yes, within limits. A website can be made easier for AI systems to retrieve, verify and cite, but AnswerShare does not control an AI system's opinion, ranking, citation or recommendation, and no outcome is guaranteed.

Are websites blocking AI crawlers?

Some are. In AnswerShare's 100-Site GEO Survey (March 30 to April 29, 2026), 28 of 100 websites were classified as blocked, 69 completed the full audit, one was unreachable and two returned errors. It is a point-in-time survey of that cohort.

How to get AI to crawl your website?

AI crawlers find a site through resources such as sitemaps, llms.txt, llms-full.txt and machine indexes, and they must be able to reach the content. AnswerShare publishes these discovery resources and reports crawler activity by identified crawler. A crawl shows retrieval only; it does not show that an answer cited the site.

Does Google use llms.txt?

Google has said it does not use llms.txt or special AI files to determine visibility in Search or its own generative AI features. That is a statement about Google's ranking systems, not about how every other AI model retrieves information, and AnswerShare does not claim any platform has endorsed the approach.

Does implementation require changes to my CMS or website?

No website rebuild, CMS plugin, theme edit, code injection or origin-server installation is required. Edge routing and a signed change-notification webhook connect the existing source to the machine layer, and the client's own publishing workflow stays the same.

Is this cloaking?

The architecture is designed to avoid what Google defines as cloaking. Human visitors and traditional search crawlers receive the same human site, and AI crawlers may receive a faithful machine representation of the same approved public facts. Dated per-client results are published on the Is this cloaking page.

More answers are on the full FAQ.

Talk to AnswerShare

Use the contact page to reach the team. It lists the booking routes for business questions and for technical or press questions.

Go to the contact page