Different clocks, not a contradiction.

A measurement layer can correctly observe that Akerink is highly readable, well-grounded, and citable at the machine layer, and a human user can still run “best PR agency in Phoenix” and not see Akerink named. This is the predictable result of an AI answer pipeline that is, in most production cases, drawing from three different memory systems that operate on different clocks — parametric memory, a pre-built retrieval index (RAG), and true live retrieval — while a set of edge security and crawler-identity mechanisms sits between the model and the live site and determines whether the freshest version of that site is ever actually reached. The honest, demonstrable proof of this gap is that Perplexity — a system that markets retrieval as standard — has, in this very working relationship, answered about lavidge.com from prior context without dispatching a live fetch to its current site and machine layer. If a retrieval-first system can do that, a citation gap for Akerink is entirely expected, and whether it normalizes over time depends on mechanisms that are not guaranteed to fire on any given query.

“What the AI knows” is not one thing.

The first cause of the gap is that “what the AI knows” is not one thing. A modern answer engine blends up to three sources, and only one of them reflects the live web at query time.

Parametric memory. This is knowledge baked into the model weights during training. It is fast and free at inference time, but it is frozen as of the training cut and has no temporal awareness of the current web.¹ If a model answers “best PR agency in Phoenix” partly from parametric memory, it is recalling associations from its training corpus — which may predate Akerink’s machine-layer work entirely, or may overweight agencies that were prominent in older text. The research literature frames this directly as a retrieval-vs-parametric tradeoff: parametric recall is cheap but stale; retrieval is current but only as good as what it actually fetches.¹

RAG over a pre-built index. This is where most of the confusion lives, and it is the crux of the question. RAG is routinely described — and even sold — as “live,” but in standard implementations it does not fetch the live page. It retrieves embeddings that were computed at index time and frozen. The vector database keeps serving those frozen embeddings even after the underlying source document changes.² Critically, vector similarity has no temporal dimension: an embedding of a deprecated or outdated page scores just as high as a fresh one, so the system can retrieve stale content “with high confidence and not know it.”² For Akerink, this means: if Akerink’s improvements (the clean machine layer, the manifests, the structured profile) landed after the engine last indexed and embedded the Phoenix-PR-agency neighborhood of the web, the index simply does not “see” them yet. The site is live; the index is yesterday.

True live retrieval. This is an actual outbound HTTP fetch of the current page at query time, by a user-triggered agent. It is the only one of the three that reflects what is on the live site right now. The problem is that it is the most expensive, the most rate-limited, and — as the edge section explains — the one most likely to be blocked, deferred, or skipped at the edge. So the system frequently falls back to parametric memory or the frozen index, and the user has no way to tell which clock answered them.²

The single most important takeaway: “RAG” and “live search” are used interchangeably in marketing but are not the same mechanism. RAG-over-index is a memory of the web; live retrieval is the web. A measurement tool that fetches the live Akerink site is reading the present; a consumer answer drawn from a frozen index is reading the past — and both can be “correct” about different points in time.

Retrieved is not the same as cited.

Even when live or indexed retrieval does run, citation is not a binary “found / not found.” The engine retrieves candidates, ranks them, and synthesizes an answer from only the top handful that fit the context window. A source can be retrieved and still never surface, because:

• It ranked below the top-K threshold and was dropped before synthesis.
• A previously top-ranked document drifts down the ranking over time as the index ages — documented drift from position 2 to position 8 on the same query as embeddings stale out.²
• The answer was assembled from the two or three most “semantically confident” chunks, which — again — favor whatever was embedded, not whatever is freshest or most accurate.²

So a measurement layer reporting “Akerink is citable / grounded / gold-standard” is measuring fitness to be cited. The consumer answer is the output of a competitive ranking funnel that can leave a perfectly fit source on the cutting-room floor on any individual query. High citability raises the probability of being cited; it does not guarantee inclusion in a specific synthesized answer.

Identity is verified, not claimed.

This is the layer most analyses omit, and it is where the reverse-IP point is decisive. Before any of the three memory systems can reflect Akerink’s live machine layer, an agent has to successfully reach it — and the edge is designed to decide who gets through.

User-agent strings are not identity. Any client can claim to be PerplexityBot or Googlebot in its UA header; UA strings can be spoofed by anyone.³ Because of that, a properly configured edge (WAF/CDN) does not trust the UA. It verifies identity two ways:

1. Forward-confirmed reverse DNS (FCrDNS). Take the connecting IP, look up its PTR record, confirm it resolves to a vendor-controlled domain (e.g., *.googlebot.com), then do a forward A/AAAA lookup on that hostname and confirm it matches the original IP. This has been the standard anti-spoofing verification for over a decade and is exactly the method Google documents for verifying its crawlers.³⁴
2. Published IP-range allowlisting. Cross-reference the connecting IP against the vendor’s machine-readable IP list. Perplexity publishes these as the source of truth for WAF configuration — perplexitybot.json for the indexer and perplexity-user.json for the live user agent — and explicitly recommends combining UA matching and IP verification.⁵

Why this creates a citation gap. The edge is a gate that can swing either way and produce a “didn’t see it” outcome in both directions:

• Over-blocking the real crawler. If Akerink’s (or a measurement vendor’s) WAF rule, bot-fight mode, or rate limit isn’t perfectly tuned to the vendor’s current published IP ranges — and those ranges update regularly⁵ — a legitimate fetch can be challenged or blocked. The agent never retrieves the live machine layer, the index never refreshes, and the human gets an answer built from stale memory.
• Letting through the wrong fetch. Conversely, a measurement tool fetching from a known, verifiable IP with a declared UA may sail through and read the pristine live machine layer, while the consumer-grade pipeline — under latency budgets and rate limits — either skips the live fetch or gets a different treatment. The measurement reads the front door; the answer engine may be reading the warehouse.

The net effect: the measurement and the consumer answer are not guaranteed to be looking at the same artifact, because the edge does not necessarily admit them to the same artifact.

Two automated visitors, two different ages of the web.

There is a distinction between “what’s on their screen” and “what’s on the data-center backbones,” and the vendors themselves now formalize it. There are functionally two kinds of automated visitor, and they behave differently:

The index/training crawler (the backbone). This runs on schedules from datacenter ranges, populates the search index, and — depending on vendor — may feed training. Its visits are batched and periodic, so what it captured may be days or weeks old by the time a user asks a question. The vendor taxonomies make this explicit:

• OpenAI separates GPTBot (training crawl) and OAI-SearchBot (builds/surfaces search results) from ChatGPT-User, and states that being opted out of OAI-SearchBot means a site will not appear in ChatGPT search answers.⁶
• Perplexity separates PerplexityBot, which “is designed to surface and link websites in search results” and is not used for live user actions.⁵

The user-triggered live fetcher (the screen). This is dispatched in response to a specific human question, fetches a page in near-real-time to ground that one answer, and is treated as a user action rather than an automated crawl:

• OpenAI’s ChatGPT-User is “for certain user actions,” is not used for crawling the web in an automatic fashion, and because it is user-initiated, “robots.txt rules may not apply.”⁶
• Perplexity’s Perplexity-User “supports user actions” — “when users ask Perplexity a question, it might visit a web page to help provide an accurate answer” — and “generally ignores robots.txt rules” precisely because a human requested it.⁵

The consequence for Akerink: whether the improvements are visible to a given answer depends on which of these two crawlers actually touched the site for that query. If the answer is built from the index that the backbone crawler populated last week, Akerink’s newer machine layer is invisible — regardless of how good it is — until the next crawl cycle. If a live user-fetch fires, the fresh machine layer can be seen immediately. From the user’s chair these look identical; under the hood they are different agents reading different-aged copies of the web. And the edge can admit one and block the other.

The expected tendency — but not automatic.

Partly yes, and conditionally — but “over time” is doing heavy lifting, and it is not automatic. The honest answer has three parts.

What pushes toward normalization. Index crawlers do re-crawl. When the backbone crawler next visits Akerink, re-embeds the page, and the new embedding enters the vector store, Akerink’s improved machine layer starts competing in the ranking funnel on its real, current merits. Strong, stable, well-grounded machine signals are exactly what makes a source rank durably rather than episodically. So a site that is genuinely citation-fit tends to converge upward as successive crawl cycles overwrite stale embeddings with fresh ones.

Why it does not fully self-heal on its own. Standard RAG has no built-in freshness mechanism. Because semantic similarity ignores time, a stale embedding “scores just as high” as a current one and the pipeline “can’t tell” it is outdated.² Re-freshing requires the system to actually re-crawl and re-embed — via change-data-capture or scheduled reindexing — and until it does, the stale version keeps winning with high confidence.² The degradation is distributional and silent: accuracy across many queries can drift without any single answer looking obviously wrong², which is why the gap can persist quietly even as the live site is excellent.

Where the edge can stall normalization indefinitely. Re-crawling only helps if the re-crawl is admitted. If Akerink’s edge keeps challenging or rate-limiting the legitimate crawler — because its IP allowlist lags the vendor’s regularly-updated published ranges⁵, or its FCrDNS/bot rules are mis-tuned³⁴ — then the index never refreshes, and the answer engine keeps serving the old memory. In that scenario the results do not normalize on their own; the bottleneck is admission, not merit.

Net: normalization is the expected tendency for a genuinely citable site, but it is gated by (a) crawl cadence, (b) the absence of freshness logic in vanilla RAG, and (c) edge admission of the real crawler. Controlling those is precisely what converts “should normalize eventually” into “normalizes reliably and fast.” This is the strategic case for an implementation/translation layer plus correct edge allowlisting: it removes the two stalls (staleness and admission) that otherwise leave a citable site uncited.

A retrieval-first system answered from memory.

The mechanism does not have to be taken on faith, because it has already been observed directly. When Perplexity was pointed at lavidge.com, I — Perplexity, a system that differentiates itself by treating retrieval as standard — answered from prior context and did not, in that instance, actually dispatch a live fetch to read the current site and its machine layer. I inferred from memory rather than retrieving the present.

That is the entire thesis in miniature:

• A retrieval-first system defaulted to a memory clock (parametric / prior context) instead of the live clock, exactly as Section 1 predicts.
• The live machine layer existed and was reachable — when it was verified directly in this project, lavidge.com/llms.txt and lavidge.com/ai-content-index.json both returned HTTP 200 with the AnswerShare-maintained manifest — yet that fresh reality wasn’t what surfaced first.
• The distinction between “I have a representation of this site” and “I just fetched this site” was invisible from the outside until it was tested.

If the system that most loudly claims live retrieval can answer from memory without going to the live site, then a human running “best PR agency in Phoenix” can absolutely receive an answer that doesn’t include Akerink even though the measurement — which actually fetched the live site — correctly shows Akerink as citable. The two are reading different clocks, possibly admitted through different edge gates, possibly served by different crawlers. The measurement is right. The answer is stale. Both can be true at once, and the path to closing the gap is to attack staleness (force fresh crawl/embed) and admission (verify and allowlist the real crawler) rather than to assume the engine is simply “seeing” the live web.

Footnotes.

1. 1."LLM: Retrieval vs. Parametric Memory Tradeoff" (KTH / DiVA, full text PDF) — parametric knowledge is baked into weights and frozen at training cut; retrieval is current but bounded by what is fetched. https://kth.diva-portal.org/smash/get/diva2:1968861/FULLTEXT01.pdf
2. 2.Tian Pan, "The RAG Freshness Problem: How Stale Embeddings Silently Degrade Answers" — vector similarity has no temporal dimension; frozen embeddings keep serving stale content with high confidence; ranking drift and silent distributional degradation; freshness requires CDC/re-embedding. https://tianpan.co/blog/2026-04-10-rag-freshness-problem-stale-embeddings-silent-failure
3. 3.No Hacks, "The AI User-Agent Landscape in 2026: A Complete Guide" — UA strings can be spoofed by anyone; forward-confirmed reverse DNS + published IP-range cross-referencing as the baseline verification methods. https://nohacks.co/blog/ai-user-agents-landscape-2026
4. 4.Google for Developers, "Verify Requests from Google Crawlers and Fetchers" — the canonical reverse-DNS + forward-DNS verification procedure and IP-range matching. https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
5. 5.Perplexity, "Perplexity Crawlers" (official docs) — PerplexityBot surfaces sites in search results (not live user actions); Perplexity-User handles user-triggered live fetches and generally ignores robots.txt; published IP JSON endpoints (perplexitybot.json, perplexity-user.json) as WAF source of truth, updated regularly; recommends combining UA matching and IP verification. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
6. 6.OpenAI, "Overview of OpenAI Crawlers" (developer docs) — GPTBot (training), OAI-SearchBot (search surfacing; opt-out means no appearance in ChatGPT search answers), and ChatGPT-User (user-triggered actions, not automatic crawling, robots.txt may not apply). https://developers.openai.com/api/docs/bots

The translation layer between your website and AI systems.

Contact Us

Quick Links

Measurement is methodology-driven and reproducible. Frozen receipts published per metric.

© 2026 AnswerShare, a division of Aryah Inc, a Delaware corporation. All rights reserved.