By Robert Maynard, Jr., AnswerShare
Markdown request preferences across five web properties
Did frontier AI crawlers ask for Markdown? In the saved request headers we could recover, none expressed an explicit preference for it.
Primary finding
Across 4,300,723 frozen field events, 1,277,965 carried the labels GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot or Meta-ExternalAgent. None of the 1,258,190 non-empty saved headers in that cohort expressed an explicit positive text/markdown preference. Another 19,775 cohort events lacked a non-empty saved header. Missing headers are not counted as observed negative preferences.
No explicit Markdown preference was observed in the available saved headers of this defined crawler cohort. This study did not measure citations, recommendations, reputation, or the effect of Markdown on model behavior.
User-triggered fetchers are a separate population (see the Claude-User note below). ChatGPT-User recorded none in 37,136 saved non-empty headers (592 missing); Perplexity-User recorded none across 5,069. These are user-agent-asserted labels, not authenticated vendor-origin traffic.
Note: Claude-User requests
Of 648 requests labelled Claude-User in the four-property release, 245 asked for Markdown. All 245 identify as Claude Code's fetch tool, which runs on users' own computers rather than as Anthropic's crawler. They come from 27 software versions, consistent with independent users, though some may be our own sessions and the data cannot rule that out. The other Claude-User-labelled requests mostly sent */*. The frozen data holds no IP addresses, so none of these labels can be verified as vendor-origin. An earlier figure of 248 of 661 could not be reconciled with the audited count and is not used.
01 / The result
What the saved headers show
User-triggered fetcher
245 / 648
Claude-User-labelled requests (four-property release) whose saved header carried a positive text/markdown preference (37.81%); all identify as Claude Code's fetch tool on users' own computers, not Anthropic's crawler. ChatGPT-User: 0 of 37,136 saved non-empty headers. Perplexity-User: 0 of 5,069.
Five-property field measurement
4,300,723
Frozen field events at AnswerShare, Top10Lists, V Digital Services, LAVIDGE, and Aker Ink. Each retained the raw Accept header where one was saved, and a derived explicit-Markdown flag.
Named crawler cohort
0 / 1,258,190
Non-empty saved headers with an explicit text/markdown preference in the named five-label cohort. A further 19,775 events of the 1,277,965 had no saved header and are counted as missing.
| Question | Answer supported by this study |
|---|---|
Does */* count as a Markdown request? | No. Wildcards are recorded separately and never count as an explicit request for text/markdown. |
| What did the large field study measure? | 4,300,723 frozen field events across five live properties. Its question was which crawler labels expressed an explicit Markdown preference in the request headers that were saved. |
| How often did frontier AI request Markdown? | Never, in the saved headers. None of the 1,258,190 non-empty headers in the 1,277,965-event cohort expressed an explicit preference; 19,775 events had no saved header and are treated as missing. |
| Can aggregate crawl totals answer either question? | No. Aggregate Cloudflare reporting has no verbatim request Accept header, so it is excluded from the prevalence denominator. |
02 / Pattern
Which crawler labels sent an explicit Markdown preference
48 distinct crawler labels were observed across the frozen window. This is the complete list of the ones with any explicit Markdown preference at all. Labels are asserted by the user-agent string; no frozen export includes IP evidence for vendor authentication.
| Crawler | Explicit Markdown requests | Observed requests | Rate |
|---|---|---|---|
| ShapBot | 3,588 | 3,588 | 100.00% |
| ExaSearchBot | 67,538 | 68,209 | 99.02% |
| Claude-User | 245 | 648 | 37.81% |
| Googlebot | 32,747 | 676,507 | 4.84% |
| unclassified-bot-like | 3,476 | 240,578 | 1.44% |
| unknown_bot | 1,139 | 388,249 | 0.29% |
text/markdown
. Only six of 48 labels ever did; the named frontier cohort never did in its saved headers (0 of 1,258,190; 19,775 events had no saved header). Values are the combined five-property totals in the table above (section 02), except Claude-User, which is the audited four-property count (245 of 648). ShapBot and unknown_bot match the AnswerShare-side manifest exactly; the other rows include the separately frozen Top10Lists component. Rates are rounded to two decimals; a zero is drawn with no bar.
Named frontier cohort
0 / 1,258,190
GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, and Meta-ExternalAgent sent no explicit Markdown preference in the non-empty saved headers of the frozen field set.
Everything else measured
42 crawlers, 0%
The remaining 42 distinct crawler labels — including the named frontier cohort above — showed exactly zero explicit Markdown preferences in their saved headers. The full per-crawler breakdown for all 48 labels is in the downloadable manifest below.
03 / Evidence release
How a reader will check it
Frozen data-release status
The AnswerShare, V Digital Services, LAVIDGE, and Aker Ink component is frozen and posted below as CSV and NDJSON packages whose SHA-256 checksums are recorded in the manifest. The manifest was signed with Ed25519 on 2026-08-17 under key ID answershare-audit-v1, whose public half is published at /.well-known/jwks.json. The signature can be verified with that public key. It protects signed-content integrity; the signer-supplied date is not a trusted timestamp and is not proof that the files predate any later dispute. Its manifest records the exact window, row count, SHA-256 hashes, query definition, and exclusions.
The separately frozen Top10Lists component supplies the remainder of the combined five-property denominator. That snapshot has been recovered read-only; an existing signature for that component has not been located. New checksums identify its recovered bytes. No changing dashboard total is used as evidence.
To check the signature yourself, download the manifest and its signature below and run our verifier, which shares no code with the signer and fetches the public key from /.well-known/jwks.json: node verify-study-release.mjs manifest.json manifest.sig.json. It is published at /receipts/verify-study-release.mjs. The canonicalization is RFC 8785 (JCS), so an independent implementation reaches the same bytes.
Four-property release manifest
The frozen AnswerShare-side component: scope, cohort definition, row counts, exclusions, and public-artifact hashes. It explicitly identifies Top10Lists as a separate source component.
Open manifest ↗Row-level log — NDJSON
One sanitized, deduplicated observed request per line. This is the byte-faithful public field rendition, suitable for reproducible analysis in code.
Download NDJSON.gz ↗Row-level log — CSV
The same frozen field events in spreadsheet form. Formula-like cells are escaped for spreadsheet safety; use NDJSON when the exact stored value matters.
Download CSV.gz ↗Ed25519 signature over the manifest
The detached provenance receipt: signature, key ID, freeze-time stamp, and the SHA-256 of the manifest bytes it covers. Verify it against the published public key; a tampered manifest fails.
Open signature ↗Checksums and collector rule
Verify the downloaded files before analysis, then inspect the exact request-boundary collector and the positive text/markdown detection rule.
Download SHA256SUMS ↗Top10Lists release manifest
The other frozen study component: its exact 1,716,846-record count, declared frontier cohort, Markdown-preference totals, and the schema for its public rows.
Open manifest ↗Top10Lists row-level log
A public NDJSON export of the Top10Lists component. It is paginated in pages of up to 5,000 records; response headers provide the next offset and total row count.
Download first NDJSON page ↗The full field denominator is the sum of the two frozen components: 2,583,877 four-property events plus 1,716,846 Top10Lists events. The Top10Lists manifest documents the pagination contract for retrieving every published row.
[graphic: Figure 2. Composition of the 4,300,723-observation field denominator] Figure 2. The two frozen components that sum to the 4,300,723-observation denominator (60.1% and 39.9%). Each component has its own manifest.The collector source andexplicit-Markdown parserare available for code review.
Response data
The four-property public export contains request-side data, not response bodies or Content-Type. Top10Lists also includes logged served_format and response_content_type fields; 88,461 Content-Type values are missing. These are not full response payloads and do not establish downstream AI use.
Evidence quality, Top10Lists
Top10Lists contains 102,982 stored positive flags but only 102,761 positives supported by non-empty saved headers. The remaining 221 have no usable saved header and remain missing. Another 4,363 presence flags disagree with the saved header strings. Original records are preserved; no flags were silently fixed. Stored-positive and header-supported counts are separate columns in the paper.
Bot identity
No frozen export includes IP evidence for vendor authentication. Positive Googlebot-labelled rows cannot be attributed to Google or confirmed as spoofed. All other vendor-labelled traffic has the same limitation.
Controlled serving test, pending
The original study reports a separate 20,000-transaction Cloudflare serving test. The receipt archive, configuration and collection dates are pending inclusion in the publication package. It is not part of the field denominator, and nothing on this page relies on it. Linking the field release does not supply the controlled receipts.
Conflict disclosure. AnswerShare conducted and funded the study and sells a competing method of delivering content to AI systems. The authors have a commercial interest in comparisons with negotiated Markdown.
The public package will not include IP addresses, cookies, query strings, authentication material, or internal warmer traffic. A raw header that is not observed is represented as NULL, not as a negative Markdown request.
04 / Method
What counts as Markdown
A request is positive only when its verbatim Accept header contains the exact media type text/markdown with a positive quality value. The missing quality value defaults to 1. A value ofq=0 is negative.
text/*, */*, and an absent header do not count as explicit Markdown preferences. An absent or unavailable header is a different state from an observed header that did not request Markdown.
The collector is at the request boundary and records the raw header before the response is returned. It also records the response content type, so the release can test the claim about negotiation rather than only describe user agents.
Exclusions
- Internal cache warmers and audit probes
- Cloudflare aggregate rows with no raw header
- Duplicate logging legs of one request
- Rows outside the declared freeze window
05 / Reproduce it
How to repeat the test yourself
The header finding can be re-checked from the published rows, and the effect of serving state on AI web-search answers can be re-tested with a simple on/off protocol. Both are written in plain words so another group can run them without us.
A. Re-check the header counts
- Download the manifest, the NDJSON or CSV rows, the signature and the checksums from the evidence release above.
- Check the SHA-256 checksums, then verify the manifest signature with
verify-study-release.mjs. - Apply the published parser to the raw Accept header on every row.
- Count rows whose parsed result is positive, per crawler. Your totals should match the manifest.
B. Repeat the serving-state test (worker off, then on)
- Fix the prompt set and its version, the list of AI engines, and the engine and model versions. Write them down before starting.
- Turn the AnswerShare worker OFF for the test property so crawlers and users see only the ordinary site. Record the time and confirm the response a crawler receives.
- Run every web-search query against every engine. Save the full returned answer, the citations and any queries the engine exposes, for each run.
- Turn the worker back ON. Record the time and confirm the machine-layer response.
- Run the same queries on the same engines again, with the same settings, and save everything the same way.
- Compare the OFF and ON runs question by question: which sources were cited, whether the brand appeared, and what the answer said.
Rules that keep the comparison honest: the same prompts and engines in both states; repeat each state more than once because engine answers vary; keep every attempt, including failures; treat a missing result as missing, never as zero; and report the dates and the instrument and parser versions with the counts. This is the method. Results of such runs are published separately with their own data and are not part of this field release.
06 / Limits
What this does not establish
The study is not a census of the web or a measurement of how often a crawler reads Markdown after receiving it. It measures observed request preferences at the participating boundary.
A user-agent string is an asserted identity unless separately verified. The release labels the verification method for each row instead of treating user-agent text as proof.
A zero is limited to the declared window, cohort and saved headers. It does not mean a crawler can never request Markdown, nor that its product does not use Markdown by another route. Missing headers are missing, not negative.
The study does not measure citations, recommendations, reputation, ranking behavior, or the effect of Markdown on model behavior. It records which request headers were saved, and says nothing about what happened after a page was fetched.
← Back to Research FAQ