Empirical Study · AUTOPSY V4

AUTOPSY V4 Benchmark

Empirical analysis of 2,000 domains and 50,665 public links declared across llms.txt implementations.

Technical study conducted July 13–18, 2026 by Hard Machine Labs. Deduplicated corpus reconciled across 8 shards with zero missing or unexpected observations.

Key Metrics

Empirical Evidence & Raw Data

50,665 destination URLs analyzed

Analyzed Domains

2,000

1,711 valid llms.txt files (85.55%), 1,492 domains with active observations.

Direct Text Delivered

28.05%

Of HTTP 200 links serve direct Markdown/Text. 71.95% point to standard HTML pages.

Median HTML Overhead

53.07×

Raw HTML documents contain 53× more tokens than the main extracted plain text content.

Dead Links (404/410)

2.76%

Dead link debt is minor. Maintenance is not the primary retrieval bottleneck.

Why This Study Matters

The Core Problem: Discovery is Not Delivery

The llms.txt specification has rapidly gained adoption as the standard for declaring AI-friendly site maps. However, a critical technical assumption remained unverified: once an URL is declared in a llms.txt file, is its content directly accessible, extractable, and machine-usable without heavy rendering dependencies?

AUTOPSY V4 does not test LLM citations or SEO traffic. It audits the raw technical pipeline: Declaration → Fetch → Content-Type → Extraction → Rendering.

"llms.txt declares where content lives, but declaration guarantees neither direct delivery, clean extraction, nor operational quality for AI agents. Discovery is not delivery."

Empirical Results

Link Breakdown & Format Analysis

Automated classification across 50,665 observations

1. Response Category Breakdown Across Declared Destinations

Exclusive analysis of all 50,665 links extracted from 1,711 valid llms.txt manifests.

Useful SSR HTML (≥ 200 extractable tokens)54.00% (27,360 links)

Direct Text / Raw Markdown26.42% (13,385 links)

Empty or Unextractable (< 200 tokens)8.31% (4,209 links)

JS-Dependent (Requires Headless Chromium)3.51% (1,780 links)

Dead Links (HTTP 404 / 410)2.76% (1,396 links)

Access Blocked / Robots / Rate-Limited / Other5.00% (2,535 links)

2. Major Finding: `llms.txt` Adoption is BIMODAL

Analyzing per-domain behavior reveals that the 28% average direct text rate stems from two completely distinct engineering patterns:

61.7%

"Pointer" Sites

61.7% of domains serve 0% direct text. Their llms.txt file lists standard HTML web pages — declaring discovery without dedicated text delivery infrastructure.

21.9%

"Mirror" Sites

21.9% of domains serve ≥ 75% (often 100%) direct text. They deploy a clean parallel Markdown representation for every single page.

The Solution & Governance Bridge

From Benchmark Diagnosis to Agent View Action

AUTOPSY V4 proves that deploying a llms.txt file is only step one. That is why we built Agent View: the platform that audits, verifies, and guarantees observability for your machine-readable web layer.

01 · Real-Time Delivery Auditing

Agent View automatically tests whether your core pages (Pricing, Specs, Docs) serve clean text or drown AI agents under 50× HTML overhead.

02 · Mirror vs Pointer Classification

Instantly assess your index-ai compliance score and upgrade your site from a "pointer" manifest to a high-fidelity mirror architecture.

03 · Remediation & SHA-256 Proofs

Fix JavaScript rendering dependencies on critical routes and issue SHA-256 anchored retrieval certificates for your enterprise AI partners.

Explore Agent View Demo Read index-ai Open Spec