INDEX-AI
An open standard that lets a website publish a clean, machine-readable version of itself, alongside the pages people see.
Today an AI agent reads a site by downloading pages built for humans and spending tokens on navigation, scripts and layout before it finds the text it came for. With INDEX-AI, the agent reads a short summary of each page first, then fetches only the pages it needs as clean text, with the exact size known before it asks. It gets exact, verifiable content at a known cost instead of guessing from markup.
This portfolio is my proof of work: every page on jordach.dev is also published as an Agent View, checked by the public reference validator.
- Conformance reported by the validator
- Level 2b
- Content nodes, one per page
- 23
- Checks passed
- 205 / 206
- Failures
- 0
The problem
A web page is written for a browser and a person. An agent that wants three paragraphs from it still has to fetch the whole document, pay for every token of markup, and reconstruct the site's structure from links and headings. It can't know in advance how large a page is or whether the page is relevant, so every fetch is a speculative spend.
Publishers have few ways to help: no standard place to say what the site contains, how its pages relate, how fresh they are, or what an agent may do with them.
The idea behind INDEX-AI comes from React. React's Virtual DOM is a lightweight mirror of the browser DOM, shaped for the job of diffing. INDEX-AI's Agent View is a lightweight mirror of a website, shaped for the job of being read by a machine. It sits next to the human site and doesn't replace it.
What I am building
I wrote the specification and its reference validator. The spec is a conformance ladder, so a site can adopt it one step at a time and each step pays off on its own. Levels 2a and 2b share one file: 2b only adds relations.
- Level 1Live here
AI Manifest
/.well-known/index-ai.jsonWho publishes the site, how fresh it is, what agents may do with it (search, citation, training), and where to go next.
- Level 2aLive here
Agent Index
/agent-index.jsonA flat inventory of content nodes. Each carries a short llm_summary and an llm_url pointing at clean text, with its size measured in advance.
- Level 2bLive here
Agent Graph
/agent-index.json + relationsSame file, plus typed parent / children / related links. Parent–child links must form a DAG, so agents can navigate structure instead of guessing it.
- Level 3Not yet here
Query Interface
MCP serverA typed MCP query interface over the Agent View, for sites too large or too dynamic to read node by node.
How an agent reads a site
The core of Level 2 is a two-phase decision model. In phase one, the agent reads each node's llm_summary, a few sentences that cost almost nothing, and decides whether the node is relevant. In phase two, it fetches only the relevant node's llm_url and gets clean text whose size it already knows. Nothing is spent on pages it would have thrown away.
without INDEX-AI fetch HTML page → unknown size → parse → discard markup → maybe relevant with INDEX-AI (Level 2) phase 1 read llm_summary → decide relevance, near-zero cost phase 2 fetch llm_url → clean text, size = content_chars (declared, exact)
The contract that makes phase two trustworthy is llm_url + content_chars + content_chars_mode. content_chars is a measurement: the Unicode NFC code point count of exactly what llm_url serves. "exact" means that count, and "max" means an upper bound for content that changes continuously. The optional content_sha256 hashes the same normalized string, so a validator can prove that the declared content is the served content.
At Level 2b, parent / children must form a directed acyclic graph: no node is its own ancestor, every child names its parent back, and there is at least one root. related is unconstrained and can point anywhere.
This site, implemented
jordach.dev publishes a full Agent View: 23 nodes covering Work, Leadership, Labs, Journal, Home, About, CV and Contact. Each node has a summary, a clean-markdown endpoint and a measured size and hash. The parent/children/related graph across all of them is acyclic and has no orphan references.
This case study's own node, rendered from the same builder that serves /agent-index.json:
{
"id": "work-index-ai",
"type": "x-case-study",
"label": "INDEX-AI — Case study",
"description": "An open specification proposal that makes websites verifiably readable by AI agents.",
"content": {
"llm_summary": "INDEX-AI is an open specification proposal that makes websites verifiably readable by AI agents, organized in progressive conformance levels from an AI manifest to a queryable MCP interface. Shipped, open source, 1.0-rc2 (RFC), with a reference validator.",
"llm_url": "/agent-content/work-index-ai",
"content_chars": 854,
"content_chars_mode": "exact",
"content_sha256": "42eb644391a6d5d3ea8171abf884a06f6066c86be3b83cb2e1e6040968fbd95f",
"summary_method": "manual",
"language": "en"
},
"meta": {
"updated": "2026-09-28",
"refresh_frequency": "monthly"
},
"relations": {
"parent": "work",
"children": [],
"related": [
"labs-index-ai-validator"
]
}
}The size and hash are computed from the exact markdown string served at /agent-content/work-index-ai, in the same process and with no separate fetch. What is declared and what is served come from the same string, so they can't drift apart.
Discovery: three independent channels
- HTML<link rel="agent-manifest"> in the <head> of every page
- HTTPLink: </.well-known/index-ai.json>; rel="agent-manifest" on every response, set in proxy.ts
- robots.txtAgent-Manifest: directive next to the sitemap pointer
Validated with the published tool
I checked the site with the reference validator as published on npm, the package anyone can run with npx, not a local copy or a mock.
$ npx @hardmachinelabs/index-ai-validator validate <url> --target-level l2b --html report.html # result on a local production build of this site Conformance: level-2b 205 / 206 checks passed · 0 failures · 1 warning
The one warning is a domain mismatch caused by testing on localhost (run with --allow-private-hosts) against a manifest that declares the production domain. It is a side effect of the test setup, not a gap in the implementation. Level 3 is not implemented on this site.
How to implement this with an agent
The toolkit I used to build this site's Level 2b implementation is now its own public repository. It holds two things, kept deliberately apart. The script is plain Node.js with zero dependencies: it writes the routes, the discovery wiring and the NFC / sha256 builder logic the same way every time. The skill is a set of instructions for an agent: it fills in the site's real content and validates it. Writing a route file is mechanical, so a script should do it. Deciding what a page is about and how pages relate takes an actual reading of the site, and no template can do that well.
Give an agent these four steps, in this order:
- 01
Read the spec.
Point the agent at the specification itself, not a summary:
docs/spec/SPEC-v1.0-rc2.mdin the index-ai repository. Before it writes anything, it should understand the conformance ladder, thellm_url/content_charscontract and the Level 2b DAG rules. - 02
Know the validator.
The agent should know about the reference validator and confirm it can run it before it needs it. The validator is published on npm as
@hardmachinelabs/index-ai-validatorand runs throughnpx, so there is no separate install step. Checking early means a registry or network problem shows up now, not in the middle of validation.npx --yes @hardmachinelabs/index-ai-validator --version
- 03
Run the script.
From a checkout of index-ai-nextjs, the script scaffolds the routes, the discovery wiring and a typed registry stub in the target Next.js project. The agent shouldn't hand-write any of this; the script exists so that it doesn't have to, and so that the output is the same every time.
node script/init.mjs \ --dir /path/to/your-nextjs-app \ --site-name "Your Site" \ --site-url "https://example.com" \ --description "One factual sentence about your site."
- 04
Run the skill.
The agent loads
skill/SKILL.mdand follows it. It reads the site's real content, writes each node's summary and clean-text body, and builds the relation graph. Then it builds the site and runs the validator from step 2, and repeats until the validator reports Level 2b conformance.
The order matters. An agent that has read the spec first knows why the script's output looks the way it does, instead of treating it as a black box. An agent that knows the validator before it writes content writes every node toward a target it can check from the start, rather than validating as an afterthought.
The toolkit targets Next.js App Router projects only. It is not an npm package yet; you run it from a local checkout.
Decisions that shaped the work
Publishers stay in control.
The site decides what agents see and under which terms. Usage preferences such as training: false live in the manifest, not in a third party's crawler settings.
Measure, don't declare.
content_chars is the NFC code point count of what llm_url actually serves, and content_sha256 hashes the same string. A validator can check both, so the size is a verifiable fact, not a publisher's promise.
Serve and measure from one string.
On this site the markdown served at each llm_url and the numbers declared about it come from the same in-memory string. Declared and served content can't drift apart, because there is only one source.
Build against the tool people run.
The published npm validator's bundled schema fixes spec_version to "1.0", while the spec's main branch already reads "1.0-rc2". This site declares "1.0" because that is what npx actually checks. Draft text and shipped tools can diverge, and I'd rather have a passing validator run than a claim.
Ship a validator, not only a document.
A standard nobody can test is an opinion. The reference validator turns every MUST in the spec into a check that can pass or fail.
Be exact about what is not done.
Level 3 is specified but not implemented here, and the spec's efficiency gains remain illustrative until its public benchmark is complete. The page says so.
The docs site states “Built with Shokunin Harness”. This is presented as the author's stated build context, exactly as displayed — not as proof of the harness.
Evidence and references
Full specification: docs/spec/SPEC-v1.0-rc2.md in the repository. Spec text CC-BY-4.0, code and examples MIT. RFC status — normative text, schemas and validator behavior may still change before a stable 1.0. Feedback, issues and pull requests welcome.
Public benchmark (benchmark/, reproducible with pnpm benchmark): 250 synthetic sites, seed 20260813. Level 1 cuts token consumption ~6.6× while answering identity/freshness (40% overall citation on this corpus); Levels 2a/2b answer all query types at 100% citation with ~6.5× fewer tokens than the page; Level 3 keeps 100% at 22% of the Level-0 cost. These are results on a synthetic corpus, not measurements of real-world agents.
“It does not, and cannot, guarantee that any given agent or search engine will discover, index, or cite the site — those outcomes depend on third-party systems and are treated here as benefits to be measured, never as promises.”