Making neurarod.com AI-First: llms.txt, JSON-LD, and Markdown for Agents
AI agents and answer engines now read neurarod.com directly: a llms.txt summary, structured data instead of parsed prose, an explicit crawler allowlist, and raw markdown on request.
Most visits to a website used to mean one thing: a browser rendering HTML for a person to read. That is no longer the whole picture. Answer engines and AI agents now read sites too, and they do not parse a page the way a browser does. We built neurarod.com to be legible to them directly, not as an afterthought bolted on later.
llms.txt at the Root
llms.txt is a plain markdown file at the site root: what NeuraRod does, which agents we build, and how to reach us. An agent reads it in one request instead of stripping tags out of rendered HTML.
Structured Data Instead of Prose
The page now carries JSON-LD. A sitewide Organization block renders on every page:
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "NeuraRod",
"url": "https://neurarod.com",
"logo": "https://neurarod.com/logo.webp",
"email": "info@neurarod.com",
"sameAs": ["https://www.linkedin.com/company/neurarod", "https://x.com/neurarod"]
}
Each blog post adds a second block, BlogPosting, built from the same title, date, and author already in the post's frontmatter. This post carries one. When an agent is asked what NeuraRod does, it reads these fields directly from a schema, not a guess pieced together from marketing copy.
An Explicit Crawler Allowlist
robots.txt allowed everything by default before, wildcard included. It now names the crawlers we want: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, among others. This makes the choice explicit, not an oversight nobody checked.
Markdown on Request
Send Accept: text/markdown to neurarod.com and you get raw markdown back, not rendered HTML. A CloudFront Function checks the header before the cache lookup and rewrites the request to a markdown mirror, so HTML and markdown responses cache separately with no extra configuration. The rewrite only fires for pages that actually have a markdown version, so nothing on the site 404s from it.
CloudFront Functions run at the edge in a restricted JavaScript runtime: sub-millisecond execution, no network access, no file I/O. Lambda@Edge would allow full Node.js and a real check for whether a markdown file exists, at the cost of higher latency and a separate deployment pinned to us-east-1. The viewer-request function on this site already existed, handling the www to apex redirect and the pretty-URL rewrite that maps /contact to /contact/index.html. Adding the Accept-header check was one more branch in a function already running on every request, not a new piece of infrastructure. The tradeoff shows up as the allowlist: since the function cannot check S3, it can only rewrite to a markdown mirror for paths it already knows have one.
Why It Matters
None of this changes what a human visitor sees. It changes what a machine reader gets: instead of inferring facts from paragraphs written for people, it reads a file built for it. We build AI agents for other companies. Making our own site something an agent can read cleanly is not optional. It is the standard we hold our own work to.
What's Next
The same four changes are done on neurotko.com, the dedicated site for our Neurotko product. That site is still in internal review before we open it up.
Pages are one layer. APIs are another. The API portal is the clearest signal of whether your company can handle AI agents makes the case for MCP and API governance as that next layer.