Serving markdown to AI agents with Accept headers: the .NET version

A small site made the Hacker News front page this week with a deceptively simple pitch: when an AI agent asks for your page, hand it markdown instead of HTML. Same URL, different representation, decided by the Accept header. The acceptmarkdown.com writeup frames it as content negotiation, which is exactly what it is, and the HN thread went the way these threads usually go: half the room loves the elegance, the other half asks who is actually sending Accept: text/markdown today.

Both reactions are right, and that gap is the interesting part. The idea is clean HTTP. The reality is messier. I want to walk through what this looks like when you build it in ASP.NET Core, where the sharp edges are, and whether it is worth your time.

Why anyone wants this

An LLM that fetches your blog post does not want your navigation, your cookie banner, your three related-content rails or the eleven kilobytes of Tailwind classes wrapped around forty words of actual prose. All of that costs tokens, and tokens are context budget the model would rather spend on your content. Strip the page down to markdown and you send a fraction of the bytes, with almost no structural noise for a retrieval pipeline to trip over.

So the argument holds. The question is how the agent tells you it wants the stripped version, and that is where Accept comes in. It is the oldest trick in HTTP: one URL, many representations, the client states a preference, the server picks. We have done this for years with application/json versus application/xml. Markdown is just another media type. It even has a registration, text/markdown, from RFC 7763.

The naive version, and why it is almost right

Here is the shortest thing that could possibly work in a minimal API:

app.MapGet("/blog/{slug}", async (string slug, HttpRequest req, IContentStore store) =>
{
    var post = await store.FindBySlug(slug);
    if (post is null)
        return Results.NotFound();

    if (req.Headers.Accept.ToString().Contains("text/markdown", StringComparison.OrdinalIgnoreCase))
        return Results.Text(post.Markdown, "text/markdown; charset=utf-8");

    return Results.Content(RenderHtml(post), "text/html; charset=utf-8");
});

This works in a demo and it will bite you in production. The Accept header is not a substring you scan. It is a ranked list with quality values, like text/html,application/xhtml+xml,text/markdown;q=0.9,*/*;q=0.8. A crude Contains says yes to a client that only mildly prefers markdown and would actually rather have HTML. It also matches */*, which every browser and every curl sends, so you end up handing markdown to people who asked for a web page.

Parse it properly. ASP.NET Core ships the tooling in Microsoft.Net.Http.Headers:

using Microsoft.Net.Http.Headers;

static bool PrefersMarkdown(HttpRequest req)
{
    var media = req.Headers.GetCommaSeparatedValues("Accept")
        .Select(v => MediaTypeHeaderValue.TryParse(v, out var mt) ? mt : null)
        .Where(mt => mt is not null)
        .Select(mt => mt!)
        .ToList();

    double QualityFor(string type) => media
        .Where(mt => mt.MatchesMediaType(type))
        .Select(mt => mt.Quality ?? 1.0)
        .DefaultIfEmpty(0)
        .Max();

    var markdown = QualityFor("text/markdown");
    var html = QualityFor("text/html");

    return markdown > 0 && markdown >= html;
}

Now a client only gets markdown if it explicitly listed text/markdown and did not rank HTML higher. A browser sending text/html,...,*/*;q=0.8 never trips it, because text/markdown scores zero. That one function is the whole feature, and it is the part the demo version got wrong.

Do not forget the Vary header

The moment one URL returns two different bodies depending on a request header, every cache in the path needs to know. Without it, a CDN caches whatever the first visitor got and serves markdown to a browser or HTML to an agent for the next hour. The fix is one line:

app.Use(async (context, next) =>
{
    context.Response.Headers.Vary = "Accept";
    await next();
});

There is a catch that the HN thread flagged and that I would not skip over. Vary: Accept means the cache keys on the full Accept string, and browsers send long, near-unique Accept strings. Your cache hit rate collapses because effectively every browser gets its own cache entry. On a busy CDN that is a real cost.

If you run behind Cloudflare, Fastly or an nginx edge, the sane move is to normalize Accept before it reaches the cache. Collapse the infinite variety of browser headers down to two buckets, markdown or not, and vary on a single derived header instead. In nginx that is a map and a proxy_set_header; on Cloudflare it is a small Worker. The origin logic stays the same, the cache stays useful.

The MVC way, if you already have controllers

If your API lives in controllers rather than minimal endpoints, do not hand-roll the negotiation. Write an output formatter and let the framework run the content negotiation it already does for JSON:

public sealed class MarkdownOutputFormatter : TextOutputFormatter
{
    public MarkdownOutputFormatter()
    {
        SupportedMediaTypes.Add(MediaTypeHeaderValue.Parse("text/markdown"));
        SupportedEncodings.Add(Encoding.UTF8);
    }

    protected override bool CanWriteType(Type? type) =>
        typeof(IMarkdownSerializable).IsAssignableFrom(type);

    public override Task WriteResponseBodyAsync(
        OutputFormatterWriteContext context, Encoding selectedEncoding)
    {
        var payload = (IMarkdownSerializable)context.Object!;
        return context.HttpContext.Response.WriteAsync(payload.ToMarkdown(), selectedEncoding);
    }
}

Register it, and turn on the two options that make ASP.NET Core respect the client instead of always returning its default:

builder.Services.AddControllers(options =>
{
    options.RespectBrowserAcceptHeader = true;
    options.ReturnHttpNotAcceptable = true;
    options.OutputFormatters.Insert(0, new MarkdownOutputFormatter());
});

ReturnHttpNotAcceptable gives you the correct 406 when a client insists on a media type you cannot produce, which is the honest answer rather than silently downgrading. RespectBrowserAcceptHeader stops the framework from ignoring the header for browser-shaped requests. Your action returns a model that knows how to render itself as markdown, and the framework picks HTML or markdown per request. No if statement in your controller.

The uncomfortable question: who actually sends this header?

This is where I put my senior hat on and slow down. The header approach is technically correct and I would still not bet my content strategy on it alone, because almost nothing sends Accept: text/markdown right now. Most crawlers, most agent frameworks and most "read this URL for me" tools fetch with a default Accept or none at all. Build the perfect negotiation layer and, today, close to nobody hits the markdown branch.

Which is why a chunk of the HN thread argued for sniffing the User-Agent instead: recognise GPTBot, ClaudeBot, PerplexityBot and friends, and serve them markdown regardless of Accept. It works, and I understand the pragmatism, but I would keep it as a fallback and not the foundation. User-Agent strings lie, they change without notice, and maintaining an allowlist of bot names is exactly the kind of chore that rots. The Accept header is the standards-correct signal, and it costs you nothing to honour it now so that it just works the day agents start sending it.

A middle path I like: honour Accept as the primary signal, keep a short, config-driven User-Agent allowlist as a deliberate override, and expose the markdown at a predictable alternate URL too, say /blog/my-post.md, so anything can grab the clean version without any negotiation at all. Three ways in, one source of truth. That last option is also what pairs naturally with an llms.txt file, which is the other half of this conversation: Accept handles per-request format, llms.txt and .md URLs handle discovery.

// Deliberate override, driven by config, not a hardcoded list buried in code.
static bool IsKnownAgent(HttpRequest req, AgentOptions opts)
{
    var ua = req.Headers.UserAgent.ToString();
    return opts.MarkdownAgents.Any(name => ua.Contains(name, StringComparison.OrdinalIgnoreCase));
}

What I would actually ship

Serve one canonical markdown body per page, generated from the same source your HTML renders from, so the two never drift. Decide format with a real Accept parser that respects q-values. Set Vary: Accept, then normalize Accept at the edge so your cache survives. Add a .md URL and a small, config-driven User-Agent override for the agents that have not caught up. And measure it: log which branch each request took, because if the markdown branch stays at zero percent for a quarter, you have your answer about whether to keep maintaining it.

The pitch that hit the front page is good HTTP hygiene and I am glad it is getting attention. Just build the grown-up version, not the Contains("text/markdown") version, because the difference between them is whether every browser on the internet accidentally gets served your raw markdown.

Frequently asked questions about serving markdown over content negotiation

Is text/markdown a real media type I am allowed to use? Yes, it was registered as text/markdown in RFC 7763, so it is a legitimate value to send in Accept and to set as your Content-Type. There is an optional variant parameter for the specific flavour, but for serving article bodies you rarely need it and plain text/markdown; charset=utf-8 is enough.

Why not just sniff the User-Agent and skip the Accept header entirely? Because User-Agent strings are unreliable, change without warning and force you to maintain an allowlist of bot names that goes stale. The Accept header is the standards-correct signal that costs nothing to honour today and works automatically once agents start sending it, so keep User-Agent sniffing as a deliberate override rather than your main mechanism.

Will Vary: Accept wreck my CDN cache? It can, because browsers send long and nearly unique Accept strings, so keying the cache on the raw header effectively gives every visitor their own entry. Normalize Accept at the edge into two buckets, markdown or not, and vary on that single derived header instead, which keeps the origin logic unchanged and the cache useful.

Should I return 406 when I cannot produce the requested type? Yes, if a client sends an Accept you genuinely cannot satisfy, a 406 Not Acceptable is the honest response, and in ASP.NET Core you enable it with ReturnHttpNotAcceptable = true. For browsers this rarely triggers because they send */*, which any representation satisfies, so the 406 only shows up for clients that ask for something specific and impossible.

How does this relate to llms.txt? They solve different halves of the same problem: Accept-based negotiation decides the format of a page you already know the URL of, while llms.txt and predictable .md URLs help agents discover and fetch your clean content in the first place. Shipping both a markdown representation and a discovery mechanism covers more real-world clients than either alone.

Further reading

• Protobuf finally has a language server. What contract-first gRPC in .NET gains
• The valley of webhooks: what to build when delivery is at-least-once
• Code was never the easy part: why programming craftsmanship still matters in the AI age

---

Conclusion: Treat "serve markdown to agents" as a content-negotiation feature, not a party trick. Parse Accept with q-values, set and then normalize Vary, add a .md URL as a fallback, and measure how often the markdown branch actually fires before you invest more in it.

Sources: Serve Markdown to AI Agents with Accept Headers · Hacker News discussion.