Self-hosting exposes your sloppy prompts: context budgets for .NET agents

A write-up hit the Hacker News front page this week that I recognised without having lived it. Patrick McCanna describes moving 35KB of carefully assembled preprompts off Anthropic and OpenAI onto a self-hosted model running on Ollama. On paper, a lift-and-shift. In practice, the agent fell apart within minutes. It re-read files it had just read, fired the same tool call twice in a row, and kept restating its own objective like someone talking out loud to find the thread they had lost.

The HN thread sprawled in every direction, but the core is simple. That 35KB ate 14 percent of his local 65K-token context window straight away. On a frontier model with a 200K window and a mountain of hidden chain-of-thought, a prompt like that never shows up. Run it on a smaller model and you suddenly see what you were paying all along. Not in dollars. In room to think.

I work in .NET a lot, and I keep building more agent-shaped features in C#. So I did not read this as "Ollama is fiddly". I read it as a mirror. Your prompt was always too fat. Your frontier model was just hiding it.

Why a bigger window moves the problem, it does not fix it

There is a stubborn idea that a bigger context window means you get to worry less about what you put in it. The opposite is true. A model does not pay equal attention to every token in its window. The fuller you make it with background, rules and examples that do not touch this particular request, the more you dilute attention on the few hundred tokens that actually matter.

McCanna's agent showed the sharp version of that problem. The moment the model added its own output, plus the tool results, plus the files it opened, the window filled up before the work was done. And a full window does not behave like a slightly slower agent. It behaves like someone who forgets, halfway through a sentence, where they started.

That is what makes his conclusion useful. He did not fix it with a bigger model. He fixed it by asking for less per step.

Measure your prompt before you send it

The first step in .NET is dull and most people still skip it: count your tokens. Not your characters, your tokens. A 35KB system prompt feels small on disk and is not small in a window.

With Microsoft.ML.Tokenizers you do this without calling out to an external service:

using Microsoft.ML.Tokenizers;

// Use the tokenizer that matches your target model. For Llama-family
// models load the model-specific vocab; cl100k shown here as an example.
var tokenizer = TiktokenTokenizer.CreateForModel("gpt-4");

string systemPrompt = await File.ReadAllTextAsync("prompts/agent-system.md");
int tokens = tokenizer.CountTokens(systemPrompt);

Console.WriteLine($"System prompt: {tokens} tokens");

const int windowSize = 65_536;
double share = (double)tokens / windowSize;
if (share > 0.10)
    Console.WriteLine($"Warning: prompt takes {share:P0} of the window before the first turn.");

Put this in a test that fails the moment your prompt crosses a threshold. I set mine at ten percent. If a pull request pushes past it, I want a reason, not the silent growth of someone tacking on one more paragraph "to be safe". Prompts grow exactly the way log files grow: one line at a time, until nobody remembers why any of them are there.

Set your context window explicitly in Ollama

One catch McCanna names will bite a lot of .NET developers: Ollama defaults to a small context window, often 2048 or 4096 tokens, regardless of what the model can handle. Leave it unset and the runtime quietly truncates your oldest context. Your agent looks like it is "forgetting", when in reality you never turned the room on.

Through Microsoft.Extensions.AI you talk to Ollama behind the same IChatClient abstraction you use for OpenAI. The num_ctx field goes in the additional options:

using Microsoft.Extensions.AI;

IChatClient client = new OllamaChatClient(
    new Uri("http://localhost:11434"),
    modelId: "llama3.1:8b");

var options = new ChatOptions
{
    Temperature = 0.2f,
    AdditionalProperties = new()
    {
        // Without this line Ollama truncates at its 4096 default.
        ["num_ctx"] = 32_768,
    },
};

var response = await client.GetResponseAsync(
    "Summarise the pending migrations in this repo.", options);

Console.WriteLine(response.Text);

A larger num_ctx costs memory on the machine running the model, so this is a real trade-off and not a free switch. But leaving it at the default while you send a 20K-token prompt is not thrift, it is a bug. Measure your prompt first, then pick your window, then leave enough room for the answer and the tool results.

Split the monolith into single-objective agents

The most portable part of McCanna's story is what he calls single objective prompting. Instead of one system prompt that knew everything, he breaks the work into separate units, each with its own small, sharp instruction. This is not a trick for local models. It is just better design, and a small model forces you into it because it stops hiding how expensive the monolith was.

In .NET this drops into place nicely. Define an agent per task with exactly the context that task needs:

public sealed record AgentDefinition(
    string Name,
    string Objective,
    string SystemPrompt,
    IReadOnlyList<string> AllowedTools);

// Each definition is small. Together they are large; alone each request is lean.
var migrationAuditor = new AgentDefinition(
    Name: "migration-auditor",
    Objective: "Find EF Core migrations not yet applied in production.",
    SystemPrompt: """
        You inspect a .NET repo. Return only migration names that exist
        in the code but not in the supplied __EFMigrationsHistory.
        Answer as a JSON array of strings. No explanation.
        """,
    AllowedTools: ["read_file", "list_migrations"]);

Notice what the system prompt does not do. It does not explain general coding standards, company history, or twelve edge cases this task never touches. All of that knowledge lives in other definitions, which only load when their task comes up. That is the difference between 35KB per turn and maybe 2KB per turn.

Positive instructions cost less and fail less

A detail that is easy to gloss over but worth copying: McCanna replaced prohibitions with directions. Not "do not use reflection and do not touch the database and do not change the config", but "limit yourself to these two files and return a diff".

There are two wins in that. The first is tokens. A long list of things-you-must-not-do is usually longer than the short description of what you may do. The second is reliability. Smaller models miss a negation more often. "Do not do X" sometimes slips, and then the model does exactly X. Say only what is allowed and there is no negation to miss.

I apply the same rule to my tool definitions. Give an agent three tools that can only do what is permitted, and you never have to write that it should stay away from the rest.

Catch thrashing in code, not in the prompt

The most expensive failure mode in the story was thrashing: the same tool call twice in a row, the same file read over and over. You can ask a model not to do that, but on a small window asking is weak. A cheap guard in your own code is stronger.

Because you orchestrate the tool calls yourself in .NET, you can keep a short call history and short-circuit an identical repeat before it goes back to the model:

public sealed class ToolCallGuard
{
    private readonly Queue<string> _recent = new();
    private const int Window = 4;

    public bool IsRepeat(string tool, string argsJson)
    {
        var key = $"{tool}:{argsJson}";
        if (_recent.Contains(key))
            return true;

        _recent.Enqueue(key);
        if (_recent.Count > Window)
            _recent.Dequeue();

        return false;
    }
}

If a call comes back that already ran in the last few steps, you do not hand the model the same result again. You return a short note that it already has this, and you nudge it to the next step. That breaks the loop that otherwise eats your whole session, or in McCanna's words, "like a burst pipe".

Persist the session state to disk

The last piece is operational. On a small window you cannot avoid handoffs: at some point you have to compress context or pass it to a fresh turn. McCanna writes session state to disk so an agent can stop and resume without rebuilding everything.

In .NET that is a few lines, and it gives you an audit trail of what the agent knew at each step for free:

public sealed record SessionState(
    string AgentName,
    string Objective,
    List<ChatMessage> History,
    Dictionary<string, string> Scratchpad);

// Save after each meaningful step; resume by reading it back.
var json = JsonSerializer.Serialize(state, _jsonOptions);
await File.WriteAllTextAsync($"sessions/{sessionId}.json", json);

This is not exciting, which is exactly why you skip it until it burns you once. An agent that crashes on step nine and has to start from scratch is not just slow. It burns your entire context budget a second time on work that was already done.

What I take from this into .NET work

I am not moving my whole stack to Ollama tomorrow, and that is not the point. The point is that a small, self-hosted model is an honest yardstick. It does not let you get away with a bloated prompt, with vague prohibitions, or with an agent spinning in circles, because there is no comfortable margin left to hide those costs in.

So count your tokens and put a test on it. Choose your context window on purpose instead of trusting the default. Split your monolith into single-objective agents. Write what is allowed instead of what is not. And catch thrashing and handoffs in your own C#, where you have control, instead of begging for them in a system prompt. Do that and your frontier model gets cheaper and sharper too. You just stop paying for the mess it used to sweep under the rug for you.

Frequently asked questions about prompts and self-hosted LLMs

Why does my agent on Ollama forget things that stayed put on OpenAI? Most likely because Ollama defaults to a small context window, often 2048 or 4096 tokens, and quietly truncates your oldest context once you exceed it. Set num_ctx explicitly to a value that fits your prompt plus the expected answer, and remember that a larger window costs more memory on the machine running the model.

How do I count the tokens in my system prompt in .NET? Use the Microsoft.ML.Tokenizers package and call CountTokens on the text with the tokenizer that matches your target model. Do this inside a unit test that fails the moment the prompt crosses a threshold, say ten percent of your window, so prompt growth shows up in code review instead of only in production.

What is single objective prompting and why does it help? It means splitting the work into separate tasks, each with its own small system prompt and only the tools that task needs, instead of one monolith that has to know everything. It keeps each individual request lean, which holds the model's attention on the few hundred tokens that matter and reduces the chance of it wandering off.

Can I use the same .NET code against both Ollama and OpenAI? Yes, Microsoft.Extensions.AI offers an IChatClient abstraction that both the Ollama and the OpenAI client sit behind, so your application code stays the same and you swap the client and the options. Model-specific settings such as num_ctx go through AdditionalProperties, so the rest of your pipeline never notices.

Is thrashing better solved in the prompt or in code? In code. On a small window an instruction not to repeat is weak, while a short guard that recognises identical tool calls in the last few steps and short-circuits them works reliably. Because you own the tool orchestration in .NET, you can break the loop before it eats your context budget.

Further reading

• Context Rot: Why a Bigger Context Window Won't Save Your LLM Feature
• Open-Weight AI's Kubernetes Moment: What It Means for .NET Teams
• Serving markdown to AI agents with Accept headers: the .NET version

---

Conclusion: Treat a small, self-hosted model as a yardstick, not a handicap. It shows you exactly which tokens your prompt wastes, and that lesson makes your frontier agent cheaper and sharper too.

Sources: Notes on migrating large preprompts to self-hosted LLMs · Hacker News discussion.