AI Resolves Your Incidents and Your Engineers Forget the System

This week a piece by Sylvain Kalache climbed high on Hacker News, under a title that leaves little room for spin: AI handles the incidents and engineers lose touch with their systems. The thread underneath filled up with people who recognised themselves in it. Not because AI-SREs do bad work, but because they do good work. And that is exactly where the trouble starts.

I have run production incidents at .NET clients for years. On-call, warm handoffs at 3am, the creeping panic when a dependency you never flagged as critical falls over. My first reaction to the piece was not resistance. It was recognition. Because what Kalache describes is not a new AI problem. It is a problem from 1983 wearing new clothes.

What the thread is really about

Kalache works at Rootly, an incident management platform, so read his piece with that hat on. His core point survives the sales pitch, and it is correct. AI assistants get good at the routine work: the known alert, the known rollback, the disk that fills up and gets cleaned out. That drives down your mean time to recovery. Great.

But that routine work was never only work. It was also practice. Every time a junior picks up a pager alert and digs through the traces, that person builds a mental model of how the system actually behaves under pressure. Take those moments away and you are left with engineers who are responsible for the system on paper, but who have not had their hands in it for months. Right up until the AI runs out of answers. And that moment always comes, because the rare, weird incidents are by definition the ones with no pattern to match.

The irony of automation, forty years old

Lisanne Bainbridge wrote a short paper in 1983 called Ironies of Automation. It is about process control in factories, not Kubernetes, and every word still holds. Her argument: the more you automate, the less the operator practices the normal work, while you still hold them responsible for the abnormal. Automation takes the easy cases. The human keeps the hard cases, and has less and less routine to handle those hard cases with.

Aviation has wrestled with this for decades. Pilots deliberately train on scenarios that almost never happen, precisely because the autopilot does the rest. They keep a hand on the stick, literally, to keep the skill warm. Software teams barely do this. We build the automation, switch it on, and assume we still understand the underlying system because we built it once. That is an assumption with an expiry date.

Why this hits .NET teams harder than the graph shows

Your MTTR dashboard does not lie, but it does not tell the whole story either. The average drops because the many small incidents close faster. The few heavy incidents, the kind of SEV1 that takes your business down for a day, actually get slower to resolve, because the engineer facing them has not looked at the logs themselves in six months.

For a typical .NET estate that is more concrete than it sounds. Think of a connection pool exhausting under a thread starvation bug. An async deadlock that only shows up under production load. An EF Core query that stops using an index after a migration and slowly strangles your database. These are not things you fix with a generic runbook. You have to know where the bodies are buried. You have to know this system.

My point is not to switch off the AI. That would be foolish. My point is that you have to deliberately carve out space where humans are still on the controls, even when the machine is faster.

Put the AI in the proposer seat, not the executor seat

The simplest change I roll out with teams: let the AI propose diagnoses and remediations, but keep a human in the approval loop for anything that mutates state. Not as bureaucracy, but as a learning moment. The engineer who presses the button reads the reasoning, checks it, and learns.

Here is a stripped-down example in C#. A remediation action is an explicit object with a risk level. Low risk the agent may run itself. Anything above that waits for a human.

public enum RemediationRisk { Low, Medium, High }

public sealed record RemediationAction(
    string Name,
    RemediationRisk Risk,
    Func<CancellationToken, Task> Execute,
    string Rationale);

public sealed class ApprovalGate
{
    private readonly IApprovalChannel _channel; // Slack, Teams, whatever
    private readonly ILogger<ApprovalGate> _log;

    public ApprovalGate(IApprovalChannel channel, ILogger<ApprovalGate> log)
        => (_channel, _log) = (channel, log);

    public async Task RunAsync(RemediationAction action, CancellationToken ct)
    {
        if (action.Risk == RemediationRisk.Low)
        {
            _log.LogInformation("Auto-running low-risk action {Name}", action.Name);
            await action.Execute(ct);
            return;
        }

        var decision = await _channel.RequestApprovalAsync(
            title: $"Approve remediation: {action.Name}",
            body: action.Rationale,
            ct: ct);

        if (!decision.Approved)
        {
            _log.LogWarning("Action {Name} rejected by {User}", action.Name, decision.User);
            return;
        }

        _log.LogInformation("Action {Name} approved by {User}", action.Name, decision.User);
        await action.Execute(ct);
    }
}

The value is not in the code, which is trivial. The value is in the behaviour. Every approval forces a human to read what is about to happen and why. That is a minute of practice you would otherwise have thrown away.

Practice is not a luxury, it is a build step

In aviation they call it a sim session. We call it a game day, and most teams I see schedule two a year and cancel both. That does not work. Practice has to be so cheap and so routine that you stop cancelling it.

One approach that works well: build fault injection in as something you can turn on for a staging or even a production canary. You do not need a separate chaos platform to start. A middleware that reacts to a header is enough to let your team rehearse real failure modes.

public sealed class FaultInjectionMiddleware
{
    private readonly RequestDelegate _next;
    private readonly IFaultConfig _config; // in-memory, feature-flagged

    public FaultInjectionMiddleware(RequestDelegate next, IFaultConfig config)
        => (_next, _config) = (next, config);

    public async Task InvokeAsync(HttpContext ctx)
    {
        if (_config.Enabled && ctx.Request.Headers.ContainsKey("X-Chaos"))
        {
            var mode = ctx.Request.Headers["X-Chaos"].ToString();
            switch (mode)
            {
                case "latency":
                    await Task.Delay(_config.LatencyMs, ctx.RequestAborted);
                    break;
                case "500":
                    ctx.Response.StatusCode = StatusCodes.Status500InternalServerError;
                    await ctx.Response.WriteAsync("Injected fault");
                    return;
                case "dbdown":
                    throw new InvalidOperationException("Injected: database unavailable");
            }
        }

        await _next(ctx);
    }
}

Put this behind a feature flag and a network restriction, so only your own team can set the header. Then run a game day on it: someone quietly flips X-Chaos: dbdown on the canary, and the on-call engineer has to work it out without knowing what is going on. The first time it takes painfully long. The third time your team has a reflex. That reflex is exactly what you lose when the AI cleans up everything for you.

One warning I make loudly: never inject faults that can corrupt real customer data, and never without a kill switch that reverts everything instantly. Chaos without blast-radius control is not practice, it is an incident you caused yourself.

Keep your telemetry readable for humans

A second problem sneaks in once the AI reads the logs and you no longer do. Teams start optimising their telemetry for the machine. Cardinality up, structure gone, everything an event. Fine for a model, useless for a human trying to reconstruct a story at 3am.

I keep traces deliberately readable. Span names that tell a story, attributes a human understands, and a handful of high-signal events instead of a swamp. OpenTelemetry in .NET makes that easy.

using System.Diagnostics;

public static class Telemetry
{
    public static readonly ActivitySource Source = new("Orders.Checkout");
}

public async Task<OrderResult> CheckoutAsync(Cart cart, CancellationToken ct)
{
    using var activity = Telemetry.Source.StartActivity("checkout");
    activity?.SetTag("cart.items", cart.Items.Count);
    activity?.SetTag("cart.value_eur", cart.TotalEur);

    try
    {
        var payment = await _payments.ChargeAsync(cart, ct);
        activity?.SetTag("payment.provider", payment.Provider);
        activity?.AddEvent(new ActivityEvent("payment.captured"));
        return OrderResult.Ok(payment);
    }
    catch (PaymentDeclinedException ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, "payment declined");
        activity?.AddEvent(new ActivityEvent("payment.declined",
            tags: new ActivityTagsCollection { { "reason", ex.Reason } }));
        throw;
    }
}

The idea is not that the AI cannot handle dense data. It can. The idea is that a human who has fallen out of routine can pull the story out of this in ten seconds. Readable telemetry is the cheapest insurance against skill decay there is, because it lowers the threshold to go look yourself.

What I actually do in practice

Concretely, at my clients, it comes down to a few things. The AI does all diagnosis and all read-only analysis. Any action that mutates state above low risk goes past a human, and that human actually reads the reasoning. We run a short game day every sprint, not half a day but forty minutes, with real fault injection on a canary. And we rotate on-call so juniors resolve the known incidents themselves, even though the AI would do it faster, because that is their practice.

Does it cost time? Yes. Does it feel like a detour sometimes? Also yes. But the bill for skill decay does not arrive in the sprint where you run it up. It arrives on the day your heaviest incident was not in your model's training set, and the only person who can fix it has not looked at the logs themselves in six months.

Frequently asked questions about AI incident handling and skill decay

So should I just not use AI for incidents? No, use it heavily for diagnosis and the routine work, because that is where it saves time. Just build in deliberate moments where humans stay on the controls, so the skill does not fade.

What exactly is the irony of automation? It is Lisanne Bainbridge's 1983 point that automation takes over the easy tasks and leaves the human the hard, rare cases, while that human is ever less prepared for those cases through lack of routine.

Why does this hit .NET teams specifically? Because the heavy incidents in .NET production are often system-specific, such as thread starvation, async deadlocks or an EF Core query that drops an index after a migration, and you do not solve those with a generic runbook but with knowledge of this system.

How often should I run a game day? Short and frequent beats long and rare. Forty minutes per sprint with real fault injection on a canary keeps the reflex warm, whereas the classic half-yearly session usually gets cancelled.

Is fault injection in production not dangerous? Only if you do it without a safety net. Put it behind a feature flag and a network restriction, never touch real customer data, and keep a kill switch that reverts everything instantly.

Further reading

• OTel Is Wobbling at the Top of Hacker News. What Does That Mean for Your .NET Telemetry?
• The security embargo is dead: patching in the age of AI-generated exploits
• Open-Weight AI's Kubernetes Moment: What It Means for .NET Teams

---

Conclusion: Let AI take the routine incidents, but treat practice as a build step and not a luxury, or on your heaviest day the only person who can fix it will be out of practice.

Sources: Sylvain Kalache on AI and incidents - Hacker News discussion.