My mildly amusing anecdote is that, whenever Claude Code produces something particularly egregious, I often find it sufficient to reply with just "wtf?" for it to present a much more sensible version of the code (which often needs further refinement, but that's another story...)

But we don't evolve IL or assembly code as the system evolves. We regenerate it from scratch every time.

It is therefore not important whether some intermediate version of that low-level code was completely impossible to understand.

It is not so with LLM-written high-level code. More often than not, it does need to be understood and maintained by someone or something.

These days, I mainly focus on two things in LLM code reviews:

1. Making sure unit tests have good coverage of expected behaviours.

2. Making sure the model is making sound architectural decisions, to avoid accumulating tech debt that'll need to be paid back later. It's very hard to check this with unit tests.

Part of it observability bias: longer, more widespread outages are more likely to draw signficant attention. This doesn't mean that there aren't also shorter, smaller-scope outages, it's just that we're much less likely to know about them.

For example, if there's a problem that gets caught at the 1% stage of a staged rollout, we're probably not going to find ourselves discussing it on HN.

This thread prompted me to look into this. It seems that all I need is a thin wrapper around whisper-ctranslate2. So I wrote one and am playing with it right now.

I'm finding language auto-detection to be a bit wonky (for example, it repeatedly identified Ladykracher audio as English instead of German). I ended up having to force a language instead. The only show in my library where this approach doesn't work is Parlement[1], but I can live with that.

On the whole this is looking quite promising. Thanks for the idea.

[1] https://en.wikipedia.org/wiki/Parlement_(TV_series)

And how is it going, in terms of finding those limit? It would be very interesting to hear about areas where the actual experience turned out to be wildly different from your expectations, in either direction.