There's a distinction worth making here. A/B testing the interface button placement, hue of a UI element, title styling — is one thing. But you wouldn't accept Photoshop silently changing your #000000 to #333333 in the actual file. That's your output, not the UI around it. That's what LLMs do. The randomness isn't in the wrapper, it's in the result you take away.

This is very different from the A/B interface testing you're referring to, what LLMs enable is A/B testing the tool's own output — same input, different result.

Your compiler doesn't do that. Your keyboard doesn't do that. The randomness is inside the tool itself, not around it. That's a fundamental reliability problem for any professional context where you need to know that input X produces output X, every time.

A professional tool is something that provides reliable and replicable results, LLMs offer none of this, and A/B testing is just further proof.

My point is not what is learned but how. Debugging, reading errors, understanding how things put together — that is the learning.

You haven't replaced assembler with C here, you've replaced programming with Scrabble.