Its a 2 day project at best to create your own bespoke llm as judge e2e eval framework. Thats what we did. Works fine. Not great. Still need someone to write the evals though.

Vehement disagree. We implemented our own context editing features 4 months back. Claude released a very similar featureset we had all along last month. We were still glad we did it because (A) it took me half a day to do that work (B) our solution is still more powerful for our use case (C) our solution works on other models as well.

It all comes down to trying to predict what will be your vendors' roadmap (or if youre savvy, get a peek into it) and whether the feature you want to create is fundamental to your applications behavior (I doubt encryption is unless youre a storage company).

Anthropic doesnt even allow temperature changes when you turn thinking on.

Sure, but its an instruction that applies and the model will consider fairly relevant in every single token. As an extremely example imagine instructing the llm to not use the letter E or to output only in French. Not as extreme but it probably does affect.

By pure nature of how companies work, a space company with this mandate and so much funding, unless its being used for money laundering, will have a modicum of progress. BO has barely had that. No space company with so much money and so much runway has achieved so little.