Author gave me the same reply. I just don't want to have to think about whether it's equivalent or not, why use a 2 param model that's strictly less appropriate even if the difference is small.

Thanks, indeed I missed this: "here's what ChatGPT suggests:"

SWE-bench scores are inflated compared against actual maintainer merge decisions as opposed to an LLM grader.

Benchmarks do a few things: 1. Help choose a model from the hundreds out there, or at least help create a shortlist to try. 2. Quantify progress/improvements (or lack thereof) over time. 3. Inform about relative strengths and weaknesses.

Figure 1 should not fit a straight line as a trend. Scores are 0 to 100%, the straight line will go outside those bounds at sufficiently large times.

The simplest reasonable model would be logistic regression. It's also got 2 parameters and the range is correct.