It's probably still true, though, says formal methods expert
"Software research is a train wreck," says Hillel Wayne, a Chicago-based software consultant who specialises in formal methods, instancing the received wisdom that bugs are way more expensive to fix once software is deployed.
Wayne did some research, noting that "if you Google 'cost of a software bug' you will get tons of articles that say 'bugs found in requirements are 100x cheaper than bugs found in implementations.' They all use this chart from the 'IBM Systems Sciences Institute'... There's one tiny problem with the IBM Systems Sciences Institute study: it doesn't exist."
Laurent Bossavit, an Agile methodology expert and technical advisor at software consultancy CodeWorks in Paris, has dedicated some time to this matter, and has a post on GitHub called "Degrees of intellectual dishonesty". Bossavit referenced a successful 1987 book by Roger S Pressman called Software Engineering: a Practitioner's Approach, which states: "To illustrate the cost impact of early error detection, we consider a series of relative costs that are based on actual cost data collected for large software projects [IBM81]."
The reference to [IBM81] notes that the information comes from "course notes" at the IBM Systems Sciences Institute. Bossavit discovered, though, that many other publications have referenced Pressman's book as the authoritative source for this research, disguising its tentative nature.
Bossavit took the time to investigate the existence of the IBM Systems Science Institute, concluding that it was "an internal training program for employees." No data was available to support the figures in the chart, which shows a neat 100x the cost of fixing a bug once software is in maintenance. "The original project data, if any exist, are not more recent than 1981, and probably older; and could be as old as 1967," said Bossavit, who also described "wanting to crawl into a hole when I encounter bullshit masquerading as empirical support for a claim, such as 'defects cost more to fix the later you fix them'."
Do software defects cost more to fix, the later they are discovered? "I think the body of research so far tentatively points in that direction, depending on how you interpret 'late-stage', 'bugs', and 'more expensive', said Wayne. "Certain bugs take more time to fix (and cause more damage) than others, and said bugs tend to be issues in the design."
Here is a 2016 paper [PDF] whose authors "examined 171 software projects conducted between 2006 and 2014," all of which used a methodology called the Team Software Process. The researchers concluded that "the times to resolve issues at different times were usually not significantly different."
Wayne is as concerned with the state of software research as with the defect question itself. He observed that papers such as the one cited above "use different definitions of defect," making it hard to draw conclusions. He said he is a proponent of Empirical Software Engineering (ESE), using evidence to learn about what works in software development, but said that "the academic incentive structures are not aligned in a way that would give industry actionable information. There's much more incentive to create new models and introduce new innovations than do the necessary 'gruntwork' that would be most useful."
He suggested focusing on what "empirical research overwhelmingly shows," which is that "code review is a good way to find software bugs and spread software knowledge. It also shows that shorter iteration cycles and feedback loops lead to higher quality software than long lead times."
The role of the "IBM Systems Sciences Institute" in cementing the authority of research that might not exist is a reminder of the importance of primary sources, which can be hard to discover in the echo chamber of the internet.
Right on cue, into our inbox popped a bit of "research" from a PR agency concerning the revenue of the the top five cloud vendors, based on a "Statista survey". Statista is not primarily a research company, however. Instead it "consolidates statistical data on over 80,000 topics from more than 22,500 sources," according to its own description.
The research mentioned did not come from Statista, but from Cloud Wars. Citing Statista as the source is like attributing a statement discovered in a Google search as having the authority of Google. The risk of confusion like this is that a poor source can be promoted to a better one (and that is not intended to suggest that the Cloud Wars data is inaccurate). ®
Read the original article
Comments
I work in HW industry, but doing SW.
If I deploy a bug despite "unit" tests, then it will probably be caught by 2nd tier tests 1 day later, which will already be more expensive since I'll need to read report, context switch from other thing, fix it, redeploy, wait for CI, wait for 2nd tier tests, etc.
If it will not be caught by 2nd tier, then it probably will be caught by validation engineers, which not only costs their time, but they also have to contact me / fill some bug, I need to get familiar with it, context switch again, fix it, redeploy, wait for tests #1 and #2 and wait for their confirmation
If it will be released to actual customers then there's reputational dmg (can be huge depending on impact), possibly some lost sales / customers, probably a few weeks before it goes back to me thru customer->???->validation->me
So, while 100x more expensive is some extreme case, then usually the cost is very significantly higher the later you find the bug.
But what about HW bugs?
I think those can be really, really expensive.
Imagine catching hardware bug which affects some computation that MUST be fixed at hardware level.
Catching it before release and catching it after selling 100m copies is tens of bilions of dollars difference for top GPUs and CPUs.
Why do you think that those companies are willing to build same product twice so they test eachother? It significantly increases the cost, but reduces the risk.
Think about CrowdStrike's incident: catching that bug at dev level would cost them shitton times less than what actually happened :P
Their stock dropped by like 45%
> So, while 100x more expensive is some extreme case, then usually the cost is very significantly higher the later you find the bug.
When you said the reputational damage, lost sales/customers.. you're already in 100x territory. It may not be exactly there but it's close or exceeding it.
Yet we'll still get the undisciplined engineers whining that they have to write a small and isolated unit test.
By cwsx 2025-06-022:51 > Yet we'll still get the undisciplined engineers whining that they have to write a small and isolated unit test.
Oh I wish that were true for my experiences - much more commonly it's a project manager that doesn't understand the value of tests....
By Dylan16807 2025-06-0123:411 reply > Think about CrowdStrike's incident: catching that bug at dev level would cost them shitton times less than what actually happened :P
> Their stock dropped by like 45%
A program that can block booting is definitely on the high side of software bug risk.
But even in that case, the post-drop stock was still trading higher than a year before and they got back to the same level in six months.
And in most corporate mindsets that's six months of growth off the table, six months of fighting just to get back to normal - that's a huge real cost and a huge opportunity cost.
By Dylan16807 2025-06-021:53 There was still plenty of growth to today. I doubt the stock is much lower than it would have been if that event never happened. Mild real cost, mild opportunity cost.
By mixmastamyk 2025-06-024:03 Yes, am reminded of the Pentium FDIV bug:
https://en.wikipedia.org/wiki/Pentium_FDIV_bug
$475 million charge, in 1990's dollars.
Every additional person who gets involved who didn't author the bug needs to research and communicate the problem. 100x sounds small after just a few layers of indirection.
By aaron695 2025-06-021:02 [dead]
I have some thoughts on this (in the context of modern SaaS companies).
The most expensive parts of fixing a bug are discovering/diagnosing/triaging the bug, cleaning up corrupted records, and customer communication. If you discover a bug in development or even better while you are coding the function or during a code review you get to bypass triaging, customer calls, escalations, RCAs, etc. At a SaaS company with enterprise customers each of those steps involves multiple meetings with your Support, Account Manager, Senior Engineer, Product Manager, Engineering Manager, Department Manager, sometimes Legal or a Security Engineer and then finally the actual coder. So of course if you can resolve an issue (at a modern SaaS company) during development it can be 10-100x less expensive just because of how much bureaucracy is involved in running a large scale enterprise SaaS company.
It also brings up the interesting side effect of companies adopting non-deterministic coding (AI Code) in that now bugs that could have been discovered during design/development by a human engineer while writing the code can now leak all the way into prod.
By ziggure 2025-06-022:55 The bureaucracy involved is usually the biggest cost driver. Another is the refactoring needed once mode code has been built atop the buggy code.
By janice1999 2025-06-0121:571 reply If you ship firmware to devices, it could be far more expensive. [1]
[1] https://www.bleepingcomputer.com/news/hardware/botched-firmw...
By mrheosuper 2025-06-022:53 We used to have a bug in our FW that was caught quite late. We have to unbox thousands of products, connecting it to our phone to download new FW, then re-package it. Not fun at all.
