[chbot] Software and system safety / Therac25 (Re: Drone Delivery Service)

[Helmut Walle]

On 10/15/2013 10:57 AM, Charles Manning wrote:
> [...]
> An extreme case of that is Therac25 which had a software bug during 
> the first 2 years of operation. This caused 6 known cases of overdose 
> which caused 3 deaths. Of course nobody writes about the thousands on 
> lives that the machines saved during that time because that is not 
> newsworthy. Did the company get lauded for saving thousands of lives? 
> No, they got taken to the cleaners for the three deaths.
>
> Some people would argue that the Therac25 should have been better 
> tested. Maybe. But it is worth considering that the machines were 
> saving dozens of lives a day and delaying the release would have 
> caused more deaths. Applying the "common good" principle, any delays 
> in the release should be avoided.
> [...]

Safety-critical systems need to be developed as such. There are key 
concepts, standards and frameworks that can, and in many cases must, be 
followed. Based on many costly and sometimes fatal lessons learnt, a lot 
of work has gone into developing suitable development practices to make 
all kinds of products safer. For quite a while now, there have been 
system and software safety regulations (i.e. effective laws in the legal 
sense) that apply to various kinds of products, including aircraft, 
cars, medical equipment (like the Therac, but also wheelchairs, hospital 
beds, drugs, etc.), whiteware and many other things.

If you look at the Therac-25 case in more detail, you will find that 
these incidents where patients were burnt and irradiated badly, should 
never have happened, and if the system had been developed under one of 
today's mandatory frameworks, they would never have happened. Today's 
technical systems are more complex and have more software than 
yesterday's. And so far, we haven't come up with any method to guarantee 
that a piece of software is free of bugs. But we have learnt a whole lot 
about neat development processes and practices, about QA, risk analysis, 
etc. And that is what puts us into quite a good position to develop 
complex systems to a high safety standard. Hindsight is a wonderful thing...

Unfortunately, sometimes safety regulations aren't followed fully for 
cost or time reasons, and bad things do still happen. But at least there 
is a lot of formal responsibility that the makers of potentially 
hazardous equipment have to accept by law. Good manufacturers take this 
responsibility very seriously.

Kind regards,

Helmut.