[chbot] Software and system safety / Therac25 (Re: Drone Delivery Service)
helmut.walle at gmail.com
Tue Oct 15 01:45:49 BST 2013
On 10/15/2013 10:57 AM, Charles Manning wrote:
> An extreme case of that is Therac25 which had a software bug during
> the first 2 years of operation. This caused 6 known cases of overdose
> which caused 3 deaths. Of course nobody writes about the thousands on
> lives that the machines saved during that time because that is not
> newsworthy. Did the company get lauded for saving thousands of lives?
> No, they got taken to the cleaners for the three deaths.
> Some people would argue that the Therac25 should have been better
> tested. Maybe. But it is worth considering that the machines were
> saving dozens of lives a day and delaying the release would have
> caused more deaths. Applying the "common good" principle, any delays
> in the release should be avoided.
Safety-critical systems need to be developed as such. There are key
concepts, standards and frameworks that can, and in many cases must, be
followed. Based on many costly and sometimes fatal lessons learnt, a lot
of work has gone into developing suitable development practices to make
all kinds of products safer. For quite a while now, there have been
system and software safety regulations (i.e. effective laws in the legal
sense) that apply to various kinds of products, including aircraft,
cars, medical equipment (like the Therac, but also wheelchairs, hospital
beds, drugs, etc.), whiteware and many other things.
If you look at the Therac-25 case in more detail, you will find that
these incidents where patients were burnt and irradiated badly, should
never have happened, and if the system had been developed under one of
today's mandatory frameworks, they would never have happened. Today's
technical systems are more complex and have more software than
yesterday's. And so far, we haven't come up with any method to guarantee
that a piece of software is free of bugs. But we have learnt a whole lot
about neat development processes and practices, about QA, risk analysis,
etc. And that is what puts us into quite a good position to develop
complex systems to a high safety standard. Hindsight is a wonderful thing...
Unfortunately, sometimes safety regulations aren't followed fully for
cost or time reasons, and bad things do still happen. But at least there
is a lot of formal responsibility that the makers of potentially
hazardous equipment have to accept by law. Good manufacturers take this
responsibility very seriously.
More information about the Chchrobotics