[chbot] Software and system safety / Therac25 (Re: Drone Delivery Service)

[Charles Manning]

In general I would completely agree with your position with regards safety
critical products, but it is interesting to note that  it is not always a
rational way of looking at the issue. Please don't get me wong here, I am
not at all suggesting we have a careless attitude to designing safety
critical products.

If we have a life saving product that will save 1000 people per year, but
it has a bug that will kill 5 people per year then the cautionary approach
would be to delay release until we have ironed out the bug that kills 5
people. Meanwhile we are not saving those other 1000 people per year, so by
withholding the product we are letting people die. Such delays in the name
of safety can actually cause more problems than they solve.

We see a similar thing happening after Sept 11th when planes were grounded
in the interests of public safety and later when people chose not to fly.
This lead to more car usage and approx 1500 more car deaths in USA than in
the previous or next year.

We get worried about electronics & software failing in a braking system and
such, but are less worried about mechanical failures (eg. broken cables)
which are far more common.

As a species we have a very irrational way of looking at risk, and, back to
my main point, lawyers exploit that irrationality to make it really hard to
release new products.



On Tue, Oct 15, 2013 at 1:45 PM, Helmut Walle <helmut.walle at gmail.com>wrote:

> On 10/15/2013 10:57 AM, Charles Manning wrote:
>
>> [...]
>> An extreme case of that is Therac25 which had a software bug during the
>> first 2 years of operation. This caused 6 known cases of overdose which
>> caused 3 deaths. Of course nobody writes about the thousands on lives that
>> the machines saved during that time because that is not newsworthy. Did the
>> company get lauded for saving thousands of lives? No, they got taken to the
>> cleaners for the three deaths.
>>
>> Some people would argue that the Therac25 should have been better tested.
>> Maybe. But it is worth considering that the machines were saving dozens of
>> lives a day and delaying the release would have caused more deaths.
>> Applying the "common good" principle, any delays in the release should be
>> avoided.
>> [...]
>>
>
> Safety-critical systems need to be developed as such. There are key
> concepts, standards and frameworks that can, and in many cases must, be
> followed. Based on many costly and sometimes fatal lessons learnt, a lot of
> work has gone into developing suitable development practices to make all
> kinds of products safer. For quite a while now, there have been system and
> software safety regulations (i.e. effective laws in the legal sense) that
> apply to various kinds of products, including aircraft, cars, medical
> equipment (like the Therac, but also wheelchairs, hospital beds, drugs,
> etc.), whiteware and many other things.
>
> If you look at the Therac-25 case in more detail, you will find that these
> incidents where patients were burnt and irradiated badly, should never have
> happened, and if the system had been developed under one of today's
> mandatory frameworks, they would never have happened. Today's technical
> systems are more complex and have more software than yesterday's. And so
> far, we haven't come up with any method to guarantee that a piece of
> software is free of bugs. But we have learnt a whole lot about neat
> development processes and practices, about QA, risk analysis, etc. And that
> is what puts us into quite a good position to develop complex systems to a
> high safety standard. Hindsight is a wonderful thing...
>
> Unfortunately, sometimes safety regulations aren't followed fully for cost
> or time reasons, and bad things do still happen. But at least there is a
> lot of formal responsibility that the makers of potentially hazardous
> equipment have to accept by law. Good manufacturers take this
> responsibility very seriously.
>
> Kind regards,
>
> Helmut.
>
> ______________________________**_________________
> Chchrobotics mailing list Chchrobotics at lists.linuxnut.**co.nz<Chchrobotics at lists.linuxnut.co.nz>
> http://lists.ourshack.com/**mailman/listinfo/chchrobotics<http://lists.ourshack.com/mailman/listinfo/chchrobotics>
> Mail Archives: http://lists.ourshack.com/**pipermail/chchrobotics/<http://lists.ourshack.com/pipermail/chchrobotics/>
> Web site: http://kiwibots.org
> Meetings 3rd Monday each month at Tait Radio Communications, 175 Roydvale
> Ave, 6.30pm
>
> When replying, please edit your Subject line to reflect new content.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ourshack.com/pipermail/chchrobotics/attachments/20131015/3cf6b287/attachment.html>