🏫 COURSES 🏫 Check out my new courses and find the SECRET discount code at lowlevel.academy

"never tested until it arrived at the hospital..." Thats got to be the worst case of testing in production ever recorded

My first year CS professor started the entire class with a lecture on how bad code can kill people and how we should take bugs seriously. It's always chilling hearing stories about this.

You'd think the medical industry would get better at preventing this kind of thing. Instead, they've hired better lawyers to deny this kind of thing.

The more you learn about engineering, not only software but hardware, mechanical, etc., the more you learn that world around you is held by duct tape and prayers

I wish the entire source code were online, but for those interested, excerpts of it can be viewed in an investigative report online.

One person, hobbyist, alone, in assembly. Recipe for disaster.

Even the name "Malfunction 54" sounds scary for a simple bug, and then is MORE scary when you see "Malfunction 54 (12777 rads delivered)"

That error message though... "Radiation is either too high or too low" WTF 😂. "50/50, let's do this!" - Medical staff

Been there. In the mid 1980's I was treated in a Therac-25. Fortunately, mine didn't fail. I did hear about the failure and it took all I had to walk in there every weekday for 30 treatment days, knowing that it COULD fail. I worked in software and had a sub-routine called SNO - for Should Not Occur that printed an error message and exited. I was amazed at how many times I hit that routine. I was very glad it was there. FYI - I am now up to 5 rounds with old man cancer and I am still here. The average for Mom and the three kids stands a 7, so I get to look forward for two more. Yea Me!

Writing async code in assembly. Of course it had bugs.

Reminds me of when I went to the dentist to get an X-ray, and saw that the machine was running Windows Vista. I felt like I was in a Final Destination movie.

Race conditions are notoriously hard to debug. Because you only have a couple milliseconds for the exact right conditions to occur to trigger the bug. This is a race condition with an EIGHT. SECOND. WINDOW. Had they tested it properly, they would have been almost guaranteed to find this. This is not just negligence, this is recklessness.

It would be a public service to have a series on this topic: "Code that kills". There are many cases like this on which code that runs essential infrastructure end up costing lives? Thanks for sharing this!

I'm a full-time software engineer and part-time nuclear/radiation nerd. I've heard this story on other channels and read about it online, but nobody else goes into detail about the software aspects, which I find the most interesting. Great stuff LLL!

I hate this "too high or too low" type of error. It's like searching for an email on Lotus Notes: "Your search returned no results or too many results". Please be specific with error messages

Sounds like a chain of failure from the machine, to the hospital in all regards. The machine manufacture did not care and the Hospital also did not care too.

I remember the Kyle Hill video on this, and he glossed over the software bugs part of the Therac tragedy. This shines a different light on the importance of software safety, especially in mission critical or life saving tools. Kyle's focused on the tragedies and their relationship to the ongoing nuclear age. Very different and interesting perspectives. As a software engineer, it is always chilling to recall this story.

Something like this, where someone's health is at stake, should have had a team of programmers agreeing on, and reviewing each others code. The root cause wasn't the lone programmer - it was all those above him who signed off on that lone programmer. Disgusting working practice and yes, that lone programmer should also have recognised the danger immediately.

This style of video is really interesting, it would be pretty cool if you could produce more videos with stories like this

Sobering and sad. A reminder that clean, thoroughly tested code is crucial, together with the assumption that there still may be bugs no matter how many edge cases are accounted for in the tests.

Software never got tested until it was shipped? Sounds like all the AAA games coming out

New fear unlocked: going into a surgery and the machine just bluescreen mid-surgery

the virgin pre-production testing vs the chad testing in production

I'm working for an organisation that creates training checklists for operators working and operating machines in manufacturing sector. This video is an eye opener for me to why I must be more focused when writing my code. People's lives depends upon what I write.

I started my career in Dental X-Ray designing and manufacturing company as a Junior Embedded System R&D engineer. There the hardware team has the master role always they critisize and having less trust in software😂. I remember how regorous regression tests they've done before going to launch a product.

I first heard of and learned about the Therac-25 in a college technological ethics class. But I never knew what exactly happened in the code! So interesting and tragic!

I'm studying programming in university right now and this was one of the examples my professor used to demonstrate how a mistake in code could have massive and sometimes even fatal consequences. He also pointed out that with more testing and a better graphical user interface this all could've been avoided.

The problem with testing is that you test what you think you should test. If they would never had the idea to change the mode afterwards, this bug might have been unnoticed despite testing.

Writting software is a weird experience. It doesn't matter how many scenarios you've simulated and prepared for, there's always something that WILL go wrong.

I'm still confused (other than for profit motive) why the same machine for 180 rad would be used for 12.5k rad dosages.

"Entirely software controlled in 1986." Is a scary sentence.

I feel the enjoyment you put into this one, thanks for the great content LLL!

I'm putting exactly 0% on the developer. The company that contracted them didn't do their due diligence, and you can't expect a solo dev to account for EVERY single edge case. They chose to test in prod.

<a href="#" class="seekto" data-time="515">8:35</a> Hardware interlocks and oversights should always be included "Too make things cheaper" Ah, another money over lives situation

Legends test in production: Ocean gates

as a programmer, I've always been afraid of going into something as serious as the medical field. I'm not always 100% confident about the code that goes out as I don't have testers, my code get's tested in live environment. I can't have blood on my hands.

I took a safety class that presented an interesting perspective on the question of "can software fail". You seem to say yes. In the class, they claimed that software does not fail, because it always does what you tell it to. Whether what you told it was what you wanted, that's where you get problems. But that's not the software failing, that's you failing.

As a Software Engineer, I can say the statement "Software Will Fail" is very true. The only real way around this is redundancy, and in software, that typically means multiple independently developed systems which must all agree on an answer for it to executed

Unbelievable that they were so careless about such a critical piece of code.

Having a hobbyist write the program honestly isn't a huge error in my eyes, as I'm sure he was plenty skilled. What blows my mind is that it was never properly tested to ensure this type of thing was impossible. It doesn't matter how skilled you are at programming, you will make mistakes. We rely on others to help us catch them and correct them.

When I graduated I was offered a job as a software engineer at a biomedical company that sold medical hardware to hospitals. I didn't read to much into the details but it was a machine that was built to automatically feed (on a timed interval or when certain conditions are met etc.) doses of medicine/substances via IV to patients. They also sold heart rate monitors etc. The pay was good and the job was very enticing but I could not bear to accept it, precisely because of things like this, that were shown in this video. I could not handle the stress. Constantly having to worry if my spaghetti code is going end up costing someone their life (accidental overdose). Fuck that! I know there are engineers out there that write better code than me that would be better suited. I have no problem admitting that. I don't need this level of worry in my life. I am good. Something like this happening and me being responsible has to be one of my biggest nightmares as a software engineer.

This situation sent a chill down my spine. It reminded me of the time when I was designing security and door opening systems and the fears I had of software bugs or electronic design flaws. The extended weeks I would leave one system working alone 24/7 while another system monitored it. I can't believe that an industry that ships a machine with the lethal potential of this one would not test for it or even be tempted to eliminate fail-safe mechanical systems.

This is why I won't ever code anything where human life is at risk

that's so screwed up wtf

Absolutely insane. Incredible video.

We had a whole unit on this in undergrad comp. Ethics. Crazy how this went on for so long

As bad as the code might have been written, you gotta give it to the guy to actually put in an error message that informs the operator that they are at fault and should rethink their settings.

What a great video! It's definitely wild that people once thought that software was invincible.

Great Video! I can also recommend Kyle Hills video about this software bug :)

This was a great video. I’m a fan of your stuff in general, but this was awesome. I’m hopeful you turn this into a regular thing - the horrifying ramifications of software/tech things gone wrong. Great job!

Ngl the 'hobbyist programmer' isnt to blame for ALL of this and is probably leagues above anyones skill from the comments that are talking bad about him. This is on the company who hired him and didnt perform any testing or specify exactly how they want it to work.

As a programmer myself, this was like watching horror movie. Like nowadays even the internet form that You use to order socks has more automatic tests and testing process then that machine that x-rayed those people to death. Really I cannot imagine the despair to be the ones that got that killing dosage :(. Like ... every programmer I know uses more or less defending coding strategies, I just cannot imagine I would even allow the machine to emit that dosage in too short timeframe. Just, I am shocked.

bro wtf why does it start before you even hit the start button, bro why doesn't it double check the conditions with something that dangerous and change if it notices new values!?!?

“If something can go wrong IT WILL go wrong, sooner or later!”

The bigger takeaway from this story isn't that ancient code lacked logic and user input safeguards. Rather, that Therac's upper management made unethical design choices to lower the cost of production. Coupled with minimal pre-shipment testing of said units. "It was decided to remove the physical (electro/mechanical) safeguards and rely entirely on software to lower costs"!

This is exactly why I fight so hard as a programmer to employ good testing strategies, a testing plan is always better than a good lawyer in my opinion. I'd not want people to die for my mistakes, I'd dedicate heart and soul to good software engineering.

Well, they also made an error when naming that thing. Therac-6 plus Therac-20 would be Therac-26. Classical off-by-one-error.

this is so reckless, I cannot comprehend it

One of my Comp Sci professors, Clark Turner, was part of the investigation into the Therac-25 incident and I remember him telling us the story about how he and another person found the race condition that led to these people's demise. They wrote a paper about the investigation. Crazy stuff

Yesterday I took an exam for a computer science / electrical engineering and those races were also part of that course. Now I feel a little guilty for having somewhat skipped over that part.

Your explanation of the actual failure is not correct, as far as I can tell, reading a report into the incident. Firstly, you can't shape X-Rays (or any photon beam) with magnets. These magnets are used to shape the electron beam when in electron mode. Secondly, the machine produces an electron beam as its native source of radiation, and needs a target in front of the beam to produce X-Rays instead, as well as a flattening filter which attenuates a lot of the beam energy. Producing X-Rays requires the maximum electron beam power, a power around 100 times greater than that used in electron mode. Therefore, the patient did NOT "receive X-Ray radiation at electron radiation doses", they actually received electron radiation at X-Ray radiation doses, since neither the target nor the flattener is placed in the beam (<a href="#" class="seekto" data-time="463">7:43</a>)

I was a surgical lighting service technician who spent 20 years and 6 months on the road and in the workshop repairing, designing, developing, and modifying imported equipment to meet local standards with occasional type testing. I learnt to no longer be surprised at how manufacturers used inappropriate materials, components, and mechanical and/or electrical designs that were sometimes fundamentally unsafe. Often, the worst features of a product would be forgotten and repeated a few equipment generations later, with each new model being fundamentally more complex, less reliable, more costly to own and with an ever shorter lifespan. I'm so glad to have left the industry and hopefully all of my trailing liability behind.

Already saw Kyle Hill's video on the topic but its cool to see a more programming oriented approach

Sorry, but ignoring an error and just unpausing is risking the life of the patient in this field.

Shoutout Kyle Hill for covering this 2 years ago, his Half-life Histories series is phenomenal!

"Idk man it worked on my machine"

"Software can't fail" has got to be the single most terrifying thing to hear someone say when they're creating medical equipment. I mean, even in hardware you don't rely on a single point of failure, why would you do any different for software?

Always good to see more about this.

a finger chopping machine sent me here

This is also the case I learn in my university embedded system class. Good case study how simple code error lead to big problem.

For me as a programmer for machinery it is the typical "blame everything on the programmer" thing. It is normal, that code has bugs and you will never find all of them. Therefore mechanical and electrical safety locks have to be implemented to prevent such malfunctions. In this case the software didn't do much wrong. It even gave an error message. The main software problem was, that it was possible to skip the message and continue. The main problems in this case were the removed hardware lock in this newer model of the machine for cost reduction. And the decision of the management to let the costumers continue using this devices, even after more than one accident was reported with this machine type.

One example I have from my fathers experience was in which he had assembled and installed robotic arms and the plc's he'd designed at a car plant and the programmer came in to do the software setup and calibration my father had made him aware the safety isolation switches hadn't been completed and he was like no its fine so he proceeds to send inputs to the robotic arms which also had in production car bodies on as you can guess the arm slammed through the roof of thirty cars as my father had to attempt to stop the incident the following day those cars had scrap marked onto them.

Please tell me the family sued the manufacturer for negligence

Don't forget, in the 1970's even 16 kilobytes was a lot of DRAM. Devoting a few KB's to error codes or safety redundancies would have been a huge deal.

As a developer this makes blood boil.

As someone who knew and (like EVERYONE) was "bored" to do testing on my software, I have now done a complete 180 degree turn and testing is ALWAYS in my mind! Test your software people! Write A LOT of SIMPLE and easy to debug tests (because remember, tests are code as well and they may have bugs)! And try to think about edge cases!

Why would the machine be able to physically give such a lethal dose in the first place, regardless of the software...I mean NO ONE is going to be prescribed such a high dose... ever !! 🙄

Heard about this from my Software Engineering class, but never really looked into it. Great video!

So from how I understand things, it was an absolute miracle that more people didn't die from this... thing. Absolutely astonishing.

The one person that write all the assembly and didnt test it managed to kill only 6 person ! Honestly, i'm very impress

The problem with bugs is you have to test every condition you don't plan for. It's always some obscure condition that no one thought about that happens and causes the issue. There is no way to test every user accident in freak cases many times. You can test code for function but you can't test user situations. There will always end up some strange case where an operator did something you had not planned for.

that's horrifying. Especially since so much of our society now depends on software.

This highlights how even simple syntax errors can compile and run, but not work as intended. There's an old joke that only people who code will get, but it's hilarious because everyone who codes in multiple languages has had to contend with the differences in syntax: if (GoNuclear = 1) { launch_nukes(); } else { remain_chill(); }

this is the shit i'm scared about

please do more of these videos.

For want of a debugger, six lives were lost.

I actually was on cancer treatment in 2023 for a tumor i have in my brain and Iv'e been in one of these machines. So scary how it could have gone wrong. Thank god i didn't live in those years when these machines were that dangerous

I've heard many scary stories, but as a programmer, this is true horror. Not just because they did almost nothing to minimize errors like this, nothing that we would do today was done there. But it almost sounds like someone seriously programmed a deadly device with speed over safety in mind. I don't know why but that thought horrifies me

I know hindsight's 2020, but I don't know why there wasn't an event handler, even a basic one, so that nothing would happen without operator input. I understand this was one person, and I truly admire that they built this by themselves. They had to be under a shit ton of stress because someone that talented should be able to foresee the issues with reading inputs prematurely.

Even in the ‘90s there was still that attitude of software doesn’t fail. Take the despatch software that the London Ambulance Service started using in the ‘90s (LASCAD).

This reminds me of picture from a Testla car recently that stopped functioning completely due to software update error. May be it's time for humanity to wake up and realise that relying on software too much creates a fragile world that can collapse at any moment. We should maintain some mechanical aspects and use software to optimise not to fully control stuff.

I wonder what some software bugs from old times could do in software controlling the nukes. I mean those systems are pretty old...

To me the worst part of this was the removal of hardware interlocks. Software can NEVER be relied on 100%, even if it has been extensivley tested. Physical switches and relays should ALWAYS be in place for safety critical applications. If there were hardware interlocks in place in the Therac-25 this would never had happened. Sure, the bug would still have been there, but the machine couldn't have hurt anybody as the beam PHYSICALLY would not have been able to activate without the magnets in place.

The Therac 25 a mandatory study for all engineering students.

Please release more stories like these :)

<a href="#" class="seekto" data-time="140">2:20</a> It still said OP. MODE: TREAT X-RAY in the bottom left. Unfortunate that the operator didn't see or recognize that.

I heard of this accident, I did not know it happened SIX TIMES! EDIT: I think I head of another completely unrelated radiation overdose... terrifying.

"This error says the radiation delivered was either too high or too low..." "I would greatly appreciate more specificity."

This was extremely interesting and very well researched and presented. Brain braining++ Thanks, mate.