This! This all day.

Your first remark is very true, and not just in an aerospace environment!

That's like releasing software without doing unit tests just right after the remote guy pushed ten thousand lines of code

I feel like something in the sensor processing design isn't fundamentally robust enough if it can be this easily confused by real terrain features. Maybe they can add a second radar or lidar sensor for dissimilar redundancy or to differentiate unexpected yet real inputs from sensor faults. We all know what happens when you run a safety-critical algorithm on a single AoA sensor...

Exactly! Mission creep eats in to the project time line and system tests degrade into delta testing for success instead of system testing for non-failure.

I'll second that. The Soviets had a number of uncrewed vehicle losses because they used ionic sensors to determine the orientation of the vehicle, which would fail on occasion. On the other hand, the gyroscopes aboard Apollo 13 held true despite being pushed well outside their comfort zone. Some sort of video about orientation sensors would be very enlightening.

Yes please!!! We need more of those videos please Scott.

Another vote. I see it in formula racing where drivers are having to "fail" various sensors to address issues with the power train.

Yep, me too. I think I'm aware of the issues already, but I'd like to know how different real sensors would be.

@@ferdievanschalkwyk1669 As a former Formula 500 driver, the fastest lap times were NOT the shortest distance lines around the track. A computer simulation takes the shortest distance, while the faster drivers took a slightly longer, but faster path that defied logic.

Good point. And I think it is because it takes a higher level of intelligence and technical knowledge to understand software systems and the media and others can't understand it. You only have to look at any news article published by the main stream media, on television, in newspapers and you will see the errors the journalists make, the incorrect use of terminology, the lack of detail and you walk away realising the news article has told you almost nothing.

oof

Boom! As a retired architect for space sensor payloads, I can say you are spot on. I watched management spend all sorts of money on convenience tooling but if SW wanted licenses for software production and testing tools, oh God, you got run through the gauntlet. So how many times must a company learn these lessons? Simple, once per program.

It's the same attitude all over the place. Games no longer ship out as completed projects... 'we can just patch it later'. Mamy other fields also do shit like this.

Software in general too

Your question (A) is a great one. It could be that the new landing site could be reached with less expenditure of propellant or something like that. They thought it was a lower margin of error. Or was it the opposite? Was there a "better" more ambitious site with more interesting geography?

@@aarondavis8943 It is interesting to speculate. IMHO "less expenditure of propellant" would fit into theory about disasters and cutting corners to meet budget targets. On the "better geography" thought... unless an asteroid suddenly impacted an area close to their original site, one would think that the geography question would have been settled long ago... the lunar surface is pretty well documented (at least the front side).

Proverbs have a sort of statistical truth. "Haste makes waste" exist exactly due to that. The sad part is that seemingly we keep doing the same mistakes

@@pierQRzt180 Yes it is sad to think about all the people who have lost their lives due to decisions on someone's part to save a few bucks by cutting corners. One of the latest examples appears to be that partial collapse of the apartment building in Davenport. Looking like the owner went with a cheaper contractor who would forego shoring up the building before proceeding.

And C) why there weren't redundant sistems with majority check before deciding to discard the most vital part of your data...

most underrated comment

Life imitates art: a common problem in the Dilbert comic results in utter failure.

It's almost *ALWAYS* a project/program management issue, not an engineering issue. This was also true for Mars Polar Lander and for Mars Climate Orbiter (the one that famously mixed up imperial and metric units).

Don’t bet on it

​@@Josh_728 Get with the program: in 2023, we measure things in bananas

The brain is a wonderful flight computer. Lander: I'm going to land here. Human: Dummy, there's a rock the size of a McMansion there! Gimme manual control.

The first moon landing was saved by the astronauts: the automation on the lander was going to put it down in a field of big boulders.

think about it like this: humans have managed to control powered airplanes since the start of the 20th century, while autonomous aircraft have only just appeared in the last decade or two. humans are just that versatile

​@@MarlinMay I am howling, when I pictured this in my head with astronaut Slapping their computer and called it dumb.

One of the NASA research reports justified the cost and risk to send human astronauts to the Moon with the allegory says "Human brain is the most lightweight and easy-to-aqcuire real-time non-linear computer"

Source of 1202 and 1201 alarms was traced to the rendezvous docking radar, used for rejoining the command/service module was inadvertently left on, at the same time the radar for landing, the only one needed for the decent phase was running. This overwhelmed the lunar module computer, but mission control knew it was still safe to land because of one man at Huston.

@@warrenpierce5542 yes, when you go into the details then these are different cases. But in the abstract, in both cases the computer was confused because it got signals which were unexpected and didn't handle them well. In Apollo's case, the human was able to use additional information to recognize that the problem wasn't severe and in this case - there was no human.

@@warrenpierce5542 In Mike Collins' excellent book, he mentioned the 1201 and 1202 weren't exactly "well known" issues. Took a bit of "looking up" (quickly, albeit)

That second antennae wasn:t inadvertently left on. I saw Buzz sheepishly confess, the engineers didn’t think the same way he did in an interview.

Just realized that manned missions are technologically easier (skill of pilot) than unmanned soft landings that are possible now due to the progress of software systems.

Sorry, Humanity doesnt land on the Moon, nations do - or dont.

...and perhaps a Third error of the program disregarding the radar altimeter instead of querying it again. 'Say what? That result is outside of parameters. Please take your reading again'

I agree this probably shouldn't be called a bug. Requirements were not properly validated, so it's a failure in their systems engineering process.

came here to say that! also, even if the result lateron would be inside parameters again: the device has already been proven to be unreliable. it might be an intermittent error, or it might be a bias that only on this occasion was noticed but existed all the time. revalidating system reliability would be a tough cookie to crack on its own if it didn't come with a redundant 2nd and 3rd system, though it should have notified ground control and gotten an update/patch. to my understanding that is how generally system failures are resolved. i don't know if their mission profile put an artificial time delay on that to prepare for longer ranged versions, or what happened.

Haven't read the report so might be wrong, but if they use something like a Kalman filter, it is likely that they are not simply not querying the sensor, but that the calculated variance to associate to the sensor readings spiked. In that case, the sensor would still be queried, but is _effectively_ disregarded since the resulting effect on output would be so low (due to the change in the assigned variance). Someone can correct me if I am wrong here.

There's also the design of the spacecraft that has to be called into question, specifically the AD&C architecture. Relying on a single altimeter means that you can't verify the data with a redundant sensor. Since accelerometers and gyroscopes can't really capture things like topography from orbit, it's like flying with one eye. I don't know how much mass, power, and space another altimeter would have taken up, but perhaps a redundant altitude sensor, possibly one with a lower resolution and/or sample rate, could have been used to verify the data coming from the primary one.

It is not a sw that changed but the parameters of the flight. You may of course argue that the sw was made for the particular landing zone which I do not buy. I may be mistaken as the video is the only source of my knowledge of the situation - sort of like this radar was. So you take a peek at the surface with radar and see this crater with it or rather a human having visual would have seen the crater - the landing module saw just a point on the surface which was 3km higher then the previous point it peeked at. I suspect what they would have needed to do is to have more points that radar is measuring especially from distance and make an average out of it or use some other technique to see where one is. When much lower this would also needed to be done to see if there is no big stone occupying part of the landing zone. I suppose this last thing was eliminated by assumption that the landing is going to be done on the flat empty surface by choice of the mission control. I suspect if they were landing on the water/liquid surface this radar error could only occur due to a massive tsunami - well no water surface and no tsunamis but hard landing. Interesting to know all this tho, aint it?

Been there! Written lots of software... made some unbelievable bone-headed mistakes, which are all *BLINDINGLY OBVIOUS* in retrospect. "This change is SOOOOOOOOOO OBVIOUS that we don't need to test it" ... ha ha ha... this is when reality bites you on the backside, informing you that you definitely *DO* need to test it again.

@@hanskloss7726 HA! Then you discover it's high tide instead of low tide... maybe you simulated it with mean sea level but a mile away someone opened the sluice gates and there was a large wave from the reservoir... etc.

This moon lander crash is an example of space sabotage. Deliberate.

@@simonmultiverse6349 low tide v. high tide does not cut it here - the surface is mostly flat still at least from a 5km perspective. The crater is a different story so you need to have many points possibly also a map? Not sure what is easier here but their method obviously failed. We know this is not a shame - we all have been there....

why are lunar manned missions not done anymore these days?

@@alamrasyidi4097 Congress dropped funding, so NASA had no money for going to the moon (canceled the last planned 4 trips).

@@alamrasyidi4097 No Soviets to beat

​@@alamrasyidi4097 isn't nasa planning to go back? Starting with an (unmanned?) Mission sometime after 2024?

@@dr.cheeze5382 so ive heard. but compared to the alternative of having to lose these spacecrafts to software error, i think "no soviet to beat" is a ridiculoua excuse. so i still really dont understand why lunar exploration has been strictly rover based these past few years...

Thanks for your awesome contribution to space science!

I'm sure mission control had a plot of the expected altitude changes, the lander may have had one as well. Problem is that the expected rate of change of the altitude, was outside what had been set as acceptable for the altitude radar. It was probably written in the specs somewhere. Proper simulation of the landing would have caught this, it could possibly even have been dealt with after launch. It's changing the landing site without simulating it that screwed them.

What did Galileo and Cassini use for altitude readings? and would they have been equally screwed if forced to switch over to gyro / accelerometer readings with an apparent failed altitude radar?

@@nocturnal6863 John's point was that "forced" switch is averted, if the switch algorithm is completely disabled at such an early phase of flight. Reread about "earliest time.. a milestone can be validly sensed".

@@u1zha except you wouldn’t disable the software monitoring a sensor for failure. Not unless you knew in advance it might give faulty readings at that point. Further thinking, I think I see what you are suggesting. That it should have been expecting by the dip in altitude and it’s failure the see it, means it should have known it’s altitude was off.

In my experience the software developers/engineers are kept out of the decision-making inner circle. Actually, this goes for engineering/tech in general. It’s fine, just change this, this and that: what’s the worst that can happen? (Changing the Moon’s landscape to more closely resemble a seedy neighborhood in Brooklyn, one spacecraft at a time.)

After fully tested, I still fear my code would break in some case that we have not looked 😂, its scary for space mission

Maybe not most expensive, but Therac-25 should be on that list somewhere. Ya know, because it ended up maiming and killing a bunch of people with intense doses of radiation.

There you go Scott - that’s an epic video right there. Top 10 most expensive Astro/Software defects.

@@a.p.2356 In a way, that is possibly the most expensive software bug ever; in another, it's quite cheap. Consider: we know for a fact that cars kill people, all the time, in every way, yet we do not ban cars. The value of a human being's life has been calculated, and apparently it's cheaper than you would expect. Electricity production has a cost measured in lives per TW/h. You can look it up. Biofuel has a cost of 12 people per TW/h. Solar is 0.44. Wind is 0.15, and new/clear is 0.07. The average American consumes 0.1-0.2 GW/h per year. In other words, over the course of your entire life you will likely kill less than one fiftieth of a person in order to keep the lights on. This does stack with the people you kill while driving, however - I'm talking about tyres particulate and excess death from pollution, not running somebody over. Ain't that grand?

@@o0alessandro0o It's not easy to respond to that kind of information. I do know that training and regular checking of pilots contributes to a high level safety for commercial aviation (ignoring mechanical failures). For drivers, I reckon that similar processes should be followed. It would not be popular among the general public, but I have said for years that licenses should be graded, based on years of experience and how many training courses a driver takes. Governments balk at the idea, however, and go on putting up cameras and roadside radars, more draconian speed limits, but never addressing the fact that poor situational awareness, slow and inappropriate reactions, and limited skills are the biggest factors in car accident rates. But I'm going down a rabbit hole!

Not strictly a bug, rather bad design; but implicit nullability - first introduced in ALGOL in 1965 and later copied into most programming languages - was famously coined by its creator as a billion dollar mistake. I think I read somewhere that at the time the estimate was quite accurate, but that was 2009 so by now it wouldn't be surprising if it is an order of magnitude too low.

Pop op o99⁹9th kiwi's😊

It is a software bug though. People problem is that the bug wasn't caught

Have you ever experience a flight termination system not working instantly, but 50 seconds late, as with the starship launch?

@@vast634 Depends on the type of Flight Termination System (FTS). For solid rocket motors, they use a shaped linear charge that opens the casing and exposes the fuel which burns up quickly in an impressive display. (I think Scott mentioned that in his previous video.) For chemical fuels, things are different. You have more choices. The basic idea is to stop thrusting the vehicle so it falls into an unpopulated area, such as a broad ocean area in the case of SpaceX. Based on the video of the flight, the FTS worked properly and detonated explosive devices that created holes in the fuel tanks. That reduced or stopped the fuel flow to the engines. The FTS did its job. After that, it's all physics. If the fuels are hypergolic, they will combust on contact and you get a near-instant explosion. Otherwise, you need combustible fuel, oxygen, and an ignition source. Guessing, it took about 40 seconds before the three elements came together in the right quantities in the case of SpaceX. An FTS doesn't need to create an explosion. Rather than connect to explosives, the FTS can connect to fuel valves that terminate fuel flow.

@@xGOKOPx I understand your perspective. My point is that the bug should have been avoided during design or implementation, and if not, then detected during development testing. Find and correct all the bugs before deployment. Since their development testing failed to react properly to "unexpected terrain" (kind of a silly term considering the moon's terrain is pretty stable), the people failed in the software development cycle and left in a failure mode (i.e., the bug) so it could be exposed during execution. The software did what it was designed to do so it worked properly. The people failed to account for something. The same thing happens with hardware but folks don't usually blame the hardware. The failure of Galloping Gertie wasn't blamed on the bridge. The people who designed and built it were blamed for not accounting for potential wind loading.

You still believe in this fable? Open your eyes

You @@PT-xi5rt didn't know that LBJ was president? Or that he used the bathroom like the rest of us?

Agree and also on my part, I always thought that radar altimeters were used « closer » to the surface.

Probably budget constrained.

@@drill_fiend1097 This is a commercial effort so they could have just gotten one normally used in aircraft. It's not NASA where they cost $750K each just because...

I hope that they can make a new one and landed on the moon the next few years

I watched the landing live stream and I felt so bad for them. Their nervous faces really hurt me too. I bet they do it next tie!

They may not accomplish it. Very often such incidents reveal a whole load of issues that have been swept under the carpet, and the necessary organisational change required to address them all can easily break a small team / organisation. Even big companies can be killed by this. This is what is going on in Boeing right now. They caused the crashes of two 737MAXes and killed people. Since then they've tried to institute root and branch reform of how they run their business. Yet, they're still having problems. The most recent one was a fuselage manufacturing defect (they were building them wrong) that had gone unnoticed for approx 700 airframes (yep they're flying, possibly with Southwest today!). Fine, they've found it, repairs needed, not immediately dangerous, but cannot be ignored. Trouble is the manner of them finding it was accidental; someone was in the right place, at the right time and realised what was going wrong. The issue is that, if despite the introduction of a root and branch reform about how they approach quality (= safety, reliability) they're still finding major issues by chance, then the root and branch reforms are junk and are not working. They should be finding such problems as part of a systematic continuous improvement process, and they're not. So the bet-your-life question is, what else have they missed, given that they've essentially admitted that they've not been looking hard enough? It's similar with 787 (fuselage barrel joints), brand new 737MAXs with FOD and rodent damage, etc. This suggests to me that Boeing are in no way adequately reformed following the MAX crashes, the problem most likely being in the senior management who never understood it before and are still there today. It's worryingly possible that they're going to make another fatal mistake. Ok, the FAA is now (belatedly) keeping a much beadier eye on Boeing, but they can't see and check everything; certification engineers / inspectors are not there to do basic QC and basic QC improvement. The Hakuto team's best bet, if they're to try it again, is to just fix that one core issue and try again, and do as much simming as they can muster. Unlike Boeing, crashes are just disappointments and money.

❤❤❤❤❤❤❤❤❤❤ Yes they will succeed…… After they spend a lot of someone else’s money……… Let’s all go to Sugar rock Candy Mountain

But they launched knowing that their testing was invalid. Kinda like practicing parachute landings in a field, then jumping over water. I hope they don't ask me for a donation, because polite refusal often offends.

Cliff case*

🎢

Edge case? A crater on the moon? But it's not just a software fix is it? Or are we talking about a KSP do-over?

Seems a pretty basic error, in what universe did they think they could figure the exact altitude without the radar? Even if it was broke too bad, damned if you do/don't.

@@thePronto They moved their landing site to align with NASA South Pole targets super late in development (post validation). The threshold for culling/re-baselining seems to be the issue. The sudden change in relative altitude wasn’t expected from their simulations.

Actually these days so much of manufacturing and coding is outsourced that the management, hardware and software teams are no longer next to each other - quality control begins to suffer massively. The more people outsource stuff, the more the work gets into the hands of rookies paid on cheap wages, who then end up making rookie mistakes that then require even more time and energy to fix. Boeing turned from an engineering firm to a management firm and the rest is history - 787, 737 max, 777x, Starliner. And as more and more automation comes in there's less and less human intervention to take care of the times where the computers reach their limits.

Feels like they might not have a great CI indeed, probably more like bunch of artifacts in git LFS type of management. But Starliner glitch might be slightly different topic IMO

I was also wondering how expensive such a simulation would be. If they aren't too expensive, I was wondering if you couldn't run landing simulations from randomized positions and flag anomalies from there. Not so much that you can just fling the lander at the moon arbitrarily, but more so you can find starting conditions which result in something weird. IDK, maybe we're getting into a space where "moon lander software testing" and later "asteroid lander software testing" might be a market, that would be amazing. With the costs of these missions, there might be some money on the line for a testing company - especially if they end up with a body of "known problematic situations" like the one from the video.

How can you put a tank that has experienced both problems and damage in test into Apollo 13? Exactly like that, only different. Or get km and miles crossed up and smash a probe into Mars (IIRC), or ... ad infinitum. You can run all the simulations in the universe and still have problems, but not running ANY sims to cover a deviation in the program...yeah, that's just begging for it. I would think, in this day and age, that you could pretty much run that sim real time in parallel with the mission, for the problem they had there, knowing the path and surface profile, I'm guessing, and have it fire up a quick "do not ignore the damned properly functioning radar" command, or some such. It might even be good to have REAL TIME simulations running against the truth of the mission. Having done some aerospace hardware design, I'm guessing that there were schedulers and/or bean counters directly in the problematical loop. Or maybe idiotic MBA wielding managers that think they are engineers, or worse know BETTER than the engineers, because they know a few buzz words, and then maybe hold people's feet to the fire to get them to sign off on VERY cold Shuttle launches, or what have you. That's the sort of feedback you do NOT want in, say, a servo. :-/ Sometimes I look back and am glad I am retired, frankly. Some of it was fun, some of it SUCKED. Doc requirements come to mind as some of the latter. I had one junior documentation fiefdom wannabe tell me that the real output of a program was the documentation. When I finally quit laughing I told her that if she actually believed that she should go talk to some F16 pilot and ask them which they'd rather have with them on a mission, a working LANTIRN pod, or the documentation that describes it. She wasn't happy, because then a couple of people standing around laughed too. She wasn't a nice person (that's putting it mildly), or I wouldn't have said it that way. My bad, I guess.

Armchair quarterbacks are a dime a dozen. You can certainly crow if you ever land a vehicle on the moon or even achieve orbit. Perhaps we can talk about "how obvious" the solution was when we stop whining about how LONG it takes to build and fly a vehicle and how the contractors are "milking the American public" for so much money. I thank Providence every day for Kathy Lueders and NASA for riding herd on SpaceX to make the Dragon 2 safe. Everyone had lots of criticism for NASA for being conservative and "delaying" the first launch of the manned spacecraft. But all that effort kept the astronauts safe. (Also SpaceX had a real leg up on Boeing, because they had a working cargo spacecraft in Dragon 1 to build on. The last time Boeing designed a manned spacecraft was the 1970s and the Space Shuttle. All those engineers are long since retired.)

This.

@@mcgilliman I like to think of an F1 analogy... Imagine if you set your car up to race in good sunny weather in Monaco at sea level, and they changed the race to be in Mexico in soaking wet weather at 2,260m above sea level... You would NEVER just race the car with the exact same set up and no testing before the race!!

True, but if history has taught us anything it's that the incompetence almost certainly wasn't the software engineers themselves and was instead a cumulative effect of multiple levels of beurecracy repeatedly ignoring the recommendations and pleas of the people who actually knew what they were doing and what additional work had to be done. I suspect this is a scaled-down version of Challenger all over again, albeit thankfully with no loss of life this time.

How do you know? Did you read the design spec? If the design spec stated it should be able to handle multiple lunar landing locations, then it's not a design spec issue.

Definitely a process bug that this wasn't picked up in testing - but premature to say that it wasn't also a software bug.

@@pigsnoutman 6:17

Mismatch at Requirements and/or Expectation levels. Activated by beyond test envelope operation. Needed a calm (seasoned?) "captain" to hold a steady, pre-planned course.

​@@simongeard4824 I think the video was quite clear on that the software started ignoring that sensor because it was programmed to do so. An intentional feature that behaved differently than expected *because* it was put into a situation that was not considered while designing it. And that this only happened because they changed the mission plan after the software was developed and did not test it again with the new landing site, because their tests would have detected the issue. That last part really hurts because they reasonably could have avoided the crash.

they didnt find the issue, they made the issue

Sounds like the software took the path of flat-Earth "science". What I see doesn't fit my preconceptions, ignore it!

People need to wake up, controversy surrounded moon landings because there is stuff there. The issue / bug was in there on purpose. They will probably never let us see the real moon.

They did not find the issue. The issue was management. The software was fine. Software did not change the landing location, management did.

@@davidbeppler3032 The whole things was planned it is every time with every country why do people not see this, every single machine that lands on the moon has issues - #1 because the surface is covered in glass domes and other hanging debris #2 to cover up such things from the public in a convincing way.

They chose tamer landing sites.

@@PMA65537 apollo 15 likes to have word wth you

Apollo missions had some issues but they handled then well..The engineers couldn't even visit their families becoz of the work pressure

You can have a video camera that is very good at finding the distance by using phase detect autofocus, same principle as a rangefinder.

I work in flight software and you're right. At a certain point if a system is so critical and irreplaceable you just have to trust it won't fail because as you said, detecting the failure isn't helpful if your SOL.

There is an argument to be made that if a $90 million project can go up into smoke due to a single sensor failure you have an expectation that it could potentially fail, you should really have some sort of redundancy even if it is unlikely. Or some other form for backup plan. The question is if it was actually considered if this sensor could fail, or if it just used the same behavior failure detection and handling as any other sensor without further consideration.

Except the one that flew into one of the first computers and caused a short circuit.

Scott's been moving to more and more clickbait titles of late. It's unfortunate to see him doing it.

When you're using accelerometer and gyroscopic data alone for position on a 2d plane, it can become hilariously inaccurate quickly, no matter how good your algo is. Doing this in a 3D plane would be basically impossible if I'm being honest. As an example, there is a reason VR relies so heavily on video processing for limb positioning. Obviously these aren't in the same ball park of cost/importance, but the same rules apply.

The unbelievable part here, honestly, is how someone expected this to work without simulating the actual final flight plan at least once.

@@winebartender6653 US missile submarines can pull it off, but their inertial navigation hardware is larger than the entire lunar probe and submarines experience much smaller accelerations. It does seem like it was selected as a fallback with rather optimistic expectations of how well it would stay accurate. In hindsight, it would have been better to try turning the radar off and back on, relying on inertial navigation only as long as it took the radar to come back on. Also, redundant radar.

I actually did not get your point . Could you please explain little bit more ?

the point being this was a bug tha should've been relatively easy to find, if thoy had simulated a couple of "landing site changed at last minute" scenarios that included heavily cratered areas or craters with steep walls. Just doing some tests on random landing sites would've triggered this. But nobodu thought of this, and because of corporate culture, everybody was dis-incentivized to even raise the question

The software was built by Astrobotic, an American company. Not sure how stereotypes of Japanese corporate culture come into this.

@@JosePineda-cy6om Oh ok . Thanks a lot for the explaination

They outsource programming. It happened due to budget and time constraints. I'm pretty sure engineers wanted to rerun the simulation

Which is a shame, really. The more expensive the project, the less management should feel like cutting corners on error handling and verification. Ah, well, what "should" happen in the real world doesn't agree closely with what actually happens in the real world.

Redundant systems and majority check: if all three of your radar sensor reports a sudden altitude change, than that's what actually happened. What surprise me is that the sudden altitude change eventuality is never accounted for...

It then accidentally creates a cult.

Which novel is this?

@@yogiwp_ Absolution Gap, it's the third book in the Revelation Space series which is kind of weird. If you're looking to check out the author, I'd recommend Pushing Ice!

@@letsburn00 And also liquifying the poor guys wife stuck in the scrimshaw suit

@@ShoeTheGreyCat I forgot about that bit. Given that series largely relates to characters that are functionally aging immortal, it's wild how easily they torture and kill each other.

I wonder if this would be a good application for AI? A computer would definitely be able to interpret way more inputs than a human pilot ever could and in real time

It's a satisfying problem to work on.

​@@andrewahern3730 AI isn't a magic fix. Those sensor fusion algorithms are supported by a a deep understanding of the system and statistics. Like with the Falcon 9, they are extremely reliable once properly tuned. Obviously an advanced enough AI system can always do the job. But if, like in this case, you simply didn't test the system with enough variations of inputs, you're not going to get good results either. The amount of simulations needed to properly train the AI would also have been plenty to find this bug in the old control code. The lesson here is that more robust testing is needed. I have a feeling that spaceflight is often seen as hardware-first. That's understandable, but without proper software the hardware is useless. I think more modern software engineering practices could be useful here.

IRL, nothing says you can’t have false positives and false negatives at the same time, while you struggle to understand the data. That’s no fun at all.

kalman filtering is pretty damn straightforward. It's a basic method, not something extraordinary. Known for more than 50 years and optimal for typical sensors (ie those with common noise distribution).

what you're describing is human decision making, and you're ready to scratch this plan and try something better when the moment comes. you can't just imagine every scenario branching out at every step and hard-coding solutions to each. At some point you'll realize you need a generic decision making algorithm. In fact the mission failed due to them having a specific solution of switching off a reading since that allowed them success in previous simulations.

That’s a very creative solution! ✌🏼✌🏼Pretty sure would work!

If they don't agree, then what do you do?

@@xonx209 In sci-fi usually it would be three independent systems with two having to agree as to what they were seeing. A two system setup would require that both system have to agree and if one system cut out a sensor as malfunctioning and the other didn't, something would have to be present to break the disagreement deadlock.

Wasn't there something about thermal modelling and pipes freezing in the fregat upper? Or am I misremembering

@@JohnMullee No, that was definitely some other story

As a software developer myself, I'm actually asking myself why those simulations were not set up to run like a CI.

​@@jarisundell8859 Good question. Seems like actually running the sim again once the final site was chosen should have caught this, maybe allowing them to upload the fix. For the record, CI is how the one I looked at was caught... prior to release 👍

Scott is also a dev by trade, too, lol, he works at Apple.

@@firefly4f4 I think it's a joke, since the bug happened because the computer didn't "believe" the radar

By unbelievable, I mean the software stopped believing the radar

One of the reasons Chandrayan-2 failed because of the mapping. When the lander moved away from the photographed landing site it tried to over correct and failed.

My thought exactly.

And that that one method could be turned off for the rest of the landing!

I guess that's part and parcel for those very small landers.

I mean to be fair, even the Apollo lunar module only had a single non-redundant landing radar altimeter for determining exact altitude. The astronauts were fairly confident they could manage to land without it but if it failed, mission rules called for an immediate abort. The weight constraints for landers are so tight, engineers have no choice but to make those trade offs.

@@GlutenEruption Ah, but they had the backup that was the Mk. 1 Eyeball and its associated biological computer!

Hope there's no software issue when Nasa lands back on the moon!😂

@@adarsh4764 agreed - one would have thought that even a small lander would have a pretty robust navigation system these days, but obviously they met an edge condition they hadn't properly tested for... and a sad oversight too as nearly all landing trajectories will have the radar return affected by craters you're passing over. There are many of them after all, and although most are small, many are large/deep and you need to keep their profile in mind as you use the radar/laser/etc surface measurement. The state vector routine needs a sanity check to make sure the drift never disagrees from projected too much without it doing some form of reliable recheck.

Sounds like a redundant sensor wouldn't have helped this particular issue though, because it would have just gotten the same confusing measurements of the cliff wall. I think they just need to thoroughly run simulations of the actual mission to catch edge cases like this early.

On a vehicle like this, without humans onboard, the space and weight requirements might be too costly compared to the risk of a failed sensor.

@@sonaxaton exactly. Proper simulation campaign would've catched that.

@@sonaxaton 3 redundant sensor and a voting system is the way to go. Worked flawlessly in many aeronautical things, from Concorde autopilot to missile guidance system.

The dead reckoning system combined with prior knowledge (a map of roughly what is expected) should have been enough of a redundant system. Seems like they should have included a reassessment/recovery routine to check if that apparent altimeter glitch (which wasn't a glitch of course) cleared and the instrument was giving reasonable data. This stuff is really tricky without a human in the loop.

Actually, I figured out 500km/h based upon the amateur radar measurments of 88seconds of freefall.

@@scottmanley Love how reliable basic physics equations are! With either bits of data it still comes up with the same results! If only the rest of landing on the moon were that simple.

What I want to know is how that equates to a violent impact on earth. Do I divide by six, which comes to 83.3km/h or just under 52mph? That's bad enough for it to need airbags...

@@travelbugse2829 You multiply by the square root of six - assuming there is no air resistance, so with air resistance you might end up with something not too different from 500 km/h for this type of vehicle.

@@travelbugse2829 When it comes to the moment of impact, 500kph is 500kph. It's about Mach 0.5. You know those old war movies where they show fighters shot down and augering in? *That.*

I was thinking same thing. Sounds like the software did exactly what it was supposed to do.

well, a bug is just unintended behaviour. The computer did exactly what you told it to do, just not what you wanted it to do

Can't imagine why they didnt run simulations of this. It's not like the moons topology isn't known down to the meter. Stick it in Kerbal and run simulations.

@@ddnguyen278 Uh, the moon's topography *isn't* known down to the meter. Some areas of the moon are, but generating meaningful maps of the moon is actually quite hard and time consuming. There are folks whose entire job is to take lower res digital elevation maps and apply reasonable interpolations to generate higher fidelity maps than we actually have. Not saying they shouldn't have done more sims, but it's harder than it sounds.

Repeat after me “That’s not a bug it’s a feature”

It’s unbelievable because the navigation software stopped believing the altimeter.

@@scottmanley Lol, good save. Wasn't really directed at your video per say, just that's the KZfaq titling by youtubers trend these days. Although yours is actually technically accurate hah 🙂

What was the 'REAL TRUTH?!!'

And then a unicorn ran up & they rode the unicorn all around the Moon going 240,000 miles back to Earth. The Unicorn didn't run 28,000mph like they would've had to go in the pop rivet aluminum can they brought them there.

Word!

They lost telemetry. If they had a connection they could saved it.

There might also be delay in communication.

I agree. This was planning error, or a failure to test error, or changing the landing into a regime that had not been tested, or all of the above. It's been know for a long time that radar altimeters can be spoofed by terrain, it is nothing new.

An engineering oversight

As a software engineer I agree lol but that's not to say it is not also partly the responsibility of software engineers to raise potential bugs in the design.

Agreed, and came to write this. As a space systems engineer, this was a systems engineering failure, not a software bug.

They should have tested their software with real data.

Those software engineers were layed off, hence the laying in bed awake at night

Yes, that makes sense, and the "forgiving" part is commonly solved by Kalman filtering, which Scott also mentioned. Here it sounds like Ispace overengineered a little bit, overeagerly dropping sensor data on the floor before giving the filter a chance.

Exactly. There are even retry and resume techniques, when data is out of scope, and how to change the code according to each retry to be less strict and try to salvage "the most possible". (Ex: a harder landing better than spinning out of control, etc)

My understanding was it believed 5km elevation was the moons surface, so it was going to hover there slowly descending until touchdown. So in other words it was always going to run out of fuel. (I think the video said it was descending at around 1km/hour, so it would have needed an additional 5 hours worth of fuel!).

Excellent point.

And do you really think the Japanese didn't deploy that?

@@alwayshiking_ Well, apparently not since it crashed after thinking for too long that it had landed, 5km above the surface ...

With the AI the need for higher processing power to multiply large matrices come. This increases the electricity power requirements and requires more RnD for creating radiation-hardened variants of processors. The Snapdragon 801 in ingenuity Mars drone is probably the state-of-the-art SoC being used. But that's the same one used for Galaxy S5 a decade ago.

I doubt AI in space crafts any time soon unless we develop more effecient processors.

It substracts where it should be, from where it wasn't.

Exactly. It would be especially helpful in this case; if the lander knew where it isn't, it would not waste fuel by trying to land as if it was just above the surface. :)

I believe that's just a sentence for lulz, engineers expressing themselves purposefully obtuse. Kalman filters are exactly the "knowing" part, and a closed loop control system is exactly the "subtracting" part.

@@u1zha kzfaq.info/get/bejne/mMCVaK1puLyniYU.html This video is full of these sentences, that are close to how control loops work, but not quite, which I find quite funny, especially if you know how it works

@@u1zha Unfortunately, it also inspires a lot of morons to quote that line constantly on KZfaq, perhaps under the mistaken impression that it makes them look smart.

The Moon has very few stable orbits, and they still require propellant. The moon is very lumpy and the earth tends to fling you off.

Dropping beacons on the surface is the exact same thing as putting a lander like this down.

@@samuraidriver4x4 To make it easier to land our probe, we will first land three probes.

Apollo did that once using a Surveyor lander as the beacon. But that was just for position, not altitude.

@@samuraidriver4x4 Really? Can you describe your design? I was thinking about a lightweight baseball sized package on the end of a long collapsible pole, throw a dozen of them out in hopes a few survived to do the job. Or how about a huge air bag with the probe suspended by rubber bands in the center, no need for an accurate landing of the beacon. Did y'all really think I was suggesting to land a huge piece of equipment to be a beacon?

Good point. However, I believe (hope?) Scott was only using it as a a well known example and they don't really use a "naive"/textbook implementation of either Kalman filters or Mahalanobis distance... 🤷🏻 I mean, the moons surface is NOT a hidden probabilistic distribution - so at least four dimensions could use euclidean distance, verified against the same using surface maps - not some exhaustive search in simulation training data for planned approach vectors 🧐 ...if they need help implementing that - I actually wanted to visit a friend in Japan for some time now 🤣 (just kidding, compared to them I also don't have enough experience - but I'm 100% sure, if the control-loop "broke", it was more indepth than due to a wrong geometry of the probability space)

Software engineering does not start at the keyboard, and end when it gets sent to a testing team that is somehow not software engineering. Engineers have a responsibility to work with testing crew, to validate the test scenarios. The teams failed to run enough variations of realistic input, so inputs outside the limited sets caused a fault. Specifically several bugs in the system as a whole 1 Spacecraft is unable to land without altimeter inputs. Relying on only inertial guidance cannot be accurate enough to land, due to inherent input noise. If the altimeter signal is discarded more than X seconds before touchdown, error margins cause failure rates approaching 100% 2. Guidance system (apparently) had no way to recover confidence in sensors 3. Guidance system would erroneously flag valid inputs altimeter as a broken sensor. 4. Testing was not done to cover new landing site (and yes, a senior engineer should have balked at the change)

So, as the lander found, is the moon.

There are very few human activities that are as complex and as routine and yet have such potential to fail catastrophically from even the smallest of errors. So yes, it's objectively a hard thing to consistently do well.