If the devs could reverse a binary tree on a whiteboard, this wouldn’t have happened

So this is the backdoor that 10x dev was talking about?

this dyslexic man is doing his best to read for us and you're laughing

My phone started blowing up last night right as I was about to take a shower. I was sitting there checking our infra butt naked. Jokes on me, the version of OpenSSH on our servers was old enough to not be impacted.

Curious that they mention that the code that fixed this was "accidentally removed" again and again. Knowing what we know about agents introducing backdoors intentionally, how are they so sure that this was an accident? Not saying we should immediately start pointing fingers for sure, but going to the other extreme and emphasizing it was an "accident" without knowing it to be so also seems like a dangerous assumption.

Im just a dumb web developer and don't understand any of this

0:33 we should -thank- *PAY* OpenSSH devs for their work.

Rare case when hacking/it security really looks like it's imagined to be: reading source code for hidden vulnerabilities that can be exploited with incomprehensible dark magic.

'So we started reading glibc malloc code' LOOOOOOL

This is big brain territory here

I just came here to say that this doesn't affect OpenBSD, the project that created and maintains OpenSSH.

19:40 - actually the point is that is packet (with a final byte) is very tiny, it doesn't get segmented and then re-assembled at any point over the internet, hence delivering it is way more reliable from timing perspective, than sending a large chunky boy.

Malloc internals (and the internal locking mechanisms) are some advanced voodoo. My traumatic encounter with malloc internals: A few years back I was troubleshooting what we thought was a memory leak. Turned out it wasn't a leak per se; what had happened was that a new "optimization" had been added to glibc's malloc implementation, which attempted to mitigate lock contention by creating new heap arenas whenever two threads collided on a lock. The idea being that threads which did a lot of malloc/free calls would effectively get their own dedicated heap arenas (eventually), thereby minimizing future lock contention. Problem was, over time this would cause the number of heap arenas to asymptotically approach the number of threads. And since heap arenas were created with a certain minimum size (64MB IIRC), in a long-running application with hundreds of threads you could eventually chew up ridiculous amounts of RAM. Mitigation involved setting an environment variable to cap the maximum number of heap arenas, and living with the (tiny) performance hit from heap lock contention.

I'm a normie corporate guy who manages some websites. This is such a great channel just for late-breaking security news. Thank you!

i'm tired boss.

I'm just thrilled to see so many references to one of my favorite modern ska bands! :D

29:00 - they send authentication KEY - which is memcopied from packet into the memory for auth checking - this is why it's important to cause sigalarm while it's being checke, because they KEY is the malicious payload that when executed right jumps the execution pointer to "yes this dude is valid and give him shell"

6-8 hours sounds long, but if you target the attack to start overnight or on on the weekend, that's incredibly serious.

The name.... is 128KB long

Just something to note, just checking the package version is not enough to assert the package is vulnerable, debian and ubuntu often backport patches for CVEs from later software versions, so even if you are using a supposed "vulnerable" version, if you check the package notes (and the package per se) you will see a lot of patches, especially in LTS versions.

i use vim btw

Love this kind of videos about vulnerabilities. Keep up, Prime! o7

On Unbuntu servers 1 line patch is: pro fix CVE-2024-6387

Wonder if something like the delay symptom they spotted in that xz backoor could be used to nail this window more consistently? Like a minor issue in one oss giving better odds at a basically probabilistic attack on another...

Google notified me of this yesterday (bc they host my VM). Went in and checked if my OpenSSH version was affected but luckily I use ancient Debian that's stable literally forever so the OpenSSH version was _older_ than the exploit. Which I believe is like more than a decade old

“ssh is a joke, I know the guy who created the back door”

i could feel my head smoking cartoonishly throughout this

Gotta get LowLevelLearning in on this, this stuff is his bread and butter

this. is. insane. ... just wow ... the effort and analysis they must have put into this! well well well but eventually did they try turning it on and off? :>

*Chuckles* I'm in danger!

I’ve never felt so dumb in my entire life. This is too hard for quice-eater devs like me.

I’m proud how well I managed to keep up with the text, yet horrified because of the implications of this. 1 day of SSH logins is nothing its not like I actually collect the logs properly most of the time…

Such a shame prime doesn't pronounce ssh as "sssh"

Chuck norris reads emails through heap overflows!

isn't this only 32bit? and can be mitigated with some config? or did i miss a bunch

That eagle looks more like a mantaray.

I tried to update on Windows 11 to the latest but then I can not connect with HeidiSQL any more. So, I had to return back to OpenSSH_for_Windows_8.6p1, LibreSSL 3.4.3. I am using only OpenSSH client and I think it is not so dangerous to keep this version.

I think you mentioned solar designer without realizing who it is. The guy is an absolute beast and author of John the Ripper, a famous password brute forcer.

Dang quiche eaters...

webdev doesnt know how to read C... the quality of the Netflix Staff right here...

The calloc vs malloc in the PAM handler. Gottem.

10:13 😂😂 "what a code!"

On some servers, sysadmins set timeout to over 600 seconds because usually if your SSH session is idle for more than 120 seconds then you're automatically logged out.

Sending all but the last byte of the DSA packet isn't about timing due to packet coalescence. It's about not having to wait for the network to transfer all that data in one go. If you have to transfer 4K of data, that's going to take time. Transferring everything except the last byte will take the same amount of time (more or less) for the first part, but then as you approach your window to win the race, only having to send a single byte will be a lot faster, and therefore easier to guesstimate when it should be sent.

You sound like me reading my college philosophy text out loud.

The interrupters are great

And this is why the Primeagen is on another level

"when you're reading malloc, you're getting DEEP" 😂❤❤❤

randomize timeout for every connection to conquer the timing attack part?

48:07 "I like how they use word 'easily'" - I agree. It would be interesting to hear what kind of task the authors would call "barely controllable" or "nearly impossible to control".

The Interrupters mentioned! "She's Kerosene" randomly plays in my head constantly. I had to focus on the music because I'm too dumb to understand the tech parts of this 😂

Dang Woody was unstable when I installed debian for the first time

_laughs in musl_

mind blown

I have no idea what any of this means. Sounds bad though

Thought Slime mentioned

With respect to the one vs two packet delay time, in TCP, if you enable nagle on the server and delay ack is always enabled on Windows then the delay will be about 200ms longer in the 2 packet scenario. God help you if you get this wrong because an entire team of engineers couldn't figure this out for over a decade until I showed up and caught it.

the fact that they are interrupting the code withing free and using quotes from "the interrupters" is funny

This is one of those vulnerabilities where the more I _think_ I know, the more I know I know nothing.

What did I get from this? Magical magic is magically magic. HURRAY!

you're insane BRO

9:46 Ska music mentioned

Now I really want to see Prime and Casey do some exploit development.

oh Jia Tan again?

24:09 Flip did not give a flip. Can't tell Flip what to do.

I love that we talk about avoiding race conditions, but they talk about /winning/ them

52:32 So, when you get to 52nd minute, you understand that this wasn't that bad at all. This condition is probably present in many softwares.

FWIW the "Malleus Maleficarum" was a book written in the 1400s and served as the Catholic Church's justification for witch burning/killing. That's where the "Malloc Maleficarum" is coming from.

so... me running ubuntu 24.04 as my SSH gateway while all the rest of my servers run debian 12 potentially saved me ? nice ! :>

There's always at least one more bug...

if this was introduced by removing a define, why did it take several months and collaboration for a fix?

i love flip

one of the information about one of the software

Ah hour later found it 😢😂

I'm under the impression that we should replace all mallocs with callocs, just to be safe. It'll be slower, but a lot harder to exploit

quite a nice vuln if u ask me

see eye aye

How and what are these people paid?

Hmmm nothing like an open public port 22 🤤 At the very least, please put a IP address whitelist!

>ssh -V >9.6p1 Oh, shid

NOT AGAIN !!!

When someone comes to me saying they want to become a "Hacker", This will be the video I point them to. 😄

...in October 2020 by commit 752250c, which **"accidentally"** removed...

I was going to make a remark about OpenBSD reducing their vulnerability clock but apparently OpenBSD survived this.

ah.. aha!.. aargghA! .. might explain things

13:55 Burger king ez

Seems like this is only for DSA keys? Does any one use those in 2024?

TL;DR: ssh was supposed to use single-threaded but was executed as effectively multi-threaded thanks to SIGALARM being implemented incorrectly (single-threaded program should not cause any non-volatile changes to program state from SIGALARM handler). Had all of ssh been written as multi-threaded code the SIGALARM handler would have worked as expected because it would have had to use proper locking to access shared memory structures. Of course, that would have been true only if somebody had been able to write *correct* multi-threaded code in C - that is, without any security vulnerabilities. Even Linux kernel fails this every now and then. Human programmers are not careful enough to write security sensitive code in C except for random happy mistakes. Update: 41:05 Yes, in other words it's re-entrant bug. Shouldn't happen in single-threaded code in theory but incorrectly written signal handlers can break those assumptions.

Not me consuming an entire bag of taki's like it's popcorn at the movies... this is a real thinker, and ngl this vuln is alot like one I theorized about and then may have found being exploited in the wild, first on windows then a few months later also on a few devices running a few different android releases. All the android devices observed had outdated linux kernels (from 2017-2018 yet in phones made in 2022-2024) I dont even wanna ask why some oems do it, but just please stop using old linux kernels with deprecated or known unsafe features!!! CAN WE AS A COMMUNITY AGREE ON THAT anyways... here's something interesting for us all to ponder upon and also wonder how TF together (btw just think ocr style grid-array encoding but used on streamed-in frames and you''ll get where im going in this comment) A Short Essay on Unsafe Decoding and Parsing Algorithms "Why we Need More Intelligent Memory Filtering to Combat Address Space Grooming" I found a really cool (i.e. SCARY) way to hide a header chunk, where hexadecimal will get read out as a series of blank space, when read as any plaint text file, even though the series of two byte values each individually do show up when read by a hex text reader app. You wanna know how stuff is getting smuggled into being used for supply chain attacks even after an org has done their job and securely restored their data, there ya go. You need to be inspecting headers and footers for all the things, and start logging possible autonomously triggered instances of 'head' and 'tail' commands. I'd share my theory in full but some of the concepts are fringe at best, so idk who really would take it totally seriously. But suffice to say, if anyone has seen the distorted psychedelic coloring on some youtube videos, while using an ARM based android device (v6 or v7) then you at least can reproduce this bug if it affects your config, and maybe you might be able to confirm. Android versions I can confirm have the behavior I referenced above: Android 11, 12 Android versions I cant confirm or have not witnessed the referenced behavior: Android 9, 10, 13 My hypothesis and/or ideas going forward: Yall I think something is straight up introduced in Android 11, and fixed/changed at Android 13 release that either knowingly or inadvertently blocked the yt video coloring issue, but I think maybe the effects of whatever those patterns are, may be still there and just better hidden maybe even completely on accident. I dont get nearly the amount of views that would necessitate making a video on this but I have been steadily gathering info about the issue for quite a while, ever since I found a memory leak issue which seemed to only plague devices connecting to a certain Sagemcom router. I have a hunch that its something similar to the recent Windstream Isp issued router vulns, which may have been the initial vector at which the issue started at least in my personal observations. (my router was NOT a windstream product, but it WAS a router from one of the US's big three carriers) If you are a programmer or another researcher, and see this comment... PLEASE look into it if you have the ability.

I think the fix they mention does not solve some root causes for these user-injected code. For example, first thing you do when you receive that username should be checking whether it is valid UTF-8 (in your unprivileged child), the only "names" that make sense. This removes lots of possibilities to include binary code because they are usually not UTF-8. This is one way Rust "could have" prevented these issues, btw, but more in a "common practice" way than a "C cannot do this" way. Of course you can do these in C.

Today is the day I realized, you look like Dr. Disrespect, but without goggles

£ me too, Prime!

Wow, rhese new vulnerabilities making me sus if stuff online. Gotta be careful

I am a C/C++ programmer myself, but now I have just been heavily brainfckd!

I understand most of the words 👍

This takes me back to the guy that wrote that article saying that writing good c code was merely a skill issue. If the open SSH developers have skill issues, I fear for the rest of humanity.

woody appears to be a stable build - looks like version 3.x & releases around 2002-2006

The title of this one should have been "Prime learns about the craziness it InfoSec exploits". And he does it superbly!

Just use Wireguard and SSH only to wg0

These guys must be bad at hiding in crowds with how large their heads are