*the agency

Spook. E.

So we can safely assume, fixes tin hat, that he had his target all prepared, knew that it would fixed super fast, so he prepared to execute his attack and is now fulfilled. I wonder who the target was… removes tin hat.

Lines up well with Russian timing tho? Step 1: Ukraine Step 2: The world?

@ It's pretty clear that they were hoping it would get into the stable release given that they submitted their updated version to Debian when it was close to releasing a new stable version. Remember Debian was supposed to have a new stable release out in 2 days from now and the attacker was trying hard to get it into that release. They definitely did not fulfill their goals, whoever the target was they got very lucky.

Yes. The correct analogy is not internal Microsoft testing (they obviously wouldn't have caught something this subtle). It's imagining a developer running benchmarks on Windows, finding something taking 0.5 seconds longer than it should, and then not being able to explore what's causing it. So they, what, email Microsoft tech support and explain something is taking 0.5 seconds too long and should look into it? This shows the danger of closed source. That email will just get deleted due to how silly it sounds and the exploit will exist in production code forever, never being caught.

Solarwinds

Meanwhile Windows takes like forever to update, can any one check if it is backdoored?

We don't know it but we I'm sure. Is that your logic?

​@@wardm4 Some functions *are* benchmarked. But it depends more on if it's a function expecting many uses. So if syndicating account login to a separate server, then the login process should be part of a benchmark to verify how many login/hour such a server can handle. But it isn't likely a single-user login would be benchmarked, since it isn't part of some critical capacity pipeline. So I would be very surprised if Microsoft would have caught this. But if an AD server can handle less requests/second, then that is very likely to be quickly caught.

It is, and it’s shown that OSS’s biggest attack surface is the humans. That’s also true of closed source software of course, but one assumes that a company paying salaries to developers has some idea who those developers really are. With OSS, we don’t and nor does anyone else. My worry is that as more package maintainers retire this is going to become easier to pull off. To guard against that there’d be pressure for maintainers to hand ownership over to some trustworthy organisation. Who is that? The danger is that it becomes RedHat. And then they own more and more of Linux. They already effectively own systems Wayland and gnome, they could be in line to own even more.

Exactly they preyed on the developer who was alone, stressed and feeling the pinch. It went on for a while with several people or puppets involved.

I still wonder if "Jia Tan" was always a hacker or was originally a college kid who eventually was contacted by his national secret police to collaborate with them and inject the backdoor into XZ after they had gained the trust of the community

@@abarratt8869 It is also the largest attack vector

Where is he from?

I agree, but for precision, I never said it was part of the kernel, of course.

But when people in the community say "in Linux", they generally mean the kernel. It's like saying "in NT" when discussing Windows. "In systemd" would be more accurate than "in Linux", but still completely untrue. It would, however, win you brownie points with the systemd haters, who tend to think that the architectural philosophy behind systemd is just asking for something like this.

@@JonBrase It's even more more like saying "in Windows", when IIS has a bug.

When common people say "Linux", they mean the whole software system, not the operating system that Linus maintains even today. Even the word "distribution" goes past most people not understanding what it means, why they mistake it as well to "different OS". So common people make wild claims that if any software that isn't part of the OS gets bad things happen, then it is the OS. The problem is that not even most software engineers understand what OS is, what it does, what is its purpose. It is simple really, but not as simple as walking someone at their car and opening engine hood and show them where their car engine and all other different parts inside the engine compartment, as not everything in there is part of the engine. But at least most people understand that car isn't engine, and engine isn't a car like people think about software systems and operating systems. Why word "Linux" is always causing misinformation when it is used to talk about anything else than the operating system itself, aka "Linux Kernel".

@@JonBrase there is, however, no denying that systemd is a steaming pile of kaka - regardless of it's philosophies.

Ohhhh yeah, I remember that. IIRC, the Sony CEO(?) when confronted about it basically went "it's fine that we did this because most people don't know what a rootkit is, and what you don't know can't hurt you". Even the most tech-illiterate people can probably tell *something* is wrong with that attitude, even if they don't know exactly what.

@@LendriMujina assuming they actually used the word "rootkit" it should sound dubious enough to the average person that the flippant attitude towards it being discovered by the CEO should warrant suspicion

@@LendriMujina It was even worse than that. The Sony rootkit used cloaking functionality which could be reused and several viruses actually used the built in cloaking functionaliy of the rootkit to hide their viruses from anti-virus programs, basically making themselves invisible to operating system and apps.

Sony seemed unable to see what the problem was. They were just doing what they wanted. You must remember that Sony has authority, granted by Sony, to do anything that Sony wants to enable Sony to fulfil its right to do anything that Sony wants.

Truth being said here. Open source is ideal to spot weaknesses, backdoors, etc and correct them. A private company would probably have hidden this

This is the strength of open source.

Don't even mention the time period between discovery and fix. Microsoft isn't even dreaming about those response times. Btw. Which OS are they using for their Azure Cloud. Windows server right? 😅

The problem is that the amount of people that can do meaningful code reviews is relatively limited. While in theory open source code can be reviewed by all and will be reviewed, the practice is different. More often then not important libs like this are maintained by just one or two persons.

One of the issues with Open Source is weak identity verification of the contributors. At this point, the alias of the contributor is known but their real identity is unknown. It could be a state actor or an individual, no one knows.

The other thing is that a large company like MS does take things seriously and check their stuff. Does a smaller company? Or what about one that makes software for their bespoke hardware? We do have a somewhat recent supply chain attack on the proprietary side of things with Solar Winds hack.

Win10 ltsc is actually pretty great though, can't really understand the hate.

@@emptylog933 If you are used to the speed and user friendly Windows 7 you will hate Windows 10 and Windows 11 a lot.

@@emptylog933 It made A LOT of stupid anti-user friendliness changes from Windows 7 to accommodate dumb users (and/or tablet users) that it's infuriating. Also the forced updates, forced maintenance, forced bloatware, forced fucking everything makes it a chore to use. Sure there are improvements in some regards but there's so many annoying details and major issues that go unfixed (like the 100% HDD usage even on clean installs) that it makes using Win10 a shitty experience for many.

@@emptylog933For one thing, Microsoft's aggressive campaign for everyone to upgrade to Windows 11. The fact you have to refer to "ltsc" should make that evident.

ROTFL just like Google which does ultra deep antivirus scanning without user consent and keeps stats for Google as a bonus. Google Play Services basically runs as root and actually owns your life.

Microsoft has publicly known of one major backdoor being utilized by hackers, the US government, and even foreign governments for over four years now, and still hasn't patched it. Pegasus.

/facepalm

I'm utterly disappointed my comment has been seemingly deleted. We DO know of multiple backdoors in Windows that still have not been patched. There, maybe this one won't get deleted.

One of stuxnets zero-days a backdoor in the elliptic curve used for auth in windows install. Then we also have the older _NSAKEY which was the same thing, but older.

According to the author of this video, windows has no back doors … 😂😂😂😂😂😂😂

Ding ding ding!

And many are probably INTENTIONAL. Just think about how the UK government wanted to pass a law to basically force backdoors into messaging apps, for example. Or in a way you could consider telemetry as a sort of backdoor as it basically grants unlimited access to your data to the companies. Open source should be the standard, if we actually want safe code

If someone told me the CIA had backdoors to all ios and android versions, I'd only half doubt them.

@@no_name4796 "Just think about how the UK government wanted to pass a law to basically force backdoors into messaging apps, for example." - Actually they DID pass that law...

@@CoolKoon yeah... And what about the apps that won't comply? They probalby install CA cert in your machine to doesn't have JUST your messages, but ALL your traffic yeah?

it also means everything done by that maintainer is suspect, going over two years of history to check if there isn't anything else nefarious probably isn't a small task

If they put that much effort into getting trust for an open-source project, how much more likely are sponsored people to be working at closed-source companies e.g. Microsoft, AWS, Oracle, Okta, Azure, Google, LastPass, Antivirus providers with nefarious goals?

@@pietersmit621 Probably even more, but by compromising the linux kernel, they will compromise all those big companies. At the very least we should start looking at how we could harden open source projects from such attacks. This was pretty much a wake up call.

@@pietersmit621 As Dave said, those big companies have a process and more people looking at everything so you'd have to get many more people involved. Afaik this was a "small" project with a single overworked maintainer that was pushed to give the bad actor access

Recently someone involved with F-Droid mentioned that a few years ago they had a similar situation happen where someone was really pushing for code to be merged that would allow SQL injections to happen in the app's search function and would result in a crash. They put off reviewing it, but when they did review it, the original submitter deleted their account to hide evidence. This could be related to the group behind the XZ backdoor, I think I heard.

GitHub allowing people to manipulate releases separately like that is a pretty exploitable feature, imo.

95% of the chain was commit to git, the last, the last 5% that triggered the integration in the build process was added only as part of the release tarball.

​@@nurmrSure, but that 5% that triggers it is the key trigger that would look suspicious for anyone who takes a closer look. The remaining 95% are the binary files in test folder, but you need a way to incorporate that in to the build system, and randomly grabbing test files during a build, and doing lots of decoding operations is going to raise eyebrows to whoever is reading it. Maybe it would have slipped in anyway since xz isn't super active, but someone like Lasse Collin (the other maintainer) might have noticed it. Hiding it in the release tarball and relying on the fact that no one checks the consistency of release artifacts is IMO the a key technique here, and it relies on maintainer access because this is something a contributor can never do.

​@@BrotherCheng "rabdomly grabbing test files" was hidden too, there were some very weird obfuscated regexes that would only match those files.

​@@The_BoctorGitHub's own tarballs are automatically generated, but you can attach any file you want to a release (otherwise it'd be a pretty shoddy distribution mechanism), and I guess for whatever reason it's a common practice in some projects for the devs to upload their own tars instead of using the automatic one? (I assume there's a reason for this since otherwise why bother with that effort, but I don't know enough to say what that reason is.)

Or tell as is the case with publicly traded US companies

And also rely on the fact that the library was only used in a roundabout way when a specially patched version of sshd was loaded via systemd. This attack is brilliant on multiple levels!

@@stephanweinberger I'm not sure it was brilliant. Certainly novel, but if you give any systems level dev who has a couple years of experience the same task they can likely all come up with something similarly hard to detect. The most clever thing about it is it uses a xz compressed file in the xz library to test xz for the backdoor. Everything else is just typical obfuscation which doesn't take much skill to create

@@__Brandon__ It's not the obfuscation alone, but also how the backdoor is activated in the field: it's planted into sshd indirectly via systemd - both of which don't need to be touched in any way. That's thinking around _two_ corners...

Sounds like the NSA, DOJ, CIA, or any other alphabet organization of the Government.

Yup, very naughty.

Why were they banned if i may ask

@@MohammedShuaybNo, you may not ask. Sorry.

@@MohammedShuayb They were purposfully adding bad code into the project to see if people would catch them. They claimed it was because of some research they were trying to do but thats a load of horseshit and they were banned and still are from what I know for a very good reason.

Yes.

and that, at least most Microsoft products, aren't as optimized, so that half a second can't be noticed

Good lord, large companies doing huge loads of time sensitive work open tickets routinely on closed sourced code everyday. I watched one get opened last week for millisecond spikes on storage after a new patch.

That would drive me nuts. Are those typical anomalies, or ominous indicators?

You obviously didn't even watch half of the video.

​@@RonJohn63 Why do you think so? Becaus he says ms has a better process?

also, their name is Andres

It was a Postgres developer who found this.

PSYOP

Me myself don't want to google for xz utils right now. Might as well keep the tin hat on for some time. The more information you recieve, the more painful it gets. Anyway the guy behind Tukaani Project must be a Finn, since his name and the Project are in Finnish. This feels sad, because in Finland we just had our suicide wave after there was a hacker exploiting psychiatric data.

That was one of the clever aspect - ssh is security sensitive so changes get checked thoroughly - but xzutils (or rather liblzma) isn’t normally used by ssh so doesn’t get checked the same way. Some configurations do happen to inject it, though, enabling this backdoor to trigger. Very very sneaky stuff, someone put a lot of time/money/work into injecting this code, to the extent of becoming a project maintainer just to do this.

To be fair, I don’t think journalists were the targets here. How many journalists use Linux??? My guess is this was supposed to be a cyber weapon to be used against a foreign adversary. The only thing that would truly be safe would be old systems that didn’t have the patch and air gapped systems that couldn’t directly connect to the outside world.

@@benjaminlynch9958 yeah not many journalists, just most of the servers across the whole internet, nothin' big.

Journalists work for the alphabet agencies anyway. They’d be spying on themselves.

​​@@benjaminlynch9958most people use Linux or UNIX but don't realise they do. Android is Linux underneath. MacOS is a flavour of BSD with a custom desktop. If this hadn't been discovered it could have found its way into all kinds of systems. If you run Windows, many of the services you use run on Linux, that email website, your VPN, that website you use for managing your source code, they probably run Linux. If someone used a backdoor like this to attack a website like GitHub or JFrog, they could inject malicious into hundreds of products, both open and closed source. The potential implications are truly jaw dropping and could have let someone tamper with release builds of code running on pretty much anything where the developer used any of the common development management tools. This could even have had safety of life implications if the tampered code found its way into things like medical equipment, airliners or equipment in nuclear power plants. If someone intended to use this code to compromise a major cloud provider, the implications could be very serious including unauthorized access to even governmental or military systems. We can't assume this is the end of the matter. The goal might not have been to compromise every Linux server in the world. It might have been to compromise a particular developer or maintainer's machine to insert some other compromise that hasn't been detected yet. Such a person could already have run the code as they're likely to be on the bleeding edge. Hopefully, this was just someone's plan to steal a boat load of cash from somewhere like a bank or cryptocurrency exchange

@@benjaminlynch9958 or *BSD or Solaris or... But yeah, Linux is rather popular, but for say government systems, one would still have to get past filters, firewalls and more to even try to log onto the server. So, they'd be looking at beyond difficult to access servers, with pinhole firewall entries for only the necessary services on the unclassified networks, all of the important traffic being on a number of classified networks that cannot be seen or see the internet or the unclassified network and are actually tunneled under it and one another, yeah, this would miss central government networks and basically nail provincial and municipal networks, as well as commercial and utilities networks. And utilities are a target of interest, as some high profile attacks on water supply networks have recently shown.

Yeah, it's complex, like VERY complex, but if you have the private key for it, and only 1 person presumably does, it allows you to run commands as root simply by sending a specially crafted handshake message that will trigger the backdoor and run the command embedded in the handshake message. Low level learning has an excellent video discussing how it works, and you can find the video here: vV_WdTBbww4

@@WarrenGarabrandt so many suspects even freund himself because of his connection with microsoft.

Dave doesn't understand it. It was done in this manner because they knew they could never get to SSH directly. It is very clever stuff, and it should make us all wonder about whether there are backdoors in closed source software like this that would never otherwise be detected. The very same techniques can be used.

like a third party attack@@elta6241

Which essentially does the same thing: root privilege escalation. For a over the top explanation from a Microsoft guy to his usual audience this is good enough.

if you are inserting a backdoor you would want to make it look like an innocent mistake

@@Schifty1988 ....how do you manage to make introducing a back door a "mistake" ?

@@user-cd5ki1ip3ithat is difficult! imagine you write code that enables you to perform SQL injection; this is a common programming mistake which would be spotted by a decent review; next imagine obfuscating that introduced vulnerability as some sort of optimization that does in fact look reasonable

This is why copyleft licenses (GPL for example) are so vital! It forces companies like MS to, if they want to rely on such libraries, to make the _entire_ code base opensource as well. This is why projects like AOSP exist. We're kidding ourselves if we think Google makes Android opensource out of the kindness of their heart. They have no choice. 💁

This is why copyleft licenses (GPL for example) are so vital! It forces companies like MS, if they want to rely on such libraries, to make the _entire_ code base opensource as well. This is why projects like AOSP exist. We're kidding ourselves if we think Google makes Android opensource out of the kindness of their heart. They have no choice. 💁

@@soulstenance Once again, "rms" is right. XZ utils isn't GPL or LGPL, it is "public domain" which means ANY company can make changes to use their own code "enhancements". The same goes for any Apache licensed code which Android is licensed except the Linux kernel. I believe that is the part annoying Google so they are developing Fuschia.

Absolutely. It’s amazing how often that gets forgotten. And in that sense, it’s difficult to say that the person who put this backdoor in did anything particularly criminal. They modified their source code and build system (ie they legitimately had access to all this). Other people copied it and used it. Ok, they were doing so out of malign intent; probably. Or it could have been a security flaw demo that simply got revealed too early (less likely). If one actually took it to court, it could be difficult to make a case stick. Probably one would have to go back to the campaign they launched to cajole the original maintainer into making them an admin, which might be construed as obtain goods and services by fraudulent means (uk ish legal language). But computer misuse? There’s no evidence that I’ve heard of that the backdoor has actually been used. What we don’t want is every flaw and mistake in OSS repos to become a reason to prosecute the maintainers. After all, what is the real difference between a blatant backdoor and a careless bit of programming? Nil.

Or worse, execute one's desired goals non-interactively, not forming the outbound connection at all until one's goal, say data exfiltration, is achieved. Now, one's only potential warning is the outbound - oh crap, data's already gone. Often enough, many are caught and halted by catching the bidirectional connection, regardless of which end initiated said connection. I've actually done it, captured the entire session in full packet captures, including the malicious software of the week, which was immediately submitted for inclusion in the IPS and antivirus and novel aspects of the attack fully documented and submitted to FBI intelligence, as it was a foreign nation-state actor and known APT. One such attack involved lateral spread that I'd gotten a sniff of, began packet capturing and captured an RDP session in progress, attacker opened notepad, did a buffer dump of binary data into notepad, saved it and used it to assemble their tools on the target system. Got not only their binary tool signature, but the latest PXE padding for their known tool. Outlined the remainder of their attack, then their session experienced a mysterious termination before the data that remained safe on the victim system could be exfiltrated... The system was then immediately isolated pending full file pulls and forensics. Two more attacks and we found their point of entry into the corporate, global network. One forgotten test server on one DMZ, prompting a full review on every DMZ in the corporate network. Yeah, as well received as the plague, but necessary after two golden ticket attacks, one of which was successful and boy, you don't want that kind of aggravation!

He probably hedged for the same reason I would... none are known, but you can't prove a negative.

@@DavesGarage You can't prove a negative *by hiding the source code.

If the NSA has a National Security Letter forcing Microsoft to allow insertion of backdoors, it is also a felony for Microsoft to disclose this fact. This is why some projects have a "warrant canary" statement in their periodic disclosures. They say, "We have no state-mandated backdoors." Their position is that if they are ever forced to introduce a backdoor, they can remove the warrant canary statements from future disclosures without explicitly breaking the law, because they are not *telling* users that they have been forced to introduce backdoors; they are simply no longer telling users that they have not been forced to introduce backdoors.

Microsoft doesn't need a back door, just use an Alternate Data Streams.

@@inpito Microsoft IS the backdoor, no wait its just the door, no secrets here.

I can't agree, because (a) it was found through testing without the source (but diagnosed with source), and (b) it was found by a guy at Microsoft who was doing the type of testing that presumably Microsoft does on its own stuff. One hopes, anyway.

Just not true. The benefit of open source is that it can easily adapted to fit your needs and those modifications can easily be shared. You can read and write assembled binaries in much the same way and people do all the time to find exploitable bugs or patch closed source software.

@@DavesGarageThe majority of maintainers getting a bug report about a 500ms delay in a separate process from a single user would've probably considered it very low priority. Luck was definitely involved, but the fact that Microsoft engineer was able to check the source to diagnose the issue themselves is the reason this got fixed so quickly.

​@@DavesGarage Just because you replaced word "found" with "diagnosed" doesn't change anything. It was found (not "diagnosed", Dave, "found") because developer had the access to the source code. To the complete source code, not the stuff that hosts on github (vs code says "hi"). Freund didn't "diagnose" that binaries were stored in tests folder, he "found" it. In the open source code. Don't replace word "found" because it breaks your ignorant narrative

@@DavesGarage The guy found it during his easter holidays, when apparently he prefers running Linux for some reason.

It sure helped me pass my ethical hacking course 😉

Bugs aren't backdoors, though. I did not and do not claim closed source produced bug free code, so I think this is a strawman fallacy!

Ms17-010/EternalBlue is not a backdoor; rather, it's an exploit that specifically targeted SMBv1 and weaponized it. The NSA did not inject the code responsible for the integer overflow and subsequent actions.

Ms17-010/EternalBlue is not a backdoor; rather, it's an exploit that specifically targeted SMBv1 and weaponized it. The NSA did not inject the code responsible for the overflow and subsequent actions.

@@DavesGarage the difference between a backdoor and a bug is intent. Hard to determine intent… who knows maybe that was malicious claimed ignorance.

Probably not, at least not early evetually I thinkl so but as Dave said it had a really small number of devs working on it and the bad guy was already trying to get it included in the stable releases of Fedora and the like.

No the public process isn't inferior. But it is by no means guaranteed (it isn't on closed software either). But a lot of people believe that open source software is inherently safe (because everyone can see the code). In practise it much more nuanced. As you say, its caught early this time. But what if that guy didn't dig in deep, or thought you know what I'll look at it after the weekend (and forgets about it). A lot of open source software is dependant on a small number of people and on the whole there are not many people that can do these kind of analysis, let alone make time for it.

But it went mainstream. Arch Linux shipped it, as well as some other beta versions of other distros. Open source isn't as well scrutinized as one would think, the stage where it would most likely be cough was on the commit review, especially for a project that wasn't that popular. The issue is, project maintainers do it voluntarily, often a thankless and stressful job. Many develop mental health issues from those problems. And that is the main problem with open source.

@@EmilioBPedrollo Arch Linux is a big offender with rolling releases, honestly if you don't live on the edge there are plenty of distro that stay behind one or two update cycles which is safe. keep in mind the other distro that use the latest updates are the reason such things get caught early, if you're not tech savy or you have sensitive info avoid such distros

@@EmilioBPedrollo I disagree with your definition of mainstream. To me, mainstream would imply an LTS release of a commonly used distro. If you want to compare with betas and rolling releases, apples to apples, compare those with Windows betas and builds sent to public testers. I understand your skepticism, but you'd also be surprised how much more scrutiny is placed between what you describe and until it reaches Ubuntu LTS for example.

Many Linux distributions don't password protect root. Without a password, root is then denied an interactive logon.

@@spvillano Locking the root account =/= no password.

And if the makefile isn't under version control is it really open source

So the main lesson to learn would be to always check out the repo and not just copy the tarball.

@@stephanweinberger save that a trusted individual could as easily then generate a new hash for the Makefile, save it and it appears as valid as everything else in the repository. So, timestamp auditing would also be necessary, as its hash would be saved at a different time and date than the rest of the files. Well, unless that also was altered. My first rule of information assurance, trust no one, not even myself. So, stumbling blocks, such as checks and balances, auditing before publishing, etc always get inserted, regardless of my gonad pain. Never had a persistent compromise on networks I was in charge of, so I obviously did something right, given we were repeatedly targeted by multiple APT's.

@@spvillano we could all go sign the commits and gpg sign all build artifacts. That would be enough to ensure that atleast we are all using the same source code and that we know which key committed the code. Without gpg you can put anyone's name or email as a commit author to try to muddy the waters

Your Rawhide box may just not have pulled the package in between when the update was put out and when it was pulled. It was RedHat themselves saying that all Rawhide boxes should be nuked to the ground.

Thank you, came to say this. Claiming that it affects all of Linux is either misinformed or disingenuous.

Umm. Ubuntu and Debian ARE affected!! But only their cutting-edge beta versions.

​@@raidensama1511 exactly, the stable versions of Ubuntu, Fedora and Debian are not affected.

We can't say stable versions are completely unaffected given the bad actor contributed over 700 commits to the project that go back years.

To find that paper, search for Ken Thompson’s paper entitled Reflections on Trusting Trust

There's an article called "Reflections on Trusting Trust" by Thompson, 1984, that touches on this idea of can you trust the compiler that compiles your compiler. Google for that title and it will come up dozens of times online. Anyway, it's an interesting thought about that.

"Reflections On Trusting Trust" by Ken Thompson. Published in 1984. If anyone hasn't read it yet, now is the time...

So the question is... does it still lives inside the compilers undetected because compiler needs compilation as well :p

Yes, I remember this. This was early days when Linux came on scene, it was evolving and more eyes were getting involved in open source.

Did not imply that. Specifically said I am not privy to the process. But if the process was as through as it should be, they would have caught this as a perf regression. But all that is in the video!

…or what if was NOT a ‘years-long campaign of espionage’… What if the ‘bad actor’ was actually honest and good, but was suborned AFTER receiving ‘the keys to the kingdom’? Maybe he was blackmailed? Or bribed? Or ‘got an offer he couldn’t refuse’??? 🤔

@@ernestgalvan9037 Hard to rule out. But the bad actor was laying groundwork for the backdoor since June/July 2023, which is a very long time for an account to be compromised without a "true" owner noticing or notifying someone.

@@DavesGarage they wouldn't have because xz on it's own wouldn't have had a perf regression during testing. It doesn't get noticeable until sshd gets involved - but not the "official" openssh, only the patched versions that end up being done by distro maintainers to pull in systemd-notify support; so there's at least 2 levels of indirection before you even get to the point where it becomes a potentially detectable issue. And it was detected, and rectified.

@@DavesGarage Liblzma has tests and a specific test file is even part of the way the backdoor is injected into the binary during the build process. One of the test files is an corrupted xz archive which contains obfuscated code for the backdoor. During build process the build script extracts this obfuscated code and adds it to the source to inject the backdoor into the binary. The developer even requested a change to Google's oss-fuzz to prevent the detection of the malicious code by this testing tools. Additionally he helped fixing a "bug" in Valgrind, which was caused by the backdoor. All of this was prepared bit by bit over several month.

Be careful! lol

Yes a binary was added but it is common in things like compression and encryption code to include TEST DATA that is encoded so that part of the test is to decode it. The bad code wasn't even obvious in the decoded version. The hack literally snipped bits and pieces of it to assemble the bad code from innocent parts.. I have seen test data include things like tar files and further encrypted files (to test effectiveness of compressing compressed data etc.) so it is not as hard to understand how this binary test data got in.

@@julianelischer6961 Dave highlighted exactly why raw Makefiles are terrible. They're a mess and a nightmare to audit. Which is what let the attacker kick the whole thing off. Unfortunately, there's a large number of developers who haven't moved past the 90s, and think C with Makefiles is the ultimate end all be all.

"But it's not in Linux! Just in the repos!" How do you get Linux installed? "Install one of the repos!" So you can see why this argument is not that compelling...

The attacker started earning the other maintainer's trust over the years, until he finally took over the project. It is believed he created other Github users to put pressure on the maintainers of those distributions to include the newer versions of xz-utils into the repos. Also, most open source projects (like this one) are done as a hobby or just for the love of contributing to open source!

@@DavesGarage…or, you know, just compile from stable/LTS. Sure, you live and die by the sword, but I don’t know of anyone or any company much less that would willingly run their production on bleeding edge software save for devs on VMs to claim a paycheck. It’s sad to see people conflating bugs and backdoors here in the comments. 🫤

@@DavesGarageYou mean distros, right?

Close source applications use open source libraries. In fact they use the same libraries that macos, android, and linux use. So by this logic window is android. Widows is macos. And windows is linux. Is very bad logic indeed

That’s the way. Keep it out of the repo but inserted into the release.

Do you think that GitHub should remove the ability to do "hot" changes to existing releases like that? Not asking a rhetorical, I personally dislike that feature because of things like this. A program called MilkyTracker was also broken on a few distros last year, because someone working on it changed a release without incrementing the version or anything. As a result, packagers didn't know there was an update!

Or somebody having expensive decompilers(IDA costs a lot, there are also free tools ghidra and radare, but they are not so good). Here it was telling that Freund is "not a security researcher, nor a reverse engineer."

how many state sponsored backdoors have been baked into CPU chips and closed source software? operation rubicon went on for years. China (im guessing) probably planned this back door for years and got caught.

@@rs.matr1x Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...

Probably is, the trick is sufficient cross checking and auditing to try to trip that up. The biggest plus in open source is, tens of thousands of eyes on all of the code. Maybe George didn't spot the makefile change, but Bob had an odd command he needed plugged in to compile the thing for his specific needs and ran into the unauthorized change. Bob asks about it, none of the devs know what's going on and start prying, then the lid pops off of that jar. All for free, whereas in closed source, all have to be paid, so are lesser in numbers or odd one-off cases that would cause them to go into a makefile. Welcome to engineering 101, where everything is a tradeoff. You've taken your first baby step into adulthood. Beware, there be dragons! No, that wasn't a dragon, I'm just not used to eating beans that much anymore... Sorry!

@@spvillano Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...

Well actually, you may be joking, but you're not wrong.

Log4j was not a bug, it was a "feature". Java strings by themselves do not go ringing the internet. Log4j intentionally made JDNI support. I don't see how paying for the work would have prevented either case. "Jia Tan" definitely wouldn't be like "I am being paid for xz, I should quit stop being a state actor on a fat payroll" Log4j developers(who made the feature) definitely wouldn't be "maybe we need less features?". If anything it'll be opposite: more bloat.

@@AM-yk5yd It's not about Jia Tan, it's about the xz maintainer. Jia Tan got through because he was able to exploit the situation of a single maintainer being on the wedge of a burnout. Jia Tan probably already was on the fat payroll of an intelligence service. He's the one getting paid for infiltrating, the maintainer is not getting paid.

@@dreamyrhodes anyone with an actual working experience can tell you that burnout does not magically disappear because there are money, so this point is moot. The moment project would be looking for other developers would be a point bad actor intervene. And most likely would look better than other candidates - even because why i say "he" there is no reason to believe even now he was a lone wolf without team. If you not even for a minute think on how bad actors would behave if it situation was different, you are not thinking enough and propose solutions no more effective than asking "are you over 18"on lewdy sites.

It would require a conspiracy amongst multiple people at a couple levels, and if you're willing to set those as a criteria, than Linux could be subverted in the same way, I'd bet. But again, since I don't know what the checks and balances are, I don't speculate.

@@DavesGarage This tells us otherwise. The techniques used here can easily be used internally in an organisation. I immediately get suspicious when people use the word ‘conspiracy’.

​@@DavesGarageThe only "conspiracy" needed is for the source code to the component in question to be under NDA, with anyone who signs the NDA being told "this backdoor was required by the feds, if you talk about it you can expect a visit from the FBI". Given the likely actor in this case, however (mainland China) an attack they made on western versions of Windows would likely be conducted in a similar manner (social engineering / corporate espionage leading to a supply chain attack on/through middleware), since they don't have direct jurisdiction over companies headquartered in Redmond.

@@elta6241 people with a corporate mindset (no offense, Dave!) are almost as a rule predisposed to not wanting to think about under-the-table dealings, even though every government has a body of people whose day jobs consist of precisely that. I'm not sure what the Canadian equivalent of the NSA would be, but I don't think I have to explain in a comment section under a tech channel what the problems they pose for software development would be.

@@DavesGarage If it's open source then it's going to need to be subtle or obfuscated or it will risk being spotted. If it's closed source it can be a lot less subtle. Even if you assume most governments wouldn't do this. There's a reason Germany and the US are advising against using Kaspersky products due to what's going on with Russia. And a similar thing with Huawei and China.

And even worse, the files in question are specifically *corrupted* compressed files (corrupted by design, as they are supposed to be test cases). That means, the file can't even be inspected as is. To inspect it, one would have to read those very obfuscated scripts which un-corrupt the files before unzipping them.

Yes, it would mitigate this vulnerability to only allow listed hosts.

​@@Sa1985Mr so essentially this isnt a problem, never knew an organization that would just leave ssh open. But then again I still see them use telnet ffs.

That’s not true. As Dave mentions in the video, the attack code isn’t actually in the source code. It’s in a precompiled binary. And it was caught not in some open source code review, but through plain old benchmarking. The CPU was pinned at 100% for a trivial task, and the benchmark reviewer started asking questions because it didn’t make sense. If the attacker hadn’t got careless and instead limited CPU cycles or network traffic to something resembling ‘normal’ activity (say 10%), this likely wouldn’t have ever been caught despite being open source.

@@benjaminlynch9958 sure it was hidden, that's was the point. It would have never been trace to the hacked binaries without OS. It shows nothing is infallible. In closed source like MS may find a rogue independent programmer but we will never known if it management approved backdoor.

@@benjaminlynch9958If it wasn't open source the engineer wouldn't really be able to see why ssh was causing so much CPU use. He would have only been able to file a bug report likely to ssh which isn't even the root of the issue. So someone working on the ssh package would have to get the bug report and diagnose the issue, but if systemd and xz-utils were also closed source those devs couldn't do anything but forward the issue. Now even if the issue managed to make it to the proper maintainer, he's one guy and he's burnt out on the project so there's always the chance that he can't figure it out easily or promptly. It is kind of nice that the engineer who knew what he was doing could follow the chain on his own without hitting roadblocks due to proprietary black boxes.

​@@theodis8134 It's easy to detect the cause of high CPU usage in Windows by using Sysinternals Process Explorer connected to MS public symbol server.

It was pure chance that a developer working on another database project noticed a discrepancy in connect times via ssh after an update.

These sorts of things always get called "Linux" problems - pretty sure this would be possible on BSDs using OpenSSH as well

@@etherweb6796 Nope! systemd does not exist on BSD's. Thank god.

Your colleague must have put in a lot of effort to get this backdoor into his Mac, considering how few linux distros adopted the change. I mean, it popped up in the nobody-should-be-using-this testing release of Debian, and some really bad timing from Manjaro got it in there, but the number of actually compromised linux systems was probably a few dozen.

@@oasntet Bvp47 was in Linux for 10 years (even after being submitted to "Virustotal" in 2013)...

@@Munenushi BVP47 was not _in_ Linux. It never shipped with a linux distro, because it is a rootkit (not a backdoor) that needs to be installed via some other vulnerability. When Pangu found it, it was notable not because it was there in plain sight for ten years, but because they finally found it in the wild and also identified the parties responsible.

Good point. That was kind of the wild west. Still, I'm not sure I'd change that by the way, and they haven't seen fit to either, so I could argue it was the right choice :-)

@@DavesGarage The FAT (File Allocation Table) design is brilliant; extremely efficient and even possible to manually fix some problems in the case of cross-linked files. This was at a time when a "full gallon" computer had 64 kilobytes of RAM and the operating system needed to function in about 16 kilobytes or less. Anyway, this is why many embedded systems such as my oscilloscope still use FAT formatted USB storage. It is simple and efficient (and the patent expired! Anyone can use it; it is nearly universal).

@@thomasmaughan4798 and fun for a joke. First FAT Table Corrupted. Wanna Try The Skinny One?

@@spvillano "First FAT Table Corrupted" While a frequent occurrence, it was also fairly easy to manually fix. Of course, by then one of the cross-linked files was demolished but you could decide which one to keep.

So true, and we would never know

Once a liar, always a liar. Dave (and his company) literally admitted "making false claims that the computer is at risk" in the court among other things. That was not the only thing, but I'm not going to test if he blacklisted keywords related to his heroic deeds, just google quoted text (include quotes) or his name on hackernews or reddit. I'm not even sure Hanlon's razor is applicable to this video.

Not to mention that IP accessable boards are in many servers, and some of those have holes you can drive trucks through.

@@javabeanz8549 Yes, Dell iDRAC for example, had some major issues early on, as did HP iLO did in its initial variants. This is why the such 'lights out' access methods of servers has to always be (recommended) on an 'administrative' rail that is highly audited/monitored, just as access to VM management infrastructure (now) is.

And just think, even our cars are getting connected now. Isn't it great?

@@darkprinc979 not only are they connected, they are becoming tattletales... soon the cars will be able to check the speed limit of the zone you are in, and prevent you from exceeding that limit by more than ten miles per hour, and the cops will be able to just shut off your car, so you can't run from them. So what happens when criminals figure out how to hijack these "features?"

@@javabeanz8549 Don't forget "anti-drunk driving measures". I'm sure nothing could possibly go wrong with all of these "safety" features. Oh, and what happens when big brother doesn't like the opinions you've been expressing on the internet?

Nobody reads that shit and you know it. If it compiles, move on with your life.

absolutely. I have always treated makefiles and build scripts as source code - using the same rigour used for the code itself. The same applies to everything used in the build process - build systems should be rebuilt from the asset/version control repositories regularly. Of course, many devops teams have a view that they don't have to eat their own dogfood, but they are very very wrong. I gues thats the main difference between devops and SCM (remember that?)

@@mithrandirthegrey7644 then they shouldn’t be in charge of vetting code changes.

Well I'll bet they'll start now.

This video missed the fact that the malicious build scripts that injected the malicious code only exists in the release tarball, not the Git source. This is why you would not see it if you just browse the source code itself. What we need to do is to have a reproducible way to generate the release tarballs and have those be checked routinely and automatically to verify that they match the source code. This is harder to do for binary releases (since you need reproducible builds) but for release tarballs for source code it should definitely be done.

And don't forget closed source projects heavily depend on open source projects. They are equally exposed

This didn't even affect Arch Linux, one of the most up to date distros

And I think that make build script wasn't even on the repo. So no commit. Afaict.

It is when it went from less than 300ms.

In gaming 500ms per frame would mean you only get 2 FPS. Thats very noticeable.

Lol

As an Android user, same.

He made tens of millions off that. He loves Microsoft.

We just pretend that doesnt exist

You don't have to use Windows or Outlook then. There are alternatives.

I think he left Microsoft before any of that

Do it. Honestly before windows 11 I would say that windows was better for the user simply because it had consistent UI for the settings. Now honestly I feel more consistency with KDE than with windows (also new settings menu in windows is bad).