 Hello and welcome to whatever this is. So I've been missing for a couple of weeks, and what do we have? Well, apparently the biggest attack on Linux I've ever seen, and somebody's entire home folder got deleted after downloading a global theme. Complete and utter chaos. Great. So let's start right away with the global theme thing, since it's the closer to kitty and I know more of it. Firstly, how? Just how could something like this have happened? Well, most themes on the kitty store are actually not just themes, like the color scheme is just a file with a lot of colors and the plasma theme is a folder with a lot of SVGs and so on. However, some themes require to change the behavior of the components too. Take the login animation theme. That one needs to be able to set a custom animation, and then there's the SDDM theme to customize the login screen, which also might want to move things around a bit. All of that, which requires changes to the behavior of components, is usually implemented via QML files, which is a markup language that uses JavaScript under the hood. Which runs code? Of course, it's not just about the themes. Widgets are another example of something that necessarily has to run code. The main difference is that you know we just can run code, but you don't really know that themes do too. Finally, we have global themes. Most people think of those as a sort of collection of different specific themes, like a plasma theme and a color scheme and a cursor theme and so on. But truly, they can be much more complex than that. Kitty Plasma is extremely modular under the hood, and global themes are allowed to use that modularity, however they want, and much more than you would expect. You shouldn't be surprised if they were even able to replace your entire desktop with some custom component. So, necessarily, they run code as well, mostly through QML, so what happened? Well, it seems like the theme on the store had some JavaScript code to handle a temporary config folder. This folder would be then deleted, maybe after installation. The directory of this config folder is specified in a file in a theme, and everything was fine. However, someone decided, someone else decided to fork that theme, and accidentally, they did not set a new config directory. Thus, the code that handled the config directory said, hey, let's just delete the config directory, and according to our file, the config directory is set to nothing. So let's just wipe out the home folder, yeah. It's not the first time such a mistake is made, even from reviewed packages from your distributions. Another important incident was this theme client, which, again, through a badly guarded RM command, accidentally deleted all user data under certain circumstances. The faulty global theme has been removed within hours, and it was a recent theme with very few actual installs. The KD store is up and running since something like 10 years ago, at least, and this is the first time something like this happens. Please note that you can still absolutely download safely color schemes, plasma themes, cursor themes, and more, since those do not run any code. And that by simply checking for a few genuine reviews of global themes, you can still safely install those two. Still, this shouldn't have happened. Discussion immediately began on how to address this and make sure that it never happens again. Firstly, the warning message. Whenever you enter the store in KD Plasma, there's a bit of a little message at the top that tells you, hey, all of this content comes from users like you, and it was interviewed by your distra, so be careful. It's pretty clear that this message can work for Plasma theme, where a bad theme could make your desktop ugly or even unusable, but global themes running arbitrary code, that's not enough. KD Dell developers have immediately added a new property, sorry, distinguishing the safe and unsafe store sections, and they have implemented new warning messages, much more noticeable and much more clear about the fact that, yes, underviewed code is being run. Secondly, there's active efforts by Marco, I think, in making sure that the global themes won't be able to run arbitrary code. This would mean losing a lot of cool functionalities, like SDDM themes and widgets, but for global themes, of course. But you can still install those extra carefully by yourself instead of being part of the global theme. This way global themes would become a safe category on the store as well. The lock screen is already out, and that was probably one of the most dangerous components, and more might be removed in the future. Thirdly, when you click on apply after selecting a global theme, there's a dialogue that asks you, hey, which components exactly would you like to apply? And you can choose. Well, we would like to add a clear warning message telling you which components are safe to apply and which ones currently run arbitrary code. The latter would probably be unticked by default, thus giving you the option to select them if you want, but still preserving a safe install out of the box. And fourth, the KT Store dialogue in KT Plasma currently gives you all the content sorted by the most recent. This is cool because you see different themes and widgets every time you open it, but also probably we should display the themes and widgets with the most downloads or the highest rating first. This way, the themes that you are most likely to click are also the safest ones. By the way, here's an entire discussion, no, sorry, by the way, there's an entire discussion that could be about how the store manages the ratings, but we don't have time for that right now. And finally, there's the long-term big deal. Can we actually review manually all of the content submitted to the KT Store? At least the potentially dangerous one. This is a big question and I would like to apologize, because I initially thought this was completely impossible and even strongly argued against it in Brody Robertson videos about this in the comments. And however, other KT developers have the opinion that this is currently feasible, and let's see what it would take. So firstly, it would still, I think, be impossible to review all the current theme and widgets on the store. However, most of them are Plasma 5 widgets and what we could do is to hide all widgets or themes and ask them to resubmit them with Plasma 6 as the goal platform, and we could start looking into the already ported Plasma 6 items. We should expect something like two to four widgets or themes to be submitted each week, which sounds feasible to manually review. However, this would have to be done in collaboration with Plink, which is the service that actually manages the store and due to the complexity of the idea, probably somebody would have to get paid to do it and paid by whom is more of a long-term or a medium-term goal right now. This is as much in-depth as I can go with the time that I get with this format. I hope I managed to convince you, or some of you, that some parts of the store are still safe to use and that we KD developers are taking this issue very seriously, even though, yes, I acted a bit not the correct way in Brody Robertson comments. And we are actively working on steps to make sure it doesn't happen again. Before I switch to the next segment though, I would like to remind all of you that, even if you think this is just a video or just a livestream, all of this takes work. We're talking about hours of research, writing often, hours of recording, and for most videos, there's also the editor's work. Some videos are written by me, some are even done by external collaborators that are more knowledgeable than me in some areas. All of this takes time, effort, and, most importantly, money. I have a donation goal of 1,000 euros every month and currently I have 420 bucks from Patreon, which is great. You can join there. You'll be able to see all the scripts in advance. You can write me messages. I usually respond. There's a dedicated community for that and you'll be supporting this show. If you prefer, by the way, there's also LibraPay, open source, Co-Fi, which takes less fees, along with the option to just send me some money over people. I'm extremely thankful to all of those people who donate and allow me to experiment with this channel as much as I do, pretty much without sponsorships. And by the way, I've been told that it's somebody's, like, some donor's birthday in just less than two hours, so happy early birthday, I guess. Finally, let's get to Aspley, because yes, that's the name that somebody gave to the backdoor that was introduced in the latest version of the XZ, is that how you pronounce it? Compression package is just, wow. I know that you already know about it, because everybody's been talking about it, but let's do a deep dive, because it's worth it. Given that I'm less of an expert here, the following is mostly inspired by articles from Wired, bohas.org, ducani.org, which is the project of the maintainer, raheve.substack.com, and more. All sources will be linked in the video description as soon as the livestream ends, so let's start at the beginning. XZ is a file format designed by developer Lasik Collin et al. between 2005 and 2008. This is now a package used pretty much in any modern distribution to compress and uncompress archives. If you use Macintosh with Umbrew, you probably also use that package. And yet, as it is customary in the open source world, at least, its development is severely underfunded and mostly driven by just Lasik Collin, who, as we will see, seems to be occasionally overwhelmed by the amount of work necessary to maintain it. In this context, a developer or a group of developers under the name of Jia Tan, started contributing to the project. They had already done a pull request to live archive and their late 2021 patches seem genuine. Those patches are sent to the XZ Devil mailing list and Collin merges them in this project. In April of 2022, Jia Tan sends another patch to the mailing list, and yet again, this patch is valid and innocuous. After a few days, another user under the name of Jigar Kumar, I don't know how it's pronounced, complains that the patch has not yet been merged. They do so in a quite aggressive tone. Patches spent years on this mailing list. 5.2.0 release was seven years ago. There is no reason to think that anything is coming soon. In May, the patch by Jia hasn't still been merged. Again, Kumar says, over one month and no closer to being merged. Not a surprise. And yet again, in June, is there any progress on this? Jia, I see you have recent comments. Why can't you just commit this yourself? Another developer under the name of Desin Hans sends an email to the same mailing list asking whether Java is supported. It is, but not actively so. Collin apologizes. It's clear that my resources are too limited, thus the many emails waiting for replies. So something has to change in the long term. Kumar also sends another email to this new thread. Yet again, aggressively pressuring calling. Progress will not happen until there is a new maintainer. XZ4C has sparse commits log 2. Daniels, you are better off waiting until a new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this. And we finally get to a key email sent by Collin. I haven't lost interest, but my ability to care has been fairly limited mostly due to long term mental health issues, but also due to some other things. Recently, I've worked off-list a bit with Zhiyotan on XZutils and perhaps he will have a bigger role in the future, we'll see. It's also good to keep in mind that this is an unpaid hobby project. Well, the plot twist here is that we have reasons to believe that developers Jigar Kumar and Dennis Hans do not actually exist, but rather they are new aliases for the same group behind Zhiyotan. Their emails are never to be seen again in the internet. Not even in data breaches and they all have a similar style and of course, none of them has reached out for comment to anyone after all of this has happened. Most likely what we are seeing is an example of a social engineering attack. Collin is already overwhelmed and the group wants to gain access to the project more quickly, so they start pressuring Lasse as much as possible, even bringing him to admit he has long term mental health issues. Anyway, anyhow, this works. After further aggressive emails, we get to this statement from Collin. As I have hinted in earlier emails, Zhiyotan may have a bigger role in the project in the future. He has been helping me off-list and is practically a commentator already. In any case, some change in maintenance is already in progress, at least for XZutils. And thus, in January of 2023, Zhiyotan is able to merge commits on his own, indicating that he has gained trust from Collin. Around March, Zhiyotan commits the testing infrastructure that will be the key component in the backdoor later on. Interestingly enough, this part of the code does not seem to come from Zhiyotan themselves. Rather, it was made by a developer called Hans Janssen. Except of course, Hans Janssen's email also has very little activity, except for this change, and they only appear again to push for the compromised version of XZ to be included in Debian. Thus, they also are most likely a fake alias for the same developer group. In January of 2024, Zhiyotan also moved the website of the XZ project to a sub-domain of tukahani.org, XZ.tukahani.org. This has a DNS record that points to the GitHub pages, thus to which, sorry, Zhiyotan has access, thus it can now change the website too. Finally, in February, the attack begins. Due to this project being a decompressed package, the testing directory often contains raw binary code to test with. According to the readme, long before Zhiyotan started contributing to the project, many of the files have been created by hand with the next editor, thus there is no better source code than the files themselves. On the 23rd of February, Zhiyotan introduces new binary files to use as tests, with names like good, large, compressed, LZMA and so on. However, they contain a backdoor, which, however, does not do anything on its own. The following day, though, it tags and builds a new version of the project, version 5.6.0, and it publishes a tar package. However, this package actually contains one extra file that is not found in the source code of this project, build2has.m4, which adds the backdoor when building a DEB RPM package. It's not yet completely understood what this backdoor does, but it seems like it would give external users complete control over the affected machine via SSH. Zhiyotan starts emailing Richard W.M. Jones, who works at Red Hat, asking for Fedora 40 to include the compromised XZ version. At the same time, Dabian also adds that package to its unstable branch. Now, this is actually somewhat exciting, because this is the moment that Zhiyotan has been working towards for years, literally. And yet, a couple of things happen that put the whole operation at risk. Firstly, a developer called Technoravver sends a pull request to stop linking lib as an A, I managed to say it, to lib system D. This probably would have defeated the backdoor, which adds some extra pressure to Zhiyot to get it merged before such a thing happens. Over the Red Hat side of things, they start seeing valgrind errors in lib-lzma function that is the entry of the backdoor. Developer Russ Cox puts it perfectly when he says the race is on to fix this before the Linux distributions dig in too deeply. Even worse, the above mentioned lib system D pull request is also merged. Another race is on to get lib-lzma backdoor before the distros break the approach entirely. Zhiyotan, on the 9th of March, commits a couple of things. Firstly, they commit a fake fix to the valgrind issue. The real issue is in the backdoor, but of course they cannot admit that, so they do this fake fix as a misdirection. But at the same time, they also update the backdoor files. They do this by updating the binary test files, saying that, and I quote, the original files were generated with a random local to my machine. To better reproduce these files in the future, a constant set was used to recreate these files. None of this is true. Regardless, a new version of XZ is immediately published, version 5.6.1, which indeed fixes the Red Hat valgrind issue. Hans Janssen, who had written the testing infrastructure, also comes back and files at Debian bug to ask for XZ utils to be upgraded to the latest version, 5.6.1. Other developers that seem just as fake as Janssen show up to support him and, couple of days later, Debian indeed updates the package. On the following day, Zhiyotan themselves file a bug to Ubuntu, asking for the package to be updated to 5.6.1 too. And finally, on the 28th, developer Andres Frond discovers the bug, or rather the backdrop. He immediately notifies Debian and other distributions. Debian rolls back to XZ 5.4.5, Arch Linux starts building XZ from Git instead of relying on the Turbo, and Red Hat assigns the critical vulnerability code of 2024-1394. The following day, Andres Frond also publishes his findings publicly. Of course, there's currently great speculation over the exact identity of Zhiyotan. Most qualified security experts seem to think that this could have only been done by a group of people, which of course raises some important questions, such as, who paid them to do this? We're talking about hiring a team of very experienced developers for years. An important part of the speculation relies on analyzing the commit history of Zhiyotan, especially the time zone and commit times. Almost all of the commits by Zhiyotan use the Chinese time zone, plus 0800. However, some commits switch between UTC plus 2 and UTC plus 3, depending on the daylight savings time, which would indicate for Zhiyotan to be working in Eastern Europe. If this were true, then the commit times would generally start from 9am and end in 6pm, which would correspond to a standard day job. On top of that, it seems like Zhiyotan often worked during multiple Chinese festivities, but there are no commits from European festivities, such as Christmas or New Year's Eve. However, the commits with those different time zones were merged by Colin, who also is in the UTC plus 2 or 3 time zone. It could be that Colin accidentally changed the time zone of the commit, but please note that other commits merged by Colin preserve the original plus 0800 time zone, so it's not so clear what the right option might be. According to a statement gave by Dave Atial, I tell, a former NSAE hacker and defender of cyber security firm immunity, twired, the majority of clues lead back to Russia, and specifically Russia's APT-29 hacking group, widely believed to work for Russia's foreign intelligence agency. Of course, it could be absolutely someone else, another group most likely. And yet again, I would like to thank Russ Cox, who has provided the most in-depth timeline and twitch I've extensively used to write this section. One final key point that I'd like to make is, yes, we've been very lucky to have discovered this attack. This could have gone wrong. Many rightfully says, hey, surely there must be lots of attacks like this that go unnoticed, and if anybody can just start maintaining some understaffed project and in a few years just throw a back door in, then surely there's lots of people doing it. But let's remember that this was a very long term, very complex and very coordinated technical and social attack. Most likely it required a lot of effort and resources, which not everybody has, not even organized groups. Nonetheless, it could be that other similar attacks happened or are currently underway, and we just don't know about them. The words aren't safe. Has any other attack vector been found on XZ? Not currently, we did find some, we I didn't do anything, but the developer community did find some other commits that were preparing probably for other attacks, but you know, everything was discovered in time, so we got lucky maybe. That was everything for this second segment.