Backdoor in upstream xz/liblzma leading to SSH server compromise (opens in new tab)

https://github.com/tukaani-project/xz/pull/53

I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is "authored by" Lasse Collin in the git metadata, but the code actually came from Hans Jansen: https://github.com/tukaani-project/xz/commit/ee44863ae88e377...

> Thanks to Hans Jansen for the original patch.

There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would've masked the payload in many testing and debugging environments.

The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another "<name><number>" account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: https://www.mail-archive.com/search?l=debian-bugs-dist@lists...

jonathanspw1y ago

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

userbinator1y ago

because of it's "great new features"

"great" for whom? I've seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It's no different from claiming a "better experience" or similar.

mongol1y ago

Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...

smeehee1y ago

Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…

drazk1y ago

After reading the original post by Andres Freund, https://www.openwall.com/lists/oss-security/2024/03/29/4, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?

api1y ago

I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.

kapouer1y ago

Github accounts of both xz maintainers have been suspended.

thayne1y ago

Do you know if it was actually the commit author, of if their commit access was compromised?

LispSporks221y ago

Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.

gigatexal1y ago

Name and shame this author. They should never be allowed anywhere near any open projects ever again.

nodesocket1y ago

Can legal action be taken against the author if it's found he maliciously added the backdoor?

landownersubgrp1y ago

It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.

yieldcrv1y ago

I wonder who the target was!

menomatter1y ago

his account is active again on github https://github.com/JiaT75

KingOfCoders1y ago

Sleeper.

Jommi1y ago

the account was either sold or stolen

move-on-by1y ago

Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.

> If you discover a security vulnerability in this project please report it privately. *Do not disclose it as a public issue.* This gives us time to work with you to fix the issue before public exposure, reducing the chance that the exploit will be used before a patch is released.

Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.

ncr1001y ago

Security Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable?

It's a SEVERE issue, to my mind, and 90 days seems too long to me.

xyst1y ago

90 day dark window for maintainers is SOP though. Then after 90 days, it’s free game for public disclosure

szundi1y ago

How many of people like this one exist?

tw041y ago

Honestly it seems like a state-based actor hoping to get whatever high value target compromised before it's made public. Reporting privately buys them more time, and allows them to let handlers know when the jig is up.

Aissen1y ago

Looks like one of the backdoor authors even went and disabled the feature the exploit relied on directly on oss-fuzz to prevent accidental discovery: https://social.treehouse.systems/@Aissen/112180302735030319 https://github.com/google/oss-fuzz/pull/10667

But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941

miduil1y ago

This is getting addressed here: https://github.com/google/oss-fuzz/issues/11760

nialv71y ago

This in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...

throwaway2901y ago

and that was in mid 2023. Very funny that Wikipedia on this issue says

> It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised

Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene

arp2421y ago

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Would be interesting to see what's going on here; the person who did the releases has done previous releases too (are they affected?) And has commits going back to 2022 – relatively recent, but not that recent. Many are real commits with real changes, and they have commits on some related projects like libarchive. Seems like a lot of effort just to insert a backdoor.

Edit: anyone with access can add files to existing releases and it won't show that someone else added it (I just tested). However, the timestamp of the file will be to when you uploaded it, not that of the release. On xz all the timestamps of the files match with the timestamp of the release (usually the .tar.gz is a few minutes earlier, which makes sense). So looks like they were done by the same person who did the release. I suspected someone else might have added/altered the files briefly after the release before anyone noticed, but that doesn't seem to be the case.

bodyfour1y ago

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563

I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.

eigenvalue1y ago

Every single commit this person ever did should immediately be rolled back in all projects.

https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...

dist-epoch1y ago

> they have commits on some related projects like libarchive

Windows started using libarchive to support .rar, .7z, ...

ptx1y ago

Couldn't the autoconf soup be generated from simpler inputs by the CI/CD system to avoid this kind of problem? Incomprehensible soup as a build artifact (e.g. executables) is perfectly normal, but it seems to me that such things don't belong in the source code.

(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)

AeroNotix1y ago

Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.

sslayer1y ago

I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account

bawolff1y ago

I mean, a backdoor at this scale (particularly if it wasn't noticed for a while and got into stable distros) could be worth millions. Maybe hundreds of millions (think of the insider trading possibilities alone, not to mention espionage). 2 years doesn't seem like that much work relative to the potential pay off.

This is the sort of case where america's over the top hacking laws make sense.

WesolyKubeczek1y ago

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup

If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?

stabbles1y ago

How about wheels in the python ecosystem

IshKebab1y ago

Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.

bonyt1y ago

For those panicking, here are some key things to look for, based on the writeup:

- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.

- A debian or RPM based distro of Linux on x86_64. In an apparent attempt to make reverse engineering harder, it does not seem to apply when built outside of deb or rpm packaging. It is also specific to Linux.

- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.

Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

It is possible there are other flaws or backdoors in liblzma5, though.

pdw1y ago

Focusing on sshd is the wrong approach. The backdoor was in liblzma5. It was discovered to attack sshd, but it very likely had other targets as well. The payload hasn't been analyzed yet, but _almost everything_ links to libzma5. Firefox and Chromium do. Keepassxc does. And it might have made arbitrary changes to your system, so installing the security update might not remove the backdoor.

treffer1y ago

Ubuntu still ships 5.4.5 on 24.03 (atm).

I did a quick diff of the source (.orig file from packages.ubuntu.com) and the content mostly matched the 5.4.5 github tag except for Changelog and some translation files. It does match the tarball content, though.

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g.

   +#: src/xz/args.c:735
   +#, fuzzy
   +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"

There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.

[Edit] formatting

blcknight1y ago

> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

NotPractical1y ago

> If you're not on a rolling release distro, your version is probably older.

Ironic considering security is often advertised as a feature of rolling release distros. I suppose in most instances it does provide better security, but there are some advantages to Debian's approach (stable Debian, that is).

idoubtit1y ago

The article gives a link to a simple shell script that detects the signature of the compromised function.

> Running OpenSSH sshd from systemd

I think this is irrelevant.

From the article: "Initially starting sshd outside of systemd did not show the slowdown, despite the backdoor briefly getting invoked." If I understand correctly the whole section, the behavior of OpenSSH may have differed when launched from systemd, but the backdoor was there in both cases.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

fransje261y ago

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

hostyle1y ago

$ dpkg-query -W liblzma5

liblzma5:amd64 5.4.1-0.2

rathermucker1y ago

Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI

1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

Epa0951y ago

I hope Lasse Collin is doing OK! Here is a older message from him [1]

"I haven't lost interest but my ability to care has been fairly limited mostly due to longterm mental health issues but also due to some other things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and perhaps he will have a bigger role in the future, we'll see.

It's also good to keep in mind that this is an unpaid hobby project. "

Github (Microsoft) are in a unique position to figure out if his account is hacked or not, and find a way to reach him. I hope they reach out and offer him some proper support! Economic support (if that's needed), or just help clearing his name.

This is another tale of how we are building multi trillion dollar industries on the back of unpaid volunteers. It's not github 'job', and many other organisations have benefited even more from Lasses work, but they are in a unique position, and would be literally pocket change for them.

cbolton1y ago

In a movie his mental health issues would likely have been caused intentionally by the attacker, setting the stage for the mole to offer to step in just at the right time. Seems a bit far fetched in this case though for what looks like a tangential attack.

slavik811y ago

Lasse appears to be active and working on undoing the sabotage. https://git.tukaani.org/?p=xz.git;a=blobdiff;f=CMakeLists.tx...

delfinom1y ago

He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.

k8svet1y ago

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.

I was actually telling my dad about this. I have a project, 500+ users, not quite root access, but enough to cause serious damage. I can think of at least one covert way to backdoor the binary artifacts from it.

About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.

syslog1y ago

Relevant xkcd:

A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz

About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.

arp2421y ago

As a bit of an aside, I would never accept a PR like this, and would always update $large_vendored_dependency myself. This is unreviewable, and trivial to insert any backdoor (unless you go through the motions of updating it yourself and diffing, at which point the PR becomes superfluous). I'd be wary even from a well-known author unless I knew them personally on some level (real-life or via internet). Not that I wouldn't trust them, but people's machines or accounts can get compromised, people can have psychotic episodes, things like that. At the very least I'd like to have some out-of-band "is this really you?" signal.

This is how I once inserted a joke in one of our (private) repos that would randomly send cryptic messages to our chat channel. This was pretty harmless and just a joke (there's some context that made it funny), but it took them years to find it – and that was only because I told them after I quit.

That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

jaredallard21y ago

Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...

Bromeo1y ago

I don't want to read too much into it, but the person (supposedly) submitting the PR seems to work at 1Password since December last year, as per his Linkedin. (And his Linkedin page has a link to the Github profile that made the PR).

pinko1y ago

> it's not the same GitHub account as upstream

This is valuable information, and a sign that this may be the tip of an iceberg.

cbmuser1y ago

There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

icambron1y ago

IMO your prior on this should be that it's most likely just someone innocently updating a dependency.

5kg1y ago

The backdoor (test binary blob and autoconf) is not part of the pull request.

baxtr1y ago

Suddenly anything like that becomes super suspicious.

I wonder how this will affect the OS community in general.

squigz1y ago

Internet detectives at work in this thread!

cf100clunk1y ago

  I am *not* a security researcher, nor a reverse engineer.  There's lots of
  stuff I have not analyzed and most of what I observed is purely from
  observation rather than exhaustively analyzing the backdoor code.

I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.

anarazel1y ago

FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).

[0] https://github.com/microsoft/vcpkg/issues/37197

bonzini1y ago

For what it's worth the author is a PostgreSQL committer, he's not a security researcher but he's a pretty damn good engineer!

vhiremath41y ago

Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.

dang1y ago

Related ongoing threads:

Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718

FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068

Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673

0xthr0w41y ago

Out of curiosity I looked at the list of followers of the account who committed the backdoor.

Randomly picked https://github.com/Neustradamus and looked at all their contributions.

Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago.

[1] https://github.com/microsoft/vcpkg/pull/37199

sroussey1y ago

OMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects.

Example: https://github.com/duesee/imap-flow/issues/96

https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev

asmor1y ago

Hey, I remember this guy! Buddy of someone who tried to get a bunch of low quality stuff into ifupdown-ng, including copying code with an incompatible license and removing the notice. He's in every PR, complaining the "project is dead". He even pushes for the account to be made "team member".

He follows 54k accounts though, so it may indeed just be coincidence.

neustradamus1y ago

Dear @0xthr0w4, do you attack me because I have requested the XZ update?

Do not mix, I am not linked to the XZ project.

perihelions1y ago

Imagine a more competent backdoor attempt on xz(1)—one that wouldn't have been noticed this quickly. xz is everywhere. They could pull off a "reflections on trusting trust": an xz which selectively modifies a tiny subset of the files it sees, like .tar.xz software tarballs underlying certain build processes. Not source code tarballs (someone might notice)—tarballs distributing pre-compiled binaries.

edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]).

[0] https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")

joeyh1y ago

A backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.

Phenylacetyl1y ago

The same authors have also contributed to Zstd

nolist_policy1y ago

deb packages are xz compressed...

https://hachyderm.io/@joeyh/112180715824680521

pfortuny1y ago

Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.

jnxx1y ago

And, Joey Hess has counted at least 750 commits to xz from that handle.

This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

hangonhn1y ago

I imagine it might be easier to just compromise a weakly protected account than to actual put in a 2 years long effort with real contributions. If we mandated MFA for all contributors who contribute to these really important projects then we can know with greater certainty if it was really a long con vs. a recently compromised account.

ametrau1y ago

Probably a state actor. You can look far into the future when you’re working for the party.

dang1y ago

(I detached this subthread from https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)

lucasRW1y ago

More likely that the account of that dev was breawched, dont you think ?

moritonal1y ago

Warning, drunk brain talking. But a LLM driven email based "collaborator" could play a very long gMw adding basic features to a code made whilst earning trust backed by a generated online presence. My money is on a resurgance in the Web of Trust.

https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/

the_errorist1y ago

Looks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188

Also, some info here: https://tukaani.org/xz-backdoor/

jwilk1y ago

Or if you can't stand the lkml.org UI:

bawolff1y ago

The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.

Makes you wonder what more competent actors can do.

rwmj1y ago

I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.

aidenn01y ago

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...

yard20101y ago

You must mean, "Makes you wonder what more competent actors are doing"

pinko1y ago

s/can do/have done/

gmnon1y ago

Funny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :)

bombcar1y ago

This is extremely suspicious.

It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it.

Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.

ncr1001y ago

Also interesting, to me, how the GMail account for the backdoor contributor ONLY appears in the context of "XZ" discussions. Google their email address. Suggests a kind of focus, to me, and a lack of reality / genuineness.

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

ui2RjUen875bfFA1y ago

those pipe usages are quite suspicious

pipeing into this shell script which now uses "eval"

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

i guess this will be revisited and removed soon

bonzini1y ago

"started to cc himself" seems to be simply "contributing to a new project and not having git-send-email fully set up". By default git-send-email Cc the sender, though in practice it's one of the first options one changes.

vhiremath41y ago

My favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes."

You only get this kind of humility when you're working with absolute wizards on a consistent basis.

asveikau1y ago

That's completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn't look malicious at all, it looks like build script gibberish.

agwa1y ago

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

[0] https://news.ycombinator.com/item?id=39154825

omoikane1y ago

It looks like an earlier commit with a binary blob "test data" contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the "Compromised Repository" section.

Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?

ptx1y ago

The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.

A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".

Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for "eval", unless the feature you're implementing really is "take a piece of arbitrary code from the user and execute it".

zb31y ago

Yeah, now imagine they succeeded and it didn't cause any performance issues...

Can we even be sure no such successful attempt has already been made?

tetromino_1y ago

A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-"test" files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.

20after41y ago

> "Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system. Unfortunately the latter looks like the less likely explanation, given they communicated on various lists about the "fixes" mentioned above."

Crazy indeed.

Decabytes1y ago

So when are we going to stop pretending that OSS maintainers/projects are reaping what they sow when they "work for free" and give away their source code away using OSS licensed software, while large companies profit off of them? If they were paid more (or in some cases even actually paid), then they could afford to quit their day jobs, reducing burn out, they could actually hire a team of trusted vetted devs instead of relying on the goodwill of strangers who step up "just to help them out" and they could pay security researchers to vet their code.

Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.

kortilla1y ago

Paid people get burnt out as well and they are just as likely to accept free help as an unpaid person.

qwery1y ago

> So when are we going to stop pretending ...

I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

I don't think moving towards unfree software is a good way to make free software more secure. It shouldn't be a surprise that proprietary software is less likely to be exploited in this way simply because they don't accept any patches from outside of the team. What you want is more people that understand and care about free software and low barriers to getting involved.

delfinom1y ago

OSS maintainers aren't reaping anything. Most OSS licenses say the software is provided without warranty.

thesnide1y ago

The discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

0x01y ago

Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!

And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity

https://youtu.be/IfhMILe8C84

nammi1y ago

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598

oasisaimlessly1y ago

For anyone else feeling some deja vu about ifunc / Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].

[2]: https://news.ycombinator.com/item?id=39733185

fullstop1y ago

I get the feeling that a number of the comments are all the same person / group.

rpigab1y ago

I'd love to be at Microsoft right now and have the power to review this user's connection history to Github, even though VPN exists, many things can be learned from connection habits, links to ISPs, maybe even guess if VPNs were used, roundtrip time on connections can give hints.

I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

qecez1y ago

> I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

Which is why there's probably nothing remotely interesting in them logs.

RockRobotRock1y ago

Nah. I'm sure Microsoft got a call from the alphabet boys and nobody, not even internal employees are allowed to look at the logs right now.

megous1y ago

Oh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.

optimalsolver1y ago

I'm guessing Microsoft just got a call from the Government telling them not to look too deeply into it.

alpb1y ago

That’d be illegal for an employee to do.

https://github.com/tukaani-project/tukaani-project.github.io...

alright25651y ago

> Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored.

The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271

rom1v1y ago

Btw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others):

- https://github.com/libusb/libusb/issues/1468#issuecomment-19...

- https://github.com/orgs/community/discussions/6003

o11c1y ago

For a long time, there was one legitimately annoying disadvantage to the git-generated tarballs though - they lost tagging information. However, since git 2.32 (released June 2021; presumably available on GitHub by August 2021 when they blogged about it) you can use `$Format:%(describe)$` ... limited to once per repository for performance reasons.

kzrdude1y ago

Jia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all

sroussey1y ago

GitHub/Microsoft likely has a backup. I’d be getting those out about now.

zamalek1y ago

Bad move. Destroying evidence is a felony.

[1] https://git.phial.org/d6/xz-analysis-mirror

zh31y ago

Comment from Andres Freund on how and why he found it [0] and more information on the LWN story about the backdoor. Recommend people read this to see how close we came (and think about what this is going to mean for the future).

[0] https://lwn.net/Articles/967194/

eBombzor1y ago

That man deserves a Nobel Prize

dhx1y ago

A mirror of the offending repository created by someone else is available at [1]. GitHub should be keeping the evidence in the open (even if just renamed or archived in a safer format) instead of deleting it/hiding it away.

The offending tarball for v5.6.1 is easier to find, an example being.[2]

m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3]

[2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz

[3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...

xyzzy_plugh1y ago

This gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

Definitely looking like they were most likely some sort of state actor. This is very well done and all in plain sight. It's reassuring that it was discovered but given a simple audit of the release build artifacts would have raised alarms, how prevalent is this behavior in other projects? Terrifying stuff.

wood_spirit1y ago

A lot of eyes will be dissecting this specific exploit, and investigating this specific account, but how can we find the same kind of attack in a general way if it’s being used in other projects and using other contributor names?

treffer1y ago

1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.

2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.

3. This was caught through runtime inspection. It should be possible to halt any Linux system at runtime, load debug symbols and map _everything_ back to the source code. If something can't map back then regard it as a potentially malicious blackbox.

There has been a strong focus and joint effort to make distributions reproducible. What we haven't managed though is prove that the project compromises only of freshly compiled content. Sorta like a build time / runtime "libre" proof.

This should exist for good debugging anyway.

It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.

Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.

mac-chaffee1y ago

Build-related fixes are only treating the symptoms, not the disease. The real fix would be better sandboxing and capability-based security[1] built into major OSes which make backdoors a lot less useful. Why does a compression library have the ability to "install an audit hook into the dynamic linker" or anything else that isn't compressing data? No amount of SBOMs, reproducible builds, code signing, or banning binaries will change the fact that one mistake anywhere in the stack has a huge blast radius.

[1]: https://en.wikipedia.org/wiki/Capability-based_security

Note that the malicious binary is fairly long and complex.

This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.

You could imagine a simple process which checks out the code, then runs some kind of entropy checker over the code to check it is all unminified and uncompressed source code, before finally kicking off the build process.

autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.

afiodorov1y ago

We should be able to produce a tar and a proof that tar was produced from a specific source code.

Quote from the article:

    That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.

Zero Knowledge virtual machines, like cartesi.io, might help with this. Idea is to take the source, run a bunch of computational steps (compilation & archiving) and at the same time produce some kind of signature that certain steps were executed.

The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.

The advantage of Zero-Knowledge technology in this case is that one doesn't need to repeat the computational steps themselves nor rely on a trusted party to do it for them (like automated build - that can also be compromised by the state actors). Just having the proof solves this trust problem mathematically: if you have the proof & the tar, you can quickly check source code that produced the tar wasn't modified.

JonChesterfield1y ago

The Guix full source bootstrap is looking less paranoid as time goes on

Avamander1y ago

More reproducible builds, maybe even across distributions? Builds based on specific commits (no tarballs like in this case), possibly signed (just for attribution, not for security per se)? Allow fewer unsafe/runtime modifications The way oss-fuzz ASAN was disabled should've been a warning on its own, if these issues weren't so common.

I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?

q3k1y ago

NixOS/Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd/liblzma, and the backdoor doesn't get configured in (only happens on .deb/.rpm systems).

jchw1y ago

It may not have really mattered much for NixOS:

> b) argv[0] needs to be /usr/sbin/sshd

For once, the lack of FHS interoperability is a benefit, if only on accident.

o11c1y ago

Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.

lostmsu1y ago

Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.

dandanua1y ago

That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.

bhaak1y ago

I looked at the differences between the GitHub repository and released packages. About 60 files are in a release package that are not in the repo (most are generated files for building) but also some of the .po files have changes.

That's devastating.

If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.

sesuximo1y ago

I think this is unfortunately very common practice

icommentedtoday1y ago

Why not `git archive`?

https://news.ycombinator.com/item?id=39872062

cryptonector1y ago

colanderman1y ago

The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Not sure what to make of this.

rany_1y ago

I think the reason is pretty obvious. They want you to waste more time after you've submitted the security report and maximize the amount of back and forth. Basically the hope is that they'd be able to pester you with requests for more info/details in order to "resolve the issue" which would give them more time to exploit their targets.

colanderman1y ago

That repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template:

    + or create a private Security Advisory instead.

Under a commit titled "Wrap text on Issue template .yaml files."

[1] https://github.com/tukaani-project/.github/commit/44b766adc4...

caelum191y ago

Potentially the purpose is that if someone goes to the effort to get those details together, they are more likely to send the same report to other trusted individuals. Maybe it was originally there to add legitimacy, then they got a report sent in, and removed it to slow the spread of awareness

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

20after41y ago

> "Docs: Simplify SECURITY.md."

Removes instructions about details relevant to security reports. Heh, nice one.

Tenobrus1y ago

It looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html

kzrdude1y ago

In various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary

kzrdude1y ago

Makes me wonder if he's an owner of the github organization, and what happens with it now?

CGamesPlay1y ago

Why has Github disabled the (apparently official) xz repository, but left the implicated account open to the world? It makes getting caught up on the issue pretty difficult, when GitHub has revoked everyone's access to see the affected source code.

https://github.com/tukaani-project/xz vs https://github.com/JiaT75

dzaima1y ago

The account has been suspended for a while, but for whatever reason that's not displayed on the profile itself (can be seen at https://github.com/Larhzu?tab=following). Repo being disabled is newer, and, while annoying and realistically likely pointless, it's not particularly unreasonable to take down a repository including a real backdoor.

5p4n9111y ago

The author (Jia Tan) also changed the xz.tukaani.org (actually the github.io, where the main contributor is, surprise, also them) release description to state all new releases are signed by their OpenPGP key. I'd guess that was one of the first steps to a complete project takeover.

I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.

weinzierl1y ago

The backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?

soneil1y ago

That's how I understand it. A build script that's in the releases tarballs but not the git repo, checks to see if it's being run as part of the debian/build or rpm build processes, and then injects content from one of the "test" files.

https://aws.amazon.com/security/security-bulletins/AWS-2024-...

elchief1y ago

"Amazon Linux customers are not affected by this issue, and no action is required. AWS infrastructure and services do not utilize the affected software and are not impacted. Users of Bottlerocket are not affected."

liveoneggs1y ago

The best part is everyone disabling security tests that started failing

ozgune1y ago

I read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :)

Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?

jaromilrojo1y ago

This is another proof that systemd is an anti-pattern for security: with its crawling and ever growing web of dependencies, it extends the surface of vulnerability to orders of magnitude, and once embraced not even large distro communities can defend you from that.

A malware code injection in upstream xz-tools is a vector for remote exploitation of the ssh daemon due to a dependency on systemd for notifications and due to systemd's call to dlopen() liblzma library (CVE-2024-3094). The resulting build interferes with authentication in sshd via systemd.

acdha1y ago

Please take the systemd trolling to Reddit. They likely targeted xz specifically because it’s so widely used but there are dozens of other libraries which are potential candidates for an attack on sshd, much less everything else which has a direct dependency unrelated to systemd (e.g. dpkg).

Rather than distracting, think about how the open source projects you use would handle an attack like this where someone volunteers to help a beleaguered maintainer and spends time helpfully taking on more responsibilities before trying to weaken something.

geggo981y ago

Actually you have a point. A collection of shell scripts (like the classical init systems) have obviously a smaller attack surface. In this case the attacker used some integration code with systemd to attack the ssh daemon. So sshd without systemd integration is safe against this specific attack.

In general, I’m not convinced that systemd makes things less secure. I have the suspicion that the attacker would just have used a different vector, if there was no systemd integration. After all it looks like the attacker was also trying to integrate exploits in owner libraries, like zstd.

Still I would appreciate it, if systemd developers would find a better protection against supply chain attacks.

saagarjha1y ago

This isn't Twitter you don't have to use hashtags

throwaway73561y ago

> systemd's call to dlopen() liblzma library (CVE-2024-3094)

That's technically wrong, but no surprise. Anti-systemd trolls usually don't understand technical details after all.

sn1y ago

For bad-3-corrupt_lzma2.xz, the claim was that "the original files were generated with random local to my machine. To better reproduce these files in the future, a constant seed was used to recreate these files." with no indication of what the seed was.

I got curious and decided to run 'ent' https://www.fourmilab.ch/random/ to see how likely the data in the bad stream was to be random. I used some python to split the data into 3 streams, since it's supposed to be the middle one that's "bad":

I used this regex to split in python, and wrote to "tmp":

    re.split(b'\xfd7zXZ', x)

I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes:

    $ ent tmp2 # bad file payload
    Entropy = 4.157806 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 48 percent.
    
    Chi square distribution for 48 samples is 1114.67, and randomly
    would exceed this value less than 0.01 percent of the times.
    
    Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
    Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
    Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
    
    $ ent tmp3 # urandom
    Entropy = 5.376629 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 32 percent.
    
    Chi square distribution for 48 samples is 261.33, and randomly
    would exceed this value 37.92 percent of the times.
    
    Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
    Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
    Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).

The data does not look random. From https://www.fourmilab.ch/random/ for the Chi-square Test, "We interpret the percentage as the degree to which the sequence tested is suspected of being non-random. If the percentage is greater than 99% or less than 1%, the sequence is almost certainly not random. If the percentage is between 99% and 95% or between 1% and 5%, the sequence is suspect. Percentages between 90% and 95% and 5% and 10% indicate the sequence is “almost suspect”."

supriyo-biswas1y ago

Now to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.

https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

0x01y ago

All these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io...

kn1001y ago

Here's a handy bash script I threw together to audit any docker containers you might be running on your machine. It's hacky, but will quickly let you know what version, if any, of xz, is running in your docker containers.

``` #!/bin/bash

# Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")

# Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")

    # Execute xz --version inside the container
    version=$(docker exec "$container" xz --version)

    # Write container name, image, and command output to a text file
    echo "Container: $container" >> docker_container_versions.txt
    echo "Image: $image" >> docker_container_versions.txt
    echo "xz Version:" >> docker_container_versions.txt
    echo "$version" >> docker_container_versions.txt
    echo "" >> docker_container_versions.txt

done

echo "Output written to docker_container_versions.txt" ```

Roark661y ago

Sadly this is exactly one of the cases where open source is much more vulnerable to a state actor sponsored attack than proprietary software. (it is also easier to find such backdoors in OS software but that's BTW)

Why? Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he. Also they have to be hiring in the right team etc. Your operative has to be in a different country, needs a CV that checks out, passports/ids are checked etc.

But to contribute to an OS project? You just need an email address. Your operative sends good contributions until they build trust, then they start introducing backdoors in the part of the code "no one, but them understands".

The cost of such attack is a lot lower for a state actor so we have to assume every single OS project that has a potential to get back doored had many attempts of doing so. (proprietary software too, but as mentioned, this is much more expensive)

So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.

alufers1y ago

Is that true? Large companies producing software usually have bespoke infra, which barely anyone monitors. See: the Solarwinds hack. Similarly to the xz compromise they added the a Trojan to the binary artifacts by hijacking the build infrastructure. According to Wikipedia "around 18,000 government and private users downloaded compromised versions", it took almost a year for somebody to detect the trojan.

Thanks to the tiered updates of Linux distros, the backdoor was caught in testing releases, and not in stable versions. So only a very low percentage of people were impacted. Also the whole situation happened because distros used the tarball with a "closed source" generated script, instead of generating it themselves from the git repo. Again proving that it's easier to hide stuff in closed source software that nobody inspects.

Same with getting hired. Don't companies hire cheap contractors from Asia? There it would be easy to sneak in some crooked or even fake person to do some dirty work. Personally I was even emailed by a guy from China who asked me if I was willing to "borrow" him my identity so he could work in western companies, and he would share the money with me. Of course I didn't agree, but I'm not sure if everybody whose email he found on Github did.

throwaway73561y ago

> Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he.

Or work for a third-party company that gets access to critical systems without any checks. See for example the incident from 2022 here: https://en.wikipedia.org/wiki/Okta,_Inc.

Or a third-party that rents critical infrastructure to the company (Cloud, SaaS solutions).

Scaevolus1y ago

It's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.

lpapez1y ago

So many security companies publishing daily generic blog posts about "serious supply chain compromises" in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.

I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

r0ckarong1y ago

> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.

CableNinja1y ago

Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.

Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.

Now when things go wrong, the company can point to the cert and go "it wasnt us, see we have the cert you told us to get and its up to date". And the company gets to wash their hands of liability (most of the time).

keepamovin1y ago

If you installed xz on macOS using brew, then you have

  xz (XZ Utils) 5.6.1
  liblzma 5.6.1

which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with

  brew upgrade xz

bawolff1y ago

> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That is actually a major point of a lot of corporate security measures (shifting risk)

[0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests

markus_zhang1y ago

That's the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.

gouggoug1y ago

List of pull request requesting the updating to liblzma 5.6.0 [0]

I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.

snickerer1y ago

When I search for "digital masquerade" on Google, the first result is a book with this title from the author Jia Tan. I assume that is how the attackers got their fake name. Or they think using this author's name is a joke.

dlenski1y ago

A lot of software (including https://gitlab.com/openconnect/openconnect of which I'm a maintainer) uses libxml2, which in turn transitively links to libzma, using it to load and store compressed XML.

I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…

But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.

enedil1y ago

This doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.

bananapub1y ago

> only to read and parse uncompressed XML…

how does libxml2 know to decompress something?

does it require you, as the caller, to explicitly tell it to?

or does it look at the magic bytes or filename or mimetype or something?

https://github.com/emirkmo/xz-backdoor-github

afh11y ago

Potentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609

youainti1y ago

Summary: "The upstream xz repository and the xz tarballs have been backdoored."

It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

Randalthorro1y ago

Since GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files:

For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)

yogorenapan1y ago

Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries

ParetoOptimal1y ago

If you have a recently updated NixOS unstable it has the affected version:

    $ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1

EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.org

mik19981y ago

Personally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.

buildbot1y ago

This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?

Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)

pdimitar1y ago

This was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors.

Obviously a bad actor will make use of these conditions and the assumption of good will.

We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.

devttyeu1y ago

Wouldn’t be surprised that the ssh auth being made slower was deliberate - that makes it fairly easy to index all open ssh servers on the internet, then to see which ones get slower to fail preauth as they install the backdoor

bananapub1y ago

people are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

it wasn't the apparently newly-created identity "Hans Jansen" just asking for a new version to be uploaded, it was "Hans Jansen" providing a new version to be uploaded as a non-maintainer-upload - Debian-speak for "the maintainer is AWOL, someone else is uploading their package". if "Hans Jansen" is another attacker then they did this cleverly, providing the new - compromised - upstream tarballs in an innocent-looking way and avoiding anyone examining the upstream diff.

userbinator1y ago

Looking at how many requests to update to the backdoored version have been made, I wonder if the fact that many people (including developers) have been conditioned to essentially accept updates as "always-good" is a huge contributing factor in how easy it is to spread something like this.

The known unknowns can be better than the unknown unknowns.

A1kmm1y ago

Looks like GitHub has suspended access to the repository, which while it protects against people accidentally compiling and using the code, but certainly complicates forensic analysis for anyone who doesn't have a clone or access to history (which is what I think a lot of people will be doing now to understand their exposure).

multimoon1y ago

It seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier.

My server runs arch w/ a LTS kernel (which sounds dumb on the surface, but was by far the easiest way to do ZFS on Linux that wasn't Ubuntu) and it seems that since I don't have SSH exposed to the outside internet for good reason, and my understanding is Arch never patched shhd to begin with that I and most people who would be in similar situations to me are unaffected.

Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.

0x01y ago

Interesting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io...

mrbluecoat1y ago

> I am *not* a security researcher, nor a reverse engineer.

Could have fooled me - impressive write-up!

secondary_op1y ago

Github making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.

I think the lesson here for packagers is that binary testdata should not be present while doing the build.

It is too easy to hide things in testdata.

https://web.archive.org/web/20240329182300/https://www.openw...

pushedx1y ago

Mirror of the report, since the Openwall servers appear to be down.

nolist_policy1y ago

Debian is considering that their infrastructure may be compromised[1].

[1] https://fulda.social/@Ganneff/112184975950858403

fourfour31y ago

Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.

rossant1y ago

Incredible. It's like discovering your colleague for 2 years at the secret nuclear weapon facility is a spy for another country, covering his tracks until the very last minute. Feels like a Hollywood movie is coming up.

Should we start doing background checks on all committers to such critical IT infrastructure?

Luker881y ago

@people who write github scanners for updates and security issues (dependabot and the like)

Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?

I can't track them and I don't want them in my projects.

Might not be very helpful as it is easy to create new identities, but I see no reason to make it easier for them. Also, I might approach differently someone with lots of contributions to known projects than a new account, so it still helps.

ikekkdcjkfke1y ago

Github should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps

8organicbits1y ago

There's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

_mlbt1y ago

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

notyoutube1y ago

Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?

rasengan1y ago

> One portion of the backdoor is solely in the distributed tarballs. For easier reference, here's a link to debian's import of the tarball, but it is also present in the tarballs for 5.6.0 and 5.6.1:

Ubuntu 22.04 version:

dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library

Whew!

bagels1y ago

Is this a crime? Has anyone been prosecuted for adding a backdoor like this?

[1] - https://checkmarx.com/blog/over-170k-users-affected-by-attac...

dmarto1y ago

Kinda relevant, as I saw few comments about how safer languages are the solution.

Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].

[0] - https://github.com/c-skills/rust1

markus_zhang1y ago

Keeps one wonder how many similar backdoors are there in the wild. What is the best way to execute such a move? This is sophisticated enough, but not good enough to stay unnoticed for a long while. If I were a state actor I'd think about at least 6-12 months.

kapouer1y ago

Both https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).

oxymoron2901y ago

Jai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Quite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor.

bheadmaster1y ago

This is exactly why I fight the windmills so hard when it comes automatic updates in Linux software.

So much damage is caused just by adding a single maintainer to a project - imagine how much power you would have to wield the remote execution systems put in place by naive developers for "automatic updates".

All it takes is a single malicious maintainer given access to the new version update of some popular user software, and they have a new botnet of thousands of devices at their disposal. Better yet, after the backdoor installation, they can just release the real update and cover their tracks forever.

Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.

byearthithatius1y ago

I hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers

AdmiralAsshat1y ago

> Red Hat assigned this issue CVE-2024-3094.

Does that mean this affects RHEL and Fedora?

PedroBatista1y ago

Given the recent ( not so recent ) attacks/"bugs" I feel there is a need to do more than the already hard task of investigating and detecting attacks but also to bring IRL consequences to these people.

My understanding is that right now it's pretty much a name and shame of people who most of the time aren't even real "people" but hostile agents either working for governments or criminal groups ( or both )

Getting punched in the face is actually a necessary human condition for a healthy civilization.

agwa1y ago

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don't use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.

12 more replies

korginator1y ago

xz is so pervasive, I just discovered on my Mac that the (affected?) version 5.6.1 made it into homebrew. The post in the linked article says that only Linux x86-64 systems are affected, but now I'm left scratching my head whether my Mac is also in trouble, just that we don't know it yet.

jchoksi1y ago

The two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>

Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03

Blog post: "Kuso拍賣．有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>

Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>

The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.

The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.

If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).

Example question 38:

    Question 38
    accounta066 (3): Are these chopsticks really that good? I kind of want to buy
    them! But I recently sent money for online shopping but didn’t receive anything.
    It’s very risky; currently jiat0218 you don’t have any reviews, you can
    interview me. Do you want to hand it over?! … A sincere buyer will keep it.
    
    Reply to
    jiat0218 (4): First of all, I would like to express my condolences to you for
    your unfortunate experience! What can I say about this kind of thing...My little
    sister has always been trustworthy. What’s more, this is a pair of spiritual 
    chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
    my little sister is very careful and takes her time when answering your
    questions. Except for the two messages that were accidentally deleted by her,
    she always answers your questions. If this still doesn’t reassure you, then I
    can only say that I still have room to work hard. You are still welcome
    to bid... ^_^

Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.

dboreham1y ago

Something about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.

0x01y ago

Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...

CanaryLayout1y ago

Well isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it.

c_rrodriguez1y ago

Everybody here In jumping into the pure malice bandwagon, I have a better hypothesis.

Abandonment and inaction, the actual developers of these tools are elsewhere, oblivious to this drama, trying to make living because most of the time you are not compensated nor any corporation cares about making things sustainable at all. This is the default status of everything your fancy cloud depends on underneath.

An attacker took over of the project slowly and stayed dormant until recently.

autoexecbat1y ago

I'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ?

Are they somehow in the clear unless we can show they actively exploited it?

jcalvinowens1y ago

Oof, this is on my Sid laptop:

  {0}[calvinow@mozart ~] dpkg-query -W liblzma5
  liblzma5:amd64  5.6.0-0.2
  {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
  1

Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/

[1] https://news.ycombinator.com/item?id=39793805

costco1y ago

Anyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.

Retr0id1y ago

The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)

It's written in Pascal, and the only (semi-)documented way to build it yourself is to use a graphical IDE, and pull in pre-compiled library binaries (stored in the git repo of a dependency which afaict Pack is the only dependent of - appears to be maintained by the same pseudonymous author but from a different account).

I've opened an issue[2] outlining my concerns. I'm certainly not accusing them of having backdoored binaries, but if I was setting up a project to be deliberately backdoorable, it'd look a lot like this.

[0] https://pack.ac/

[2] https://github.com/PackOrganization/Pack/issues/10

We need to get these complex & bloated build-systems under control.

haolez1y ago

I'm not trying to troll, but I'm wondering if a distro like Gentoo is less susceptible to such attacks, since the source code feels more transparent with their approach. But then again, it seems that upstream was infected in this case, so I'm not sure if a culture of compiling from source locally would help.

crispyambulance1y ago

I am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know?

I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

If I am keeping stuff up to date, is there anything at all to worry about?

joshhansen1y ago

Is it time to deprecate the ability for code to implement linker symbols in other libraries? Shouldn't there be a strict namespace separation between binaries/libraries? liblzma being to implement openssh symbols seems like a symptom of a much larger problem.

jeffbee1y ago

Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.

BobbyTables21y ago

Why doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such…

I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)

The real villain is not the perpetrator, it is Microsoft, and it is all of us.

wannacboatmovie1y ago

Really disappointed in the number of posters here who are playing down rushing to judgement and suggesting perhaps a legitimate developer was compromised, when it's very clear this is sophisticated and not the work of a single person.

I'm recalling bad memories of the Juniper backdoor years ago.

Whoever did this, was playing the long game. As the top post pointed out, there was an effort to get this into Fedora.... which eventually makes its way into RHEL (read: high value targets). This was not for short term payoffs by some rogue developer trying to mine crypto or other such nonsense. What you are seeing here is the planting of seeds for something months or a year down the road.

Brian_K_White1y ago

It doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz.

I agree with the lzip guy

qxfys1y ago

So, it's been almost 24 hours since I read this yesterday. Is it confirmed that Jia Tan is the perpetrator? do we know who he/she really is? Or are we going to live for the rest of our lives only knowing the pseudo name? just like Satoshi Nakamoto did to us. ;)

n3uman1y ago

https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??

sschueller1y ago

So much for a quiet Easter holiday. Fuck

BarbaryCoast1y ago

There's a bug in the detection script. The line:

if [ "$path" == "" ]

should be

if [ "$path" = "" ]

vasili1111y ago

Could anyone please tell me if current stable version of Debian has that backdoor or not?

[0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...

ptx1y ago

Python for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right?

[1] https://github.com/python/cpython-source-deps/tree/xz

17e55aab1y ago

a user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg

5.4.5 can be compromised

https://github.com/microsoft/vcpkg/issues/37197

croemer1y ago

Which nation state (if any) is most likely behind this? China based on name, or is this a red herring?

The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.

See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

lacoolj1y ago

What a disappointment.

It's something always in the back of our minds as developers using public libraries, but when something like this happens, non-developers that hear about it start to associate it with the rest of the open-source community.

It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.

kazinator1y ago

Doesn't this call for criminal charges?

Dribble46331y ago

Hello,

Github just disabled the repo : https://github.com/tukaani-project/xz

Do someone have an up to date fork to see the project history ?

_zephyrus_1y ago

Is there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).

sirsinsalot1y ago

I think we have to assume that all community software is a target. The payoff for bad actors is too great.

For every one of these we spot, assume there are two we have not.

frankjr1y ago

Now consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.

3v1n01y ago

Also the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to...

It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.

65a1y ago

Is there a proper reverse engineering of the payload yet?

mdip1y ago

Anyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.

hcks1y ago

It was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?

sylware1y ago

This is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...

https://www.openwall.com/lists/oss-security/2024/03/29/4

zeehio1y ago

On Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...

Rucadi1y ago

Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.

squarefoot1y ago

State actor or not, let's not ignore that the backdoor has been discovered thanks to the open nature of the projects involved that allowed digging into the code. Just another example like the infamous Borland InterBase backdoor in the early 2K that remained dormant for years and was discovered months after the source code has been released. If the xz malware authors worked for any corp that produced closed source drivers or blobs that can't be properly audited, we would be fucked; I just hope this is not already happening, because the attack surface in all those devices and appliances out there running closed code is huge.

perryizgr81y ago

Why are projects like xz and sshd still active? Just freeze it, it works fine. Only changes should be fixes for vulnerabilities. None of this complicated new functionality. If you want something like that make a new project. If it is truly better people will use it.

dfgdfg345454561y ago

chmod u+x running detect_sh script just runs with no output on my arch linux box?

[1] https://github.com/JiaT75?tab=following

notmysql_1y ago

Interestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ

west0n1y ago

It seems that to counter this type of supply chain attack, the best practices for managing software dependencies are to pin the version numbers of dependencies instead of using `latest`, and to use static linking instead of dynamic linking.

hypnagogic1y ago

In the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?

bicepjai1y ago

For someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.

https://ubuntu.com/security/notices/USN-5378-2

itsTyrion1y ago

that's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder

wowserszzzzz1y ago

Wow

fwungy1y ago

Brain fart: would it be possible to attach passwords to a crypto based micro transaction such that every time you attempted a password entry your crypto account was charged a small fee for the login attempt?

This would thwart brute force attacks, but not be a significant cost for users. If you could attach your login to the crypto account it would mean the account would have to be funded to allow the attempt. The token wouldn't store passwords it would just be a gatekeeper to the login attempt.

The fees would be paid to the service providers as mining fees.

E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.

neoneye21y ago

Damn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer!

I ran "brew upgrade" and that downgraded to version 5.4.6.

LeoPanthera1y ago

xz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent.

https://www.nongnu.org/lzip/

bitwize1y ago

Looks like Jonathan Blow was right about open source.

user201801201y ago

Why is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.

zingelshuher1y ago

Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.

jum41y ago

Maybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.

andix1y ago

Is there already a list of distributions that included the affected versions in non-prereelase channels?

inevitable1121y ago

Surely the real target of this was Tor (which links liblzma) not random SSH servers.

Has this affected OpenBSD at all?

imanhodjaev1y ago

I wonder which browsers link liblzma and can this lead to https eavesdropping?

nateskulic1y ago

Fairly deep bugs for a Bazaar.

shp0ngle1y ago

we should take this diagram and change "random person in nebraska" to "possibly a state-level attacker"

nice

elintknower1y ago

Candidly how would someone protect against a vulnerability like this?

evilmonkey191y ago

Which OS are affected by this compromise?? Is Ubuntu affected?

betaby1y ago

How that backdoor is triggered and what exactly it does?

xvilka1y ago

Maybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.

llmblockchain1y ago

Was Debian 12/stable unaffected? Only sid?

https://imgur.com/WGaK3Tn

hypnagogic1y ago

- * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"

pinley1y ago

KOLANICH1y ago

Please note: the changes have been made after GitHub has enforced 2FA (certainly not for "better security", but for promotion of FIDO2 and Windows Hello biometric impl of FIDO2, see https://codeberg.org/KOLANICH/Fuck-GuanTEEnomo for more info. Until recent times (for now access via git protocol is blocked for my acc, I guess based on lack of 2FA set up) it was even possible to push into all repos one has access by just using single-factor SSH key even without enabling 2FA in the account). As I have warned, nothing will protect when a backdoor is introduced by a malicious maintainer, or a "smart entrepreneur" who sold his project to a ad-company, or a loyal "patriot" living and earning money within reach of some state, or just a powerless man who got an offer he can't refuse. In general supply chain attacks by "legitimate" maintainers cannot be prevented. "Jia Tan" is just a sockpuppet to mitigate consequences to maintainers to make it look like they are not involved into it. They surely are. At least according to the current info it were they who have given the malicious account the permission to publish releases on behalf of the project and access to the repo.

IMHO all maintainers of the backdooored projects anyhow related to accepting the malicious changes should be considered as accomplices and boycotted. We don't need evidence of their liability, it is they who need to maintain their reputation. We are just free to take our decisions based on their reputation. Even if they were hacked themselves, it is not our problem, it is their problem. Our problem is to keep ourselves safe. It may feel "unjust" to ruin reputation of a person based on the fact he may be cheated or hacked… But if a person can be cheated or hacked, why should he/she have such a good reputation as everyone else?! So, it makes a lot of sense to just exclude and replace everyone, for whome there exists evidence of comprometation, no matter due to unconcern or malice. But FOSS is a doocracy serving products at dumpling prices ($0, free of charge), and for majority backdoored software is completely acceptable given that they get them free of charge. And powerful actors who can afford to pay for software will just hire devs to develop their private versions, while allowing the public to pay $0 for their free versions and use the backdoors placed into them themselves. In other words a complete market failure.

I think that 1. xz project must be shut down completely. I mean projects should stop using it as a dependency, exclude from distros, boycott it. LZMA algo was developed by Igor Pavlov in 7z project, but somehow it has happenned that liblzma was developed and maintained by unrelated folks. liblzma should be developed as a part of 7z project taking no code other than the trivial one for API compatibility adapter from xz. 2. Projects created by compromised authkrs should be boycotted. 3. Other projects touched by the compromised devs/maintainers should be audited. 4. All the projects using autotools should be audited and must replace autotools with cmake/meson. Autotools is a piece of shit, completely uncomprehensible. There is no surprise it was used to hude a backdoor - according to my experience in FOSS noone likes to touch its scripts anyhow. 5. No project should be built from releases. Project should be built from git directly. Implementing full support of SHA256 in git and git forges (GitHub, GitLab, Codeberg, sr.ht) should be accelerated to mitigate attacks using collisions to replace approved commits (I guess the randomness can be concealed from reviewer's eye in binary resource files, like pictures).

Rhea_Karty1y ago

TLDR: Some people have been throwing around “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:

3: + 0200 (in winter: February and November)

6: +0300 (in summer: in Jun, Jul, early October)

440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.

2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March).

Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Let’s analyze the few times where Jia was recorded in a non +800 time zone. Here, we notice that there are some situations where Jia switches between +800 and +300/+200 in a seemingly implausible time. Indicating that perhaps he is not actually in +800 CST time, as his profile would like us to believe.

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800

Jia Tan Sat, 19 Nov 2022 23:18:04 +0800

Jia Tan Mon, 7 Nov 2022 16:24:14 +0200

Jia Tan Sun, 23 Oct 2022 21:01:08 +0800

Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)

Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)

Jia Tan Wed, 5 Oct 2022 23:54:12 +0800

Jia Tan Wed, 5 Oct 2022 20:57:16 +0800

—> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800

Jia Tan Thu, 8 Sep 2022 15:07:00 +0300

Jia Tan Mon, 25 Jul 2022 18:30:05 +0300

Jia Tan Mon, 25 Jul 2022 18:20:01 +0300

Jia Tan Fri, 1 Jul 2022 21:19:26 +0800

Jia Tan Thu, 16 Jun 2022 17:32:19 +0300

Jia Tan Mon, 13 Jun 2022 20:27:03 +0800

—> the ordering of these time stamps, and the switching back and forth looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800

Jia Tan Thu, 15 Feb 2024 01:53:40 +0800

Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

Jia Tan Tue, 13 Feb 2024 22:38:58 +0800

—> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023):

- Working on 2023, 29 September: Mid Autumn Festival

- Working on 2023, 05 April: Tomb Sweeping Day

- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays:

- Never working on Dec 25: Christmas (for many EET countries)

- Never working Dec 31 or Jan 1: New Years

Observation 5: No weekend work —> salary job?

The most common working days for Jia was Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to be EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

alathers1y ago

Thank the gods I didn't plan on having a life this weekend

7ero1y ago

Is this sev0?

krascovict1y ago

Hello everybody.

I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.

The purpose of this commentary is focused on open source information (OSINT).

I am not a judge of anyone or any action that may occur, the objective of this comment is to help through accurate and quick information to help the core developers of the affected packages and consequently the Linux kernel (which may have been indirectly or directly affected) take action necessary in relation to the fact that occurred.

NOTE: This comment will always have "edit" so always review it for information.

Information I have so far.

Summary: 1. GitHub Account Suspension: - The accounts of @JiaT75 and @Larhzu were suspended by GitHub. - All Tukaani repositories, including downloads, were disabled. - Investigate the cause of the account suspensions and whether there is any correlation with suspicious activities.

2. Possible Backdoor in xz/liblzma: - There are concerns about the presence of a backdoor in xz/liblzma. - Investigate whether there is evidence of compromise in the source code and recent updates. - Examine potential impacts, especially if the software is used in critical systems.

3. Updates and Patches in Packages: - Note recent updates in packages such as MinGW w64, pacman-static, Alpine, and OpenSUSE. - Review changelogs to understand if these updates are related to security fixes.

4. Jia's Activities on Platforms and Projects: - Investigate Jia's contributions to different projects and platforms, such as Arch Linux, Alpine Linux, and OpenSUSE. - Check for correlations between Jia's activities and reported security issues.

5. Libera Registration Information: - Analyze Jia's registration details on Libera to determine the timeline of their online activities. - Consider correlating this information with other online activities of Jia.

6. VPN Usage: - Confirm Jia's use of VPN and assess its impact on security investigations. - Explore possible reasons for using a VPN and how it may affect the identification and tracking of online activities.

Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760

Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...

xyst1y ago

Time for another OS wipe. Glad I keep bleeding edge versions VMd

7ero1y ago

is this sev0?

kosolam1y ago

Jesus! Does anyone know if Debian stable is affected?

imanhodjaev1y ago

now I wonder which browsers link liblzma?

Rhea_Karty1y ago

Notes on time stamps and time zones.

A few interesting bits that I haven't fully fleshed out. TLDR: Some people have been throwing around that Jia is from “China,” but it seems also quite possible that Jia is from somewhere in Eastern Europe pretending to be from China. In addition, Lasse Collin and Hans Jansen are from the same EET time zone.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone: 3: + 0200 (in winter: February and November) 6: +0300 (in summer: in Jun, Jul, early October) 440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert. 2. The +0200/+0300, if we are assuming that this is one location, is likely on EET (Finland, Estonia, Latvia, Lithuania, Ukraine, Moldavia, Romania, Bulgaria, Greece, Turkey). This is because we see a switch from +300 in the winter (past the last weekend of October) and +200 in the summer (past the last Sunday in March). 1. Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Jia Tan Tue, 27 Jun 2023 23:38:32 +0800 —> 23:38 + 8 = 7:30 (+ 1) Jia Tan Tue, 27 Jun 2023 17:27:09 +0300 —> 17:27 + 3 = 20:30 —> about a 9 hour difference, but a flight from China to anywhere in Eastern Europe is at a min 10 hours

Jia Tan Thu, 5 May 2022 20:53:42 +0800 Jia Tan Sat, 19 Nov 2022 23:18:04 +0800 Jia Tan Mon, 7 Nov 2022 16:24:14 +0200 Jia Tan Sun, 23 Oct 2022 21:01:08 +0800 Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1) Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1) Jia Tan Wed, 5 Oct 2022 23:54:12 +0800 Jia Tan Wed, 5 Oct 2022 20:57:16 +0800 —> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800 Jia Tan Thu, 8 Sep 2022 15:07:00 +0300 Jia Tan Mon, 25 Jul 2022 18:30:05 +0300 Jia Tan Mon, 25 Jul 2022 18:20:01 +0300 Jia Tan Fri, 1 Jul 2022 21:19:26 +0800 Jia Tan Thu, 16 Jun 2022 17:32:19 +0300 Jia Tan Mon, 13 Jun 2022 20:27:03 +0800 —> the ordering of these time stamps and the switching back and forth between time zones looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800 Jia Tan Thu, 15 Feb 2024 01:53:40 +0800 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Mon, 12 Feb 2024 17:09:10 +0200 Jia Tan Tue, 13 Feb 2024 22:38:58 +0800 —> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps

It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bank holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023): - Working on 2023, 29 September: Mid Autumn Festival - Working on 2023, 05 April: Tomb Sweeping Day - Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years

Observation 5: Little weekend work —> salary job?

The most common working days for Jia were Tue (86), Wed (85), Thu (89), and Fri (79). If we adjust his time zone to EET, then that means he is usually working 9 am to 6 pm. This makes much more sense than someone working at midnight and 1 am on a Tuesday night.

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

Another interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...

At the time I thought it was just rude, but maybe this is when it all started.

k8svet1y ago

Wait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source?

Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.

EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.

port4431y ago

I think its much more likely this was not a bad actor, given their long history of commits.

It's a known fact that China will "recruit" people to operate them. A quote:

> They talk to them, say my friend, I see you like our special menu. Are you from China? Are you here on a VISA? Do you have family back there? Would you like your family to stay alive? Is your loyalty to this temporary employer or is your loyalty to your motherland? You know, a whole bunch of stuff like that. That’s how Chinese intelligence operations acts...

This just gives feelings of less "compromised account" and more "Your account is now our account"

AdmiralAsshat1y ago

Yikes! Do you have any info on the individual's background or possible motivations?

13 more replies

mikolajw1y ago

Tukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat.

WHOWAS jiatan provided me the following information:

jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024

WHOIS yields nothing, the user is not present on the network at the moment.

Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.

mrcoffee4u1y ago

can someone ELI5 ?

fullstackchris1y ago

pRoBaBlY a StaTe AcToR

zero definition of what that means...

egos of people who just like to say cool words they don't understand

lol

this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad

state actor? who? what motive? what country? all comments involving "state actor" are very broad and strange... i would like people to stop using words that have no meaning, as it really takes away from the overall conversation of what is going on.

i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid

throwaway677431y ago

It's always Debian, like last time when they removed RNG randomness from ssh because of a warning.

mise_en_place1y ago

This is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.

circusfly1y ago

Waiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.

stephc_int131y ago

I guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety.

Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.

shortsunblack1y ago

Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/

Zigurd1y ago

"Lasse Collin," as other posters here have found, does not seem to exist as an experienced coder. Oddly, there is a Swedish jazz musician named Lasse Collin, which would otherwise be one of those names, especially the last name, that would stick out. Instead it is buried under a lot of mentions of a musician.

https://gist.github.com/Qix-/f1a1b9a933e8847f56103bc14783ab7...

j / k navigate · click thread line to collapse

1849 comments

rwmj1y ago

junon1y ago

GitHub has suspended @JiaT75's account.

EDIT: Lasse Collin's account @Larhzu has also been suspended.

EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.

Here's a mailing list message from them ca. 2022.

https://listor.tp-sv.se/pipermail/tp-sv_listor.tp-sv.se/2022...

MinGW w64 on AUR was last published by Jia on Feb 29: https://aur.archlinux.org/cgit/aur.git/log/?h=mingw-w64-xz (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)

pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: https://aur.archlinux.org/cgit/aur.git/?h=pacman-static

EDIT: I've emailed the maintainer to have the key removed.

Alpine was patched as of 6 hours ago.

https://git.alpinelinux.org/aports/commit/?id=982d2c6bcbbb57...

OpenSUSE is still listing Jia's public key: https://sources.suse.com/SUSE:SLE-15-SP6:GA/xz/576e550c49a36... (cross-ref with https://web.archive.org/web/20240329235153/https://tukaani.o...)

EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.

Arch appears to still list Jia as an approved publisher, if I'm understanding this page correctly.

https://gitlab.archlinux.org/archlinux/packaging/packages/xz...

EDIT: Just sent an email to the last committer to bring it to their attention.

EDIT: It's been removed.

jiatan's Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.

    -NickServ- Information on jiatan (account jiatan):
    -NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
    -NickServ- Last seen : (less than two weeks ago)
    -NickServ- User seen : (less than two weeks ago)
    -NickServ- Flags : HideMail, Private
    -NickServ- jiatan has enabled nick protection
    -NickServ- *** End of Info ***

/whowas expired not too long ago, unfortunately. If anyone has it I'd love to know.

They are not registered on freenode.

EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).

EDIT: Jia Tan was using a VPN to connect; that's all I'll be sharing here.

8 more replies

https://github.com/tukaani-project/xz/pull/53

> Thanks to Hans Jansen for the original patch.

jonathanspw1y ago

Yesterday sure was fun wasn't it :p Thanks for all your help/working with me on getting this cleaned up in Fedora.

userbinator1y ago

because of it's "great new features"

mongol1y ago

smeehee1y ago

Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version's safe; we shall see…

drazk1y ago

api1y ago

I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.

kapouer1y ago

Github accounts of both xz maintainers have been suspended.

thayne1y ago

Do you know if it was actually the commit author, of if their commit access was compromised?

LispSporks221y ago

Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.

gigatexal1y ago

Name and shame this author. They should never be allowed anywhere near any open projects ever again.

nodesocket1y ago

Can legal action be taken against the author if it's found he maliciously added the backdoor?

landownersubgrp1y ago

It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.

yieldcrv1y ago

I wonder who the target was!

menomatter1y ago

his account is active again on github https://github.com/JiaT75

KingOfCoders1y ago

Sleeper.

Jommi1y ago

the account was either sold or stolen

move-on-by1y ago

Fascinating. Just yesterday the author added a `SECURITY.md` file to the `xz-java` project.

Reading that in a different light, it says give me time to adjust my exploits and capitalize on any targets. Makes me wonder what other vulns might exist in the author's other projects.

ncr1001y ago

Security Researchers: Is this request-for-private-disclosure + "90-days before public" reasonable?

It's a SEVERE issue, to my mind, and 90 days seems too long to me.

xyst1y ago

90 day dark window for maintainers is SOP though. Then after 90 days, it’s free game for public disclosure

szundi1y ago

How many of people like this one exist?

tw041y ago

Aissen1y ago

But luckily there was some serendipity: "I accidentally found a security issue while benchmarking postgres changes." https://mastodon.social/@AndresFreundTec/112180083704606941

miduil1y ago

This is getting addressed here: https://github.com/google/oss-fuzz/issues/11760

nialv71y ago

This in of itself can be legitimate. ifunc has real uses and it indeed does not work when sanitizer is enabled. Similar change in llvm: https://github.com/llvm/llvm-project/commit/1ef3de6b09f6b21a...

throwaway2901y ago

and that was in mid 2023. Very funny that Wikipedia on this issue says

> It is unknown whether this backdoor was intentionally placed by a maintainer or whether a maintainer was compromised

Yeah, if you've been compromised for a year your attacker is now your identity. Can't just wave hands, practice infosec hygiene

arp2421y ago

I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

bodyfour1y ago

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup (and not in "curl | sh" scripts).

Yeah, I've been banging on that same drum for ages too... for example on this very site a decade ago: https://news.ycombinator.com/item?id=7213563

I'm honestly surprised that this autoconf vector hasn't happened more often... or more often that we know of.

eigenvalue1y ago

Every single commit this person ever did should immediately be rolled back in all projects.

https://arstechnica.com/gadgets/2023/05/cancel-your-winrar-t...

dist-epoch1y ago

> they have commits on some related projects like libarchive

Windows started using libarchive to support .rar, .7z, ...

ptx1y ago

(This means you too, gradle-wrapper! And your generated wrapper for your generated wrapper. That junk is not source code and doesn't belong in the repo.)

AeroNotix1y ago

Pure speculation but my guess is a specific state actor ahem is looking for developers innocently working with open source to then strongarm them into doing stuff like this.

sslayer1y ago

I would be curious if their commits could be analyzed for patterns that could then be used to detect commits from their other account

bawolff1y ago

This is the sort of case where america's over the top hacking laws make sense.

WesolyKubeczek1y ago

> I've long since said that if you want to hide something nefarious you'd do that in the GNU autoconf soup

If I recall correctly, xz can be built with both autoconf and cmake, are cmake configs similarly affected?

stabbles1y ago

How about wheels in the python ecosystem

IshKebab1y ago

Yeah this was my first thought too. Though I think the case against autoconf is already so overwhelming I think anyone still using it is just irredeemable; this isn't going to persuade them.

bonyt1y ago

For those panicking, here are some key things to look for, based on the writeup:

- A very recent version of liblzma5 - 5.6.0 or 5.6.1. This was added in the last month or so. If you're not on a rolling release distro, your version is probably older.

- Running OpenSSH sshd from systemd. OpenSSH as patched by some distros only pulls in libsystemd for logging functionality, which pulls in the compromised liblzma5.

Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

It is possible there are other flaws or backdoors in liblzma5, though.

pdw1y ago

treffer1y ago

Ubuntu still ships 5.4.5 on 24.03 (atm).

So for 5.4.5 the tagged release and download on github differ.

It does change format strings, e.g.

   +#: src/xz/args.c:735
   +#, fuzzy
   +#| msgid "%s: With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgid "With --format=raw, --suffix=.SUF is required unless writing to stdout"
   +msgstr "%s: amb --format=raw, --suffix=.SUF és necessari si no s'escriu a la sortida estàndard"

There is no second argument to that printf for example. I think there is at least a format string injection in the older tarballs.

[Edit] formatting

blcknight1y ago

> Debian testing already has a version called '5.6.1+really5.4.5-1' that is really an older version 5.4, repackaged with a newer version to convince apt that it is in fact an upgrade.

I'm surprised .deb doesn't have a better approach. RPM has epoch for this purpose http://novosial.org/rpm/epoch/index.html

NotPractical1y ago

> If you're not on a rolling release distro, your version is probably older.

idoubtit1y ago

The article gives a link to a simple shell script that detects the signature of the compromised function.

> Running OpenSSH sshd from systemd

I think this is irrelevant.

Maybe some distributions that don't use systemd strip the libxz code from the upstream OpenSSH release, but I wouldn't bet on it if a fix is available.

fransje261y ago

I did notice that my debian-based system got noticeably slower and unresponsive at times the last two weeks, without obvious reasons. Could it be related?

I read through the report, but what wasn't directly clear to me was: what does the exploit actually do?

My normal internet connection has such an appalling upload that I don't think anything relevant could be uploaded. But I will change my ssh keys asap.

hostyle1y ago

$ dpkg-query -W liblzma5

liblzma5:amd64 5.4.1-0.2

rathermucker1y ago

Tumbleweed has a package: liblzma5-5.6.1.revertto5.4-3.2.x86_64 FYI

1:https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.h...

Epa0951y ago

I hope Lasse Collin is doing OK! Here is a older message from him [1]

It's also good to keep in mind that this is an unpaid hobby project. "

cbolton1y ago

slavik811y ago

Lasse appears to be active and working on undoing the sabotage. https://git.tukaani.org/?p=xz.git;a=blobdiff;f=CMakeLists.tx...

delfinom1y ago

He came on IRC, he seemed ok. He did some cleanup of access and signed off for easter.

k8svet1y ago

I would like to see more attention given to this. I'm capable of compartmentalization and not over-guilting myself, but holy hell, I really hope he's doing alright. This would kind of destroy me.

About two years ago, someone showed up, started making good commits. In this case, they have some other community rep that goes back a bit further but... man it's an unsettling feeling.

syslog1y ago

Relevant xkcd:

A couple of years ago I wrote a Go library that wraps the xz C code and allows you to do xz compression in Go: https://github.com/jamespfennell/xz

About a week ago I received the first PR on that repo, to upgrade to 5.6.1. I thought it was odd to get such a random PR...it's not the same GitHub account as upstream though.

arp2421y ago

That said, looking at the GitHub account I'd be surprised if there's anything nefarious going on here. Probably just someone using your repo, seeing it's outdated, and updating it.

jaredallard21y ago

Hey all, I’m the author of that PR. Just posted to Github with additional context: https://github.com/jamespfennell/xz/pull/2#issuecomment-2027...

Bromeo1y ago

pinko1y ago

> it's not the same GitHub account as upstream

This is valuable information, and a sign that this may be the tip of an iceberg.

cbmuser1y ago

There was also a bug report in Debian which requested updating xz-utils to 5.6.1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

icambron1y ago

IMO your prior on this should be that it's most likely just someone innocently updating a dependency.

5kg1y ago

The backdoor (test binary blob and autoconf) is not part of the pull request.

baxtr1y ago

Suddenly anything like that becomes super suspicious.

I wonder how this will affect the OS community in general.

squigz1y ago

Internet detectives at work in this thread!

cf100clunk1y ago

  I am *not* a security researcher, nor a reverse engineer.  There's lots of
  stuff I have not analyzed and most of what I observed is purely from
  observation rather than exhaustively analyzing the backdoor code.

anarazel1y ago

FWIW, it felt intimidating as hell. And I'm fairly established professionally. Not sure what I'd have done earlier in my career (although I'd probably not have found it in the first place).

[0] https://github.com/microsoft/vcpkg/issues/37197

bonzini1y ago

For what it's worth the author is a PostgreSQL committer, he's not a security researcher but he's a pretty damn good engineer!

vhiremath41y ago

Honestly, you only get this kind of humility when you're working with absolute wizards on a consistent basis. That's how I read that whole analysis. Absolutely fascinating.

dang1y ago

Related ongoing threads:

Xz: Disable ifunc to fix Issue 60259 - https://news.ycombinator.com/item?id=39869718

FAQ on the xz-utils backdoor - https://news.ycombinator.com/item?id=39869068

Everything I Know About the XZ Backdoor - https://news.ycombinator.com/item?id=39868673

0xthr0w41y ago

Out of curiosity I looked at the list of followers of the account who committed the backdoor.

Randomly picked https://github.com/Neustradamus and looked at all their contributions.

Interestingly enough, they got Microsoft to upgrade ([0],[1]) `vcpkg` to liblzma 5.6.0 3 weeks ago.

[1] https://github.com/microsoft/vcpkg/pull/37199

sroussey1y ago

OMG: look at the other contributions. He is trying to take over projects and pushing some change to sha256 in a hundred projects.

Example: https://github.com/duesee/imap-flow/issues/96

https://github.com/ifupdown-ng/ifupdown-ng/pulls/easynetdev

asmor1y ago

He follows 54k accounts though, so it may indeed just be coincidence.

neustradamus1y ago

Dear @0xthr0w4, do you attack me because I have requested the XZ update?

Do not mix, I am not linked to the XZ project.

perihelions1y ago

edit to add: Arch Linux' entire package system used to run on .tar.xz binaries (they switched to Zstd a few years ago [0]).

[0] https://news.ycombinator.com/item?id=19478171 ("Arch Linux propose changing compression method from xz to zstd (archlinux.org)")

joeyh1y ago

A backdoored xz could also run payloads hidden inside other xz files, allowing targeted attacks.

Phenylacetyl1y ago

The same authors have also contributed to Zstd

nolist_policy1y ago

deb packages are xz compressed...

https://hachyderm.io/@joeyh/112180715824680521

pfortuny1y ago

Unfortunately, this is how good bad actors work: with a very long-term point of view. There is no “harmless” project any more.

jnxx1y ago

And, Joey Hess has counted at least 750 commits to xz from that handle.

This does not look trust-inspiring. If the code is complex, there could be many more exploits hiding.

hangonhn1y ago

ametrau1y ago

Probably a state actor. You can look far into the future when you’re working for the party.

dang1y ago

(I detached this subthread from https://news.ycombinator.com/item?id=39866275, for the sake of pruning the top heavy thread.)

lucasRW1y ago

More likely that the account of that dev was breawched, dont you think ?

moritonal1y ago

https://lore.kernel.org/lkml/20240330144848.102a1e8c@kaneli/

the_errorist1y ago

Looks like Lasse Collin has commented on LKML: https://lkml.org/lkml/2024/3/30/188

Also, some info here: https://tukaani.org/xz-backdoor/

jwilk1y ago

Or if you can't stand the lkml.org UI:

bawolff1y ago

The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.

Makes you wonder what more competent actors can do.

rwmj1y ago

I've analysed the backdoor myself and it's very sophisticated, not poorly made at all. The performance problem is surprising in this context, but I think next time they won't make that mistake.

aidenn01y ago

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

https://lore.kernel.org/lkml/20240320183846.19475-2-lasse.co...

yard20101y ago

You must mean, "Makes you wonder what more competent actors are doing"

pinko1y ago

s/can do/have done/

gmnon1y ago

Funny how Lasse Collin started to ccing himself and Jia Tan from 2024-03-20 (that was a day of tons of xz kernel patches), he never did that before. :)

bombcar1y ago

This is extremely suspicious.

It looks like someone may have noticed a unmaintained or lightly maintained project related to various things, and moved to take control of it.

Otherwhere in the discussion here someone mentions the domain details changed; if you have control of the domain you have control of all emails associated with it.

ncr1001y ago

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

ui2RjUen875bfFA1y ago

those pipe usages are quite suspicious

pipeing into this shell script which now uses "eval"

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-n...

i guess this will be revisited and removed soon

bonzini1y ago

vhiremath41y ago

My favorite part was the analysis of "I'm not really a security researcher or reverse engineer but here's a complete breakdown of exactly how the behavior changes."

You only get this kind of humility when you're working with absolute wizards on a consistent basis.

asveikau1y ago

agwa1y ago

Thanks to autoconf, we're now used to build scripts looking like gibberish. A perfect place to hide a backdoor.

[0] https://news.ycombinator.com/item?id=39154825

omoikane1y ago

ptx1y ago

The use of "eval" stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.

A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of "eval".

zb31y ago

Yeah, now imagine they succeeded and it didn't cause any performance issues...

Can we even be sure no such successful attempt has already been made?

tetromino_1y ago

20after41y ago

Crazy indeed.

Decabytes1y ago

Turns out burned out maintainers are a great attack vector and if you are willing to play the long game you can ingratiate yourself with the community with your seemingly innocuous contributions.

kortilla1y ago

Paid people get burnt out as well and they are just as likely to accept free help as an unpaid person.

qwery1y ago

> So when are we going to stop pretending ...

I'm not sure that we are. Doesn't everybody know that developing/maintaining free software is largely thankless work, with little to no direct recompense?

delfinom1y ago

OSS maintainers aren't reaping anything. Most OSS licenses say the software is provided without warranty.

thesnide1y ago

The discussion to upload it to Debian is interesting on its own https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

0x01y ago

Wow, that's a lot of anonymous accounts adding comments there urging for a fast merge!

And this "Hans Jansen" guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: https://salsa.debian.org/users/hjansen/activity

https://youtu.be/IfhMILe8C84

nammi1y ago

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can't be a real person

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=2267598

oasisaimlessly1y ago

For anyone else feeling some deja vu about ifunc / Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].

[2]: https://news.ycombinator.com/item?id=39733185

fullstop1y ago

I get the feeling that a number of the comments are all the same person / group.

rpigab1y ago

I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

qecez1y ago

> I really don't think some random guy wants to weaken ssh just to extract some petty ransomware cash from a couple targets.

Which is why there's probably nothing remotely interesting in them logs.

RockRobotRock1y ago

Nah. I'm sure Microsoft got a call from the alphabet boys and nobody, not even internal employees are allowed to look at the logs right now.

megous1y ago

Oh my, another reason not to use github. :D So many reasons poping up just in this comment section alone.

optimalsolver1y ago

I'm guessing Microsoft just got a call from the Government telling them not to look too deeply into it.

alpb1y ago

That’d be illegal for an employee to do.

https://github.com/tukaani-project/tukaani-project.github.io...

alright25651y ago

> Note: GitHub automatically includes two archives Source code (zip) and Source code (tar.gz) in the releases. These archives cannot be disabled and should be ignored.

The author was thinking ahead! Latest commit hash for this repo: 8a3b5f28d00ebc2c1619c87a8c8975718f12e271

rom1v1y ago

Btw, this is not the only project providing a source tarball different from the git repo, for example libusb also does this (and probably others):

- https://github.com/libusb/libusb/issues/1468#issuecomment-19...

- https://github.com/orgs/community/discussions/6003

o11c1y ago

kzrdude1y ago

Jia Tan "cleaned up" in all their ZSTD branches some hours ago, probably hiding something https://github.com/JiaT75/zstd/branches/all

sroussey1y ago

GitHub/Microsoft likely has a backup. I’d be getting those out about now.

zamalek1y ago

Bad move. Destroying evidence is a felony.

[1] https://git.phial.org/d6/xz-analysis-mirror

zh31y ago

[0] https://lwn.net/Articles/967194/

eBombzor1y ago

That man deserves a Nobel Prize

dhx1y ago

The offending tarball for v5.6.1 is easier to find, an example being.[2]

m4/.gitignore was updated 2 weeks ago to hide build-to-host.m4 that is only present in the release tarball and is used to inject the backdoor at build time.[3]

[2] https://mirrors.xtom.ee/gentoo/distfiles/9f/xz-5.6.1.tar.gz

[3] https://git.phial.org/d6/xz-analysis-mirror/commit/4323bc3e0...

xyzzy_plugh1y ago

This gist summarizes the current situation very well: https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78b...

wood_spirit1y ago

treffer1y ago

1. Everything must be visible. A diff between the release tarball and tag should be unacceptable. It was hidden from the eyes to begin with.

2. Build systems should be simple and obvious. Potentially not even code. The inclusion was well hidden.

This should exist for good debugging anyway.

It wouldn't hinder source code based backdoors or malicious vulnerable code. But it would detect a backdoor like this one.

Just an initial thought though, and probably hard to do, but not impossibly hard, especially for a default server environment.

mac-chaffee1y ago

[1]: https://en.wikipedia.org/wiki/Capability-based_security

Note that the malicious binary is fairly long and complex.

This attack can be stopped by disallowing any binary testdata or other non-source code to be on the build machines during a build.

autogenerated files would also not be allowed to be in the source repo - they're too long and could easily hide bad stuff. Instead the build process should generate the file during the build.

afiodorov1y ago

We should be able to produce a tar and a proof that tar was produced from a specific source code.

Quote from the article:

    That line is not in the upstream source of build-to-host, nor is build-to-host used by xz in git.

The verifiers can then easily check that the signature and indeed be convinced that the code was executed as it is claimed and source code wasn't tampered with.

JonChesterfield1y ago

The Guix full source bootstrap is looking less paranoid as time goes on

Avamander1y ago

I'm not aware of any efforts towards it, but libraries should also probably be more confined to only provide intended functionality without being able to hook elsewhere?

q3k1y ago

jchw1y ago

It may not have really mattered much for NixOS:

> b) argv[0] needs to be /usr/sbin/sshd

For once, the lack of FHS interoperability is a benefit, if only on accident.

o11c1y ago

lostmsu1y ago

dandanua1y ago

That's one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.

bhaak1y ago

That's devastating.

If you don't build your release packages from feeding "git ls-files" into tar, you are doing it wrong.

sesuximo1y ago

I think this is unfortunately very common practice

icommentedtoday1y ago

Why not `git archive`?

https://news.ycombinator.com/item?id=39872062

cryptonector1y ago

colanderman1y ago

The latest commit from the user who committed those patches is weirdly a simplification of the security reporting process, to not request as much detail:

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Not sure what to make of this.

rany_1y ago

colanderman1y ago

That repository is now disabled. But here's a similar change to the .github repository of tukaani-project from @JiaT75 to the bug report template:

    + or create a private Security Advisory instead.

Under a commit titled "Wrap text on Issue template .yaml files."

[1] https://github.com/tukaani-project/.github/commit/44b766adc4...

caelum191y ago

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

20after41y ago

> "Docs: Simplify SECURITY.md."

Removes instructions about details relevant to security reports. Heh, nice one.

Tenobrus1y ago

It looks like the person who added the backdoor is in fact the current co-maintainer of the project (and the more active of the two): https://tukaani.org/about.html

kzrdude1y ago

In various places they say Lasse Collin is not online right now, but he did make commits a week ago https://git.tukaani.org/?p=xz.git;a=summary

kzrdude1y ago

Makes me wonder if he's an owner of the github organization, and what happens with it now?

CGamesPlay1y ago

https://github.com/tukaani-project/xz vs https://github.com/JiaT75

dzaima1y ago

5p4n9111y ago

I hope Lasse Collin still has control of his accounts, though the CC on the kernel mailing list looks kind of suspicious to me.

weinzierl1y ago

The backdoor is not in the C source directly, but a build script uses data from files in the test dir to only create the backdoor in the release tars. Did I summarize that correctly?

soneil1y ago

https://aws.amazon.com/security/security-bulletins/AWS-2024-...

elchief1y ago

liveoneggs1y ago

The best part is everyone disabling security tests that started failing

ozgune1y ago

I read through the entire report and it gradually got more interesting. Then, I got to the very end, saw Andres Freund's name, and it put a smile on my face. :)

Who else would have run a PostgreSQL performance benchmark and discover a major security issue in the process?

jaromilrojo1y ago

acdha1y ago

geggo981y ago

Still I would appreciate it, if systemd developers would find a better protection against supply chain attacks.

saagarjha1y ago

This isn't Twitter you don't have to use hashtags

throwaway73561y ago

> systemd's call to dlopen() liblzma library (CVE-2024-3094)

That's technically wrong, but no surprise. Anti-systemd trolls usually don't understand technical details after all.

sn1y ago

I used this regex to split in python, and wrote to "tmp":

    re.split(b'\xfd7zXZ', x)

I manually used dd and truncate to strip out the remaining header and footer according to the specification, which left 48 bytes:

    $ ent tmp2 # bad file payload
    Entropy = 4.157806 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 48 percent.
    
    Chi square distribution for 48 samples is 1114.67, and randomly
    would exceed this value less than 0.01 percent of the times.
    
    Arithmetic mean value of data bytes is 51.4167 (127.5 = random).
    Monte Carlo value for Pi is 4.000000000 (error 27.32 percent).
    Serial correlation coefficient is 0.258711 (totally uncorrelated = 0.0).
    
    $ ent tmp3 # urandom
    Entropy = 5.376629 bits per byte.
    
    Optimum compression would reduce the size
    of this 48 byte file by 32 percent.
    
    Chi square distribution for 48 samples is 261.33, and randomly
    would exceed this value 37.92 percent of the times.
    
    Arithmetic mean value of data bytes is 127.8125 (127.5 = random).
    Monte Carlo value for Pi is 3.500000000 (error 11.41 percent).
    Serial correlation coefficient is -0.067038 (totally uncorrelated = 0.0).

supriyo-biswas1y ago

Now to be fair, such an archive could have been created with a “store” level of compression that doesn’t actually perform any compression.

https://en.wikipedia.org/wiki/2020_United_States_federal_gov...

0x01y ago

All these older (4.x, 5.0.x etc) releases that were suddenly uploaded a few months ago should probably also be considered suspect: https://github.com/tukaani-project/tukaani-project.github.io...

kn1001y ago

``` #!/bin/bash

# Get list of all running Docker containers containers=$(docker ps --format "{{.Names}}")

# Loop through each container for container in $containers; do # Get container image image=$(docker inspect --format='{{.Config.Image}}' "$container")

    # Execute xz --version inside the container
    version=$(docker exec "$container" xz --version)

    # Write container name, image, and command output to a text file
    echo "Container: $container" >> docker_container_versions.txt
    echo "Image: $image" >> docker_container_versions.txt
    echo "xz Version:" >> docker_container_versions.txt
    echo "$version" >> docker_container_versions.txt
    echo "" >> docker_container_versions.txt

done

echo "Output written to docker_container_versions.txt" ```

Roark661y ago

So what is the solution? IDK, but enforcing certain "understandability" requirements can be a part of it.

alufers1y ago

throwaway73561y ago

> Well, consider this, to "contribute" to a proprietary project you need to get hired by a company, go through their he.

Or work for a third-party company that gets access to critical systems without any checks. See for example the incident from 2022 here: https://en.wikipedia.org/wiki/Okta,_Inc.

Or a third-party that rents critical infrastructure to the company (Cloud, SaaS solutions).

Scaevolus1y ago

It's wild that this could have laid dormant for far longer if the exploit was better written-- if it didn't spike slow down logins or disturb valgrind.

lpapez1y ago

r0ckarong1y ago

> cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That's the entire point. You did everything you could by getting someone else look at it and saying it's fine.

CableNinja1y ago

Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.

keepamovin1y ago

If you installed xz on macOS using brew, then you have

  xz (XZ Utils) 5.6.1
  liblzma 5.6.1

which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with

  brew upgrade xz

bawolff1y ago

> the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

That is actually a major point of a lot of corporate security measures (shifting risk)

[0] https://github.com/search?q=liblzma+5.6.0&type=pullrequests

markus_zhang1y ago

gouggoug1y ago

List of pull request requesting the updating to liblzma 5.6.0 [0]

I wonder what amount of scrutiny all the accounts that proposed the upgrade should be put under.

snickerer1y ago

dlenski1y ago

I'm not *too* worried about OpenConnect given that we use `libxml2` only to read and parse uncompressed XML…

But I am wondering if there has been any statement from libxml2 devs (they're under the GNOME umbrella) about potential risks to libxml2 and its users.

enedil1y ago

This doesn't matter, if libxml2 loads .so and the library is malicious, you are already potentially compromised, as it is possible to run code on library load.

bananapub1y ago

> only to read and parse uncompressed XML…

how does libxml2 know to decompress something?

does it require you, as the caller, to explicitly tell it to?

or does it look at the magic bytes or filename or mimetype or something?

https://github.com/emirkmo/xz-backdoor-github

afh11y ago

Potentially malicious commit by same author on libarchive: https://github.com/libarchive/libarchive/pull/1609

youainti1y ago

Summary: "The upstream xz repository and the xz tarballs have been backdoored."

It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

Randalthorro1y ago

Since GitHub disabled the repos.. I uploaded all GitHub Events from the two suspected users and from their shared project repo as easy to consume CSV files:

For those who want to see the GitHub events (commits, comments, pull_requets, diffs, etc.)

yogorenapan1y ago

Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries

ParetoOptimal1y ago

If you have a recently updated NixOS unstable it has the affected version:

    $ xz --version
    xz (XZ Utils) 5.6.1
    liblzma 5.6.1

EDIT: I've been informed on the NixOS matrix that they are 99% sure NixOS isn't affected, based on conversations in #security:nixos.org

mik19981y ago

Personally, I use lzip ever since I read https://www.nongnu.org/lzip/xz_inadequate.html Seems like the complexity of XZ has backfired severely, as expected.

buildbot1y ago

This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?

pdimitar1y ago

This was only a matter of time. Open source projects are under-staffed, maintainers are overworked and burned out, and everyone relies on the goodwill of all actors.

Obviously a bad actor will make use of these conditions and the assumption of good will.

We need automated tooling to vet for stuff like this. And maybe migrate away from C/C++ while we are at it because they don't make such scanning easy at all.

devttyeu1y ago

bananapub1y ago

people are mis-reading the Debian bug report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

userbinator1y ago

The known unknowns can be better than the unknown unknowns.

A1kmm1y ago

multimoon1y ago

It seems like based on the (very well written) analysis that this is a way to bypass ssh auth, not something that phones out which would've been even scarier.

Still insane that this happened to begin with, and I feel bad for the Archlinux maintainers who are now going to feel more pressure to try to catch things like this.

0x01y ago

Interesting commit in January where the actual OpenPGP key was changed: https://github.com/tukaani-project/tukaani-project.github.io...

mrbluecoat1y ago

> I am *not* a security researcher, nor a reverse engineer.

Could have fooled me - impressive write-up!

secondary_op1y ago

Github making suspect repository private and hiding recent account activity is wrong move and is interfering with citizens investigation efforts.

I think the lesson here for packagers is that binary testdata should not be present while doing the build.

It is too easy to hide things in testdata.

https://web.archive.org/web/20240329182300/https://www.openw...

pushedx1y ago

Mirror of the report, since the Openwall servers appear to be down.

nolist_policy1y ago

Debian is considering that their infrastructure may be compromised[1].

[1] https://fulda.social/@Ganneff/112184975950858403

fourfour31y ago

Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.

rossant1y ago

Should we start doing background checks on all committers to such critical IT infrastructure?

Luker881y ago

@people who write github scanners for updates and security issues (dependabot and the like)

Can we start including a blacklist of emails and names of contributors (with reasons/links to discussions)?

I can't track them and I don't want them in my projects.

ikekkdcjkfke1y ago

Github should probably remove the dopamine hits of green checkmarks etc. like in serious stock broker apps

8organicbits1y ago

There's good discussion of the timeline here: https://boehs.org/node/everything-i-know-about-the-xz-backdo...

_mlbt1y ago

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

notyoutube1y ago

Is the solution against such attacks in the future only to scrutinize more, or are there other reasonable options in terms of hardening?

rasengan1y ago

Ubuntu 22.04 version:

dpkg -l |grep liblzma ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression library

Whew!

bagels1y ago

Is this a crime? Has anyone been prosecuted for adding a backdoor like this?

[1] - https://checkmarx.com/blog/over-170k-users-affected-by-attac...

dmarto1y ago

Kinda relevant, as I saw few comments about how safer languages are the solution.

Here[0] is a very simple example, that shows how easy such supply chain attacks are in Rust; and lets not forget that there was a very large python attack just a few days ago[1].

[0] - https://github.com/c-skills/rust1

markus_zhang1y ago

kapouer1y ago

Both https://github.com/tukaani-project members accounts have been suspended. (to see that, you can list the followers of each account).

oxymoron2901y ago

Jai Tan's commit history on his github profile suggests he took off for Christmas, new years, and spring break. I smell an American.

https://github.com/tukaani-project/xz/commit/af071ef7702debe...

Quite ironic: The most recent commit in the git repo is "Simplify SECURITY.md", committed by the same Github account which added the backdoor.

bheadmaster1y ago

This is exactly why I fight the windmills so hard when it comes automatic updates in Linux software.

Automatic updates are like running web applications, but without any sandboxing or protection usually implemented by the browser.

byearthithatius1y ago

I hope mainstream news cover this so the general population can understand the issue with our software ecoysystems reliance on unpaid open-source maintainers

AdmiralAsshat1y ago

> Red Hat assigned this issue CVE-2024-3094.

Does that mean this affects RHEL and Fedora?

PedroBatista1y ago

Getting punched in the face is actually a necessary human condition for a healthy civilization.

agwa1y ago

> openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.

12 more replies

korginator1y ago

jchoksi1y ago

The two active maintainers seem to be: Lasse Collin <lasse.collin@tukaani.org> and Jia Tan <jiat0218@gmail.com>

Searching DDG for "jiat0218" I came across a blog post which I found weird. Seems to be dated: 2006-05-03

Blog post: "Kuso拍賣．有靈氣的筷子 - 闕小豪" <https://char.tw/blog/post/24397301>

Internet Archive link: <https://web.archive.org/web/20240329182713/https://char.tw/b...>

The contents of the page when translated seems to be about jiat0218 auctioning a pair of spiritual chopsticks as a prank.

The blog entry is basically a QA between jiat0218 and various other people about these chopsticks.

If Jia Tan does turn out to be a compromised maintainer working for a state actor then some of the content on the blog page can be viewed in a more sinister way (i.e. spycraft / hacks for sale etc.).

Example question 38:

    Question 38
    accounta066 (3): Are these chopsticks really that good? I kind of want to buy
    them! But I recently sent money for online shopping but didn’t receive anything.
    It’s very risky; currently jiat0218 you don’t have any reviews, you can
    interview me. Do you want to hand it over?! … A sincere buyer will keep it.
    
    Reply to
    jiat0218 (4): First of all, I would like to express my condolences to you for
    your unfortunate experience! What can I say about this kind of thing...My little
    sister has always been trustworthy. What’s more, this is a pair of spiritual 
    chopsticks, so I hope to have a good one. It’s the beginning! As you can see,
    my little sister is very careful and takes her time when answering your
    questions. Except for the two messages that were accidentally deleted by her,
    she always answers your questions. If this still doesn’t reassure you, then I
    can only say that I still have room to work hard. You are still welcome
    to bid... ^_^

Note however, it could all just be what it purports to be which is a prank auction of spiritual chopsticks.

dboreham1y ago

Something about this I found surprising is that Linux distros are pulling and packaging pre-built binaries from upstream projects. I'd have expected them to build from source.

0x01y ago

Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?

https://git.tukaani.org/?p=xz.git;a=commitdiff;h=4323bc3e0c1...

CanaryLayout1y ago

Well isn't this an interesting commit. He finished his inject macro to compose the payload at build, so now he can start clearing up the repo so none of that shit gets seen when cruising through it.

c_rrodriguez1y ago

Everybody here In jumping into the pure malice bandwagon, I have a better hypothesis.

An attacker took over of the project slowly and stayed dormant until recently.

autoexecbat1y ago

I'm really curious about if the act of injecting a backdoor into OSS software is legal/illegal ?

Are they somehow in the clear unless we can show they actively exploited it?

jcalvinowens1y ago

Oof, this is on my Sid laptop:

  {0}[calvinow@mozart ~] dpkg-query -W liblzma5
  liblzma5:amd64  5.6.0-0.2
  {0}[calvinow@mozart ~] hexdump -ve '1/1 "%.2x"' /lib/x86_64-linux-gnu/liblzma.so.5 | grep -c f30f1efa554889f54c89ce5389fb81e7000000804883ec28488954241848894c2410
  1

Glad I stopped running sshd on my laptop a long time ago... still probably going to reinstall :/

[1] https://news.ycombinator.com/item?id=39793805

costco1y ago

Anyone have any idea what the code in the malicious liblzma_la-crc64-fast.o is actually doing? It's difficult to follow statically.

Retr0id1y ago

The `pack`[0] compression utility that reached the HN front page the other day[1] is setting off my alarm bells right now. (It was at the time too, but now doubly so)

[0] https://pack.ac/

[2] https://github.com/PackOrganization/Pack/issues/10

We need to get these complex & bloated build-systems under control.

haolez1y ago

crispyambulance1y ago

I am not embarrassed to say... is there anything in there that someone who runs a server with ssh needs to know?

I literally can't make heads or tails of the risk here. All I see is the very alarming and scary words "backdoor" and "ssh server" in the same sentence.

If I am keeping stuff up to date, is there anything at all to worry about?

joshhansen1y ago

jeffbee1y ago

Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.

BobbyTables21y ago

Why doesn’t GitHub force “releases” to be a simple repo tarball for sources and with binaries from GitHub actions or such…

I find it incredibly ironic that a “version control” site gives no assurance of reproducible builds (nor reproducible source!!)

The real villain is not the perpetrator, it is Microsoft, and it is all of us.

wannacboatmovie1y ago

I'm recalling bad memories of the Juniper backdoor years ago.

Brian_K_White1y ago

It doesn't really relate to this issue other than that both issues share a common source, but I wish we'd never fallen for xz.

I agree with the lzip guy

qxfys1y ago

n3uman1y ago

https://github.com/tukaani-project/tukaani-project.github.io... Does this mean anything that it changed to a parameter??

sschueller1y ago

So much for a quiet Easter holiday. Fuck

BarbaryCoast1y ago

There's a bug in the detection script. The line:

if [ "$path" == "" ]

should be

if [ "$path" = "" ]

vasili1111y ago

Could anyone please tell me if current stable version of Debian has that backdoor or not?

[0] https://github.com/python/cpython/blob/main/PCbuild/get_exte...

ptx1y ago

Python for Windows bundles liblzma from this project, but it appears to be version 5.2.5 [0] vendored into the Python project's repo on 2022-04-18 [1], so that should be fine, right?

[1] https://github.com/python/cpython-source-deps/tree/xz

17e55aab1y ago

a user offered 5.6.0 and 5.4.5 in an issue to microsoft/vcpkg

5.4.5 can be compromised

https://github.com/microsoft/vcpkg/issues/37197

croemer1y ago

Which nation state (if any) is most likely behind this? China based on name, or is this a red herring?

The perpetrator did most GitHub actions between 10 and 18 UTC, which sort of rules out US based, unless the messages were scheduled. Consistent with Europe to Asia.

See clickhouse for data: https://play.clickhouse.com/play?user=play#U0VMRUNUICogRlJPT...

lacoolj1y ago

What a disappointment.

It's essentially a terrorist attack on developer experience. Thankfully, management doesn't follow the same approach as the TSA.

kazinator1y ago

Doesn't this call for criminal charges?

Dribble46331y ago

Hello,

Github just disabled the repo : https://github.com/tukaani-project/xz

Do someone have an up to date fork to see the project history ?

_zephyrus_1y ago

Is there any news concerning the payload analysis? Just curious to see if it can be correlated with something I have in my sshd logs (e.g. login attempt with specific RSA keys).

sirsinsalot1y ago

I think we have to assume that all community software is a target. The payoff for bad actors is too great.

For every one of these we spot, assume there are two we have not.

frankjr1y ago

Now consider that your average Linux distribution pulls in tens of thousands of packages, each of which can be similarly compromised. Pretty scary to think about.

3v1n01y ago

Also the attacker included in the 5.6.0 release the support for the long-awaited multi-threading decompression (and - broken - sandbox) making it very attractive to upgrade to...

It was probably a tactic to give a reason to upgrade. It's not always a fault for those who did or tried to do.

65a1y ago

Is there a proper reverse engineering of the payload yet?

mdip1y ago

Anyone keeping current with OpenSUSE Tumbleweed got a update...downgrade. Prior to `zypper dup --no-allow-vendor-change` I had 5.6.0, now I'm at 5.4.6.

hcks1y ago

It was caught out of luck due to performance degradation. So nobody reads the code - not even once- prior to merging into upstream supply chain?

sylware1y ago

This is why the less the better... even if it means less comfortable... to a certain point obviously. And that includes SDKs...

https://www.openwall.com/lists/oss-security/2024/03/29/4

zeehio1y ago

On Ubuntu there is a bug report asking to sync the 5.6 version from Debian experimental https://bugs.launchpad.net/ubuntu/+source/xz-utils/+bug/2055...

Rucadi1y ago

Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.

squarefoot1y ago

perryizgr81y ago

dfgdfg345454561y ago

chmod u+x running detect_sh script just runs with no output on my arch linux box?

[1] https://github.com/JiaT75?tab=following

notmysql_1y ago

Interestingly on of the accounts that the GitHub account who introduced the backdoor follows was suspended very recently [1] who is also part of the org who runs XZ

west0n1y ago

hypnagogic1y ago

In the future: automated `diff` or any other A/B check to see whether or not the tarball matches the source repo (if not, auto-flag with a mismatch warning attribute), is that feasible to implement?

bicepjai1y ago

For someone who does not understand the packages used, could you please summarize in layman non technical terms. Thanks I did read the main post.

https://ubuntu.com/security/notices/USN-5378-2

itsTyrion1y ago

that's... creative. and patient. 11/10 concerning - now I'm wondering how many other projects could have shit like this in them or added right as I'm writing this shudder

wowserszzzzz1y ago

Wow

fwungy1y ago

The fees would be paid to the service providers as mining fees.

E.g. foo@bar.com needs a password and a token provided from a designated crypto address to gain access to the service.

neoneye21y ago

Damn. I'm on macOS and use homebrew. To my surprise I had "xz" version 6.5.1 installed on my computer!

I ran "brew upgrade" and that downgraded to version 5.4.6.

LeoPanthera1y ago

xz is just a horribly designed format, and always has been. If you use it, please switch to Lzip. Same compression level, but designed by someone competent.

https://www.nongnu.org/lzip/

bitwize1y ago

Looks like Jonathan Blow was right about open source.

user201801201y ago

Why is the Long Range Zip lrzip compression format not used? It gives better compression than xz when using the correct switches.

zingelshuher1y ago

Why isn't he identified personally? Very likely he is 'contributing' to other projects under different accounts.

jum41y ago

Maybe @JiaT75 got forced to do it. Maybe someone has more personal contact with him and can check how he is doing.

andix1y ago

Is there already a list of distributions that included the affected versions in non-prereelase channels?

inevitable1121y ago

Surely the real target of this was Tor (which links liblzma) not random SSH servers.

Has this affected OpenBSD at all?

imanhodjaev1y ago

I wonder which browsers link liblzma and can this lead to https eavesdropping?

nateskulic1y ago

Fairly deep bugs for a Bazaar.

shp0ngle1y ago

we should take this diagram and change "random person in nebraska" to "possibly a state-level attacker"

nice

elintknower1y ago

Candidly how would someone protect against a vulnerability like this?

evilmonkey191y ago

Which OS are affected by this compromise?? Is Ubuntu affected?

betaby1y ago

How that backdoor is triggered and what exactly it does?

xvilka1y ago

Maybe it's finally time to start sunsetting LZMA and xz all together in favor of newer algorithms like Zstandard that also offer better performance but compression rates on par with LZMA.

llmblockchain1y ago

Was Debian 12/stable unaffected? Only sid?

https://imgur.com/WGaK3Tn

hypnagogic1y ago

- * _ring ring_ * - "Hello?" - "It's Lasse Collin." - "Why are you collin me? Why not just use the backdoor?"

pinley1y ago

KOLANICH1y ago

Rhea_Karty1y ago

These are my notes on time stamps/zones. There are a few interesting bits that I haven't fully fleshed out.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Here is the data on Jia’s time zone and the number of times he was recorded in that time zone:

3: + 0200 (in winter: February and November)

6: +0300 (in summer: in Jun, Jul, early October)

440: +0800

1. The +800 is likely CST. China (or Indonesia or Philippines), given that Australia does daylight savings time and almost no one lives in Siberia and the Gobi dessert.

Incidentally, this seems to be the same time zone as Lasse Collin and Hans Jansen…

Observation 2: Time zone inconsistencies

Jia Tan Thu, 5 May 2022 20:53:42 +0800

Jia Tan Sat, 19 Nov 2022 23:18:04 +0800

Jia Tan Mon, 7 Nov 2022 16:24:14 +0200

Jia Tan Sun, 23 Oct 2022 21:01:08 +0800

Jia Tan Thu, 6 Oct 2022 21:53:09 +0300 —> 21:53 + 3 = 1:00 (+1)

Jia Tan Thu, 6 Oct 2022 17:00:38 +0800 —> 17:00 + 8 = 1:00 (+1)

Jia Tan Wed, 5 Oct 2022 23:54:12 +0800

Jia Tan Wed, 5 Oct 2022 20:57:16 +0800

—> again, given the flight time, this is even more impossible

Jia Tan Fri, 2 Sep 2022 20:18:55 +0800

Jia Tan Thu, 8 Sep 2022 15:07:00 +0300

Jia Tan Mon, 25 Jul 2022 18:30:05 +0300

Jia Tan Mon, 25 Jul 2022 18:20:01 +0300

Jia Tan Fri, 1 Jul 2022 21:19:26 +0800

Jia Tan Thu, 16 Jun 2022 17:32:19 +0300

Jia Tan Mon, 13 Jun 2022 20:27:03 +0800

—> the ordering of these time stamps, and the switching back and forth looks strange.

Jia Tan Thu, 15 Feb 2024 22:26:43 +0800

Jia Tan Thu, 15 Feb 2024 01:53:40 +0800

Jia Tan Mon, 12 Feb 2024 17:09:10 +0200

Jia Tan Tue, 13 Feb 2024 22:38:58 +0800

—> this travel time is possible, but the duration of stay is unlikely

Observation 3: Strange record of time stamps It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seem to align much better with an Eastern European than a Chinese person.

Disclaimer: I am not an expert in Chinese holidays, so this very well could be inaccurate. I am referencing this list of bak holidays:(https://www.bankofchina.co.id/en-id/service/information/late...)

Chinese bank holidays (just looking at 2023):

- Working on 2023, 29 September: Mid Autumn Festival

- Working on 2023, 05 April: Tomb Sweeping Day

- Working on 2023, 26, 22, 23, 24, 26, 27 Jan: Lunar New Year

Eastern European holidays:

- Never working on Dec 25: Christmas (for many EET countries)

- Never working Dec 31 or Jan 1: New Years

Observation 5: No weekend work —> salary job?

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

alathers1y ago

Thank the gods I didn't plan on having a life this weekend

7ero1y ago

Is this sev0?

krascovict1y ago

Hello everybody.

I am taking the initiative to gather more information regarding the possible precursors and perpetrators of the backdoor.

The purpose of this commentary is focused on open source information (OSINT).

NOTE: This comment will always have "edit" so always review it for information.

Information I have so far.

Links related to user JiaT75 [xz] Remove JiaT75 as a contact, determine correct contacts #11760 - Google/oss-fuzz https://github.com/google/oss-fuzz/issues/11760

Tuktest index hash #7 - tukaani-project/xz/pull/7 https://web.archive.org/web/20240329230522/https://github.co...

xyst1y ago

Time for another OS wipe. Glad I keep bleeding edge versions VMd

7ero1y ago

is this sev0?

kosolam1y ago

Jesus! Does anyone know if Debian stable is affected?

imanhodjaev1y ago

now I wonder which browsers link liblzma?

Rhea_Karty1y ago

Notes on time stamps and time zones.

The following analysis was conducted on JiaT75’s (https://github.com/JiaT75?tab=overview&from=2021-12-01&to=20...) commits to the XZ repository, and their time stamps.

Observation 1: Time zone basic analysis

Observation 2: Time zone inconsistencies

Observation 3: Strange record of time stamps

It seems that from the commits, often the time stamps are out of order. I am not sure what would cause this other than some tampering.

Observation 4: Bank holiday inconsistencies

We notice that Jia’s work schedule and holidays seems to align much better with an Eastern European than a Chinese person.

Eastern European holidays: - Never working on Dec 25: Christmas (for many EET countries) - Never working Dec 31 or Jan 1: New Years

Observation 5: Little weekend work —> salary job?

These times also line up well with Hans Jansen and Lasse Collin.

I think it is more likely that Jia does this as part of his work… somewhere in Eastern Europe. Likely working with, or in fact being one and the same as, Hans Jansen and Lasse Collin.

Another interesting data point: about 2 years ago there was a clear pressure campaign to name a new maintainer: https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.h...

At the time I thought it was just rude, but maybe this is when it all started.

k8svet1y ago

Wait, I'm on mobile. Did this partially slip by because of the ABSURD PRACTICE of publishing release.tarballs that do not 1:1 correspond with source?

Let me guess, autotools? I want to rage shit post but I guess I'll wait for confirmation first.

EDIT: YUP, AT LEAST PARTIALLY. Fucking god damn autotools.

port4431y ago

I think its much more likely this was not a bad actor, given their long history of commits.

It's a known fact that China will "recruit" people to operate them. A quote:

This just gives feelings of less "compromised account" and more "Your account is now our account"

AdmiralAsshat1y ago

Yikes! Do you have any info on the individual's background or possible motivations?

13 more replies

mikolajw1y ago

Tukaani website states "jiatan" as the nickname of the malicious code committer on Libera Chat.

WHOWAS jiatan provided me the following information:

jiatan ~jiatan 185.128.24.163 * :Jia Tan jiatan 185.128.24.163 :actually using host jiatan jiatan :was logged in as jiatan tungsten.libera.chat :Fri Mar 14:47:40 2024

WHOIS yields nothing, the user is not present on the network at the moment.

Given that 185.128.24.163 is covered with a range-block on the English Wikipedia, it appears this is a proxy.

mrcoffee4u1y ago

can someone ELI5 ?

fullstackchris1y ago

pRoBaBlY a StaTe AcToR

zero definition of what that means...

egos of people who just like to say cool words they don't understand

lol

this comment will probably get deleted, but let the action of this comment being deleted stand that in 2024 we're all allowed to use big words with no definition of what they mean -> bad

i mean you're seriously going to say "state actor playing the long game" to what end? the issue was resolved in 2 hours... this is stupid

throwaway677431y ago

It's always Debian, like last time when they removed RNG randomness from ssh because of a warning.

mise_en_place1y ago

This is why we never upgrade software versions. I’ve been asked by our customers why we use such an old AMI version. This is why.

circusfly1y ago

Waiting for the new YouTube videos on this. "Woah! Linux has a back door dudes!". My distribution, Ubuntu (now Kubuntu) 2022 isn't affected.

stephc_int131y ago

I guess that rewriting liblzma in Rust would not have prevented this backdoor. But would have likely increased the confidence in its safety.

Using the build system (and potentially the compiler) to insert malicious backdoors is far from a new idea, and I don't see why this example would the only case.

shortsunblack1y ago

Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/

Zigurd1y ago