Tumbleweed today - xz security alert and CVE-2024-3094

English Applications
61

Endless reading for me. (it appears there may be some malware side cases, where other apps / libs are included, so this story may not be over yet - but I’ll not delve there).

I just checked my two TW desktops, which I last booted on Mar 11:
probably vulnerable

I’ve removed the ssh-server package and downgraded xz and liblzma.

Why was my Mar 11 last-accessed TW vulnerable? See the timeline below, esp the two Tumbleweed acquisition entries of xz I’ve bolded.

Timeline quote from an article at JFrog
==== quote plus URL ====

2024, February 24th – JiaT75 releases version 5.6.0 with the malicious build-to-host.m4. At this stage, the malicious payload is fully operational (any subsequent XZ version is compromised). Malicious xz-utils version 5.6.0 pulled by Debian, Gentoo and Arch Linux.

2024, February 27th – Malicious xz-utils version 5.6.0 pulled by Fedora.

2024, March 5th – Malicious xz-utils version 5.6.0 pulled by openSUSE.

2024, March 9th – JiaT75 updates the backdoor’s binaries to an improved version, and releases version 5.6.1. Malicious xz-utils version 5.6.1 pulled by Fedora, Gentoo and Arch Linux

2024, March 10th – Malicious xz-utils version 5.6.1 pulled by openSUSE.

2024, March 11th – Malicious xz-utils version 5.6.1 pulled by Alpine.

2024, March 26th – Malicious xz-utils version 5.6.1 pulled by Debian.

2024, March 29th – A detailed account of the malicious activity found in XZ utils was published on the oss-security mailing list by Andres Freund.
==== end quote ====

1 Like
62

You would have to look at what this script is actually checking to understand why it reported this.

63

That was a rhetorical question. I know exactly what the script is checking for. I also understand what else is possible as a result of the xz compromise (because I’m delving into all the posts happening elsewhere, both directly and the researchers looking elsewhere).

64

You can consider my answer a rhetorical answer, then. :wink:

65

How long have the vulnerable libraries been in Tumbleweed?

66

Revisions of xz - openSUSE Build Service

I am not sure if there is simple way to check when the Tumbleweed snapshot with the specific package revision was published.

1 Like
67

Does anyone know what steps are being taken to prevent similar compromises in the future? :thinking:
Or are the devs still dealing with the fallout and didn’t yet have time to consider future course of action, I hope some of the admins can bring up the issue in the next Heroes meeting or the appropriate equivalent. :white_check_mark:

To me, a layperson who’s been following this incident since it was announced here a few things stand out that could be potential targets for improvement:

  1. Using source tarballs instead of building from the git source directly, the attacker sneakily put some of the more obvious changes in the tarball and not in the git tree. Didn’t know this was a thing, I guess a relic from the past but should be an easy fix to do a full git clone initially and pulling/reviewing future diff carefully.
  2. Tools like Valgrind knew something was wrong and the attacker had to get past its defenses by social engineering, perhaps more testing of this sort could be done by OpenSuse without relying on upstream.
  3. The attack was made largely possible by obscure build scripts, perhaps there is a better way to compile from source without depending on these obscure upstream build scripts? Or if that’s not possible perhaps run the build scripts and code through an anti-malware heuristic scanner to flag suspicious bits for manual review.

Anyway, I really like the concept and largely the execution of Tumbleweed (not having to do a major upgrade like Fedora every 6 months) so this is only meant as constructive criticism. :hugs:

68

There is a simple way to find when the vulnerabilities hit the TW users.

Read @myswtest previous post for timelines, then read the Snapshot updates

2024, March 5th – Malicious xz-utils version 5.6.0 pulled by openSUSE.

2024, March 10th – Malicious xz-utils version 5.6.1 pulled by openSUSE.

… and then:

TW release 20240307 xz (5.4.6 → 5.6.0)

TW release 20240320 xz (5.6.0 → 5.6.1)

New Tumbleweed snapshot 20240320 released! - openSUSE Factory - openSUSE Mailing Lists

2 Likes
69

Re: xz security alert and CVE-2024-3094 - openSUSE Factory - openSUSE Mailing Lists

70

Thank you.

Is there a way to find my zypper dup history? I can find the calls in my bash history but that doesn’t allow me to pinpoint the update exactly enough.

71

/var/log/zypp/history*

2 Likes
72

@sboehringer aside from the history file, using some verbosity will show what is happening on the fly;

zypper -vvv dup

The following 5 packages are going to be downgraded:
  liblzma5            5.6.1.revertto5.4-3.2 -> 5.6.1.revertto5.4-2.1  x86_64  Main Repository (OSS)  openSUSE
  liblzma5-32bit      5.6.1.revertto5.4-3.2 -> 5.6.1.revertto5.4-2.1  x86_64  Main Repository (OSS)  openSUSE
  liblzma5-x86-64-v3  5.6.1.revertto5.4-3.2 -> 5.6.1.revertto5.4-2.1  x86_64  Main Repository (OSS)  openSUSE
  xz                  5.6.1.revertto5.4-3.2 -> 5.6.1.revertto5.4-2.1  x86_64  Main Repository (OSS)  openSUSE
  xz-devel            5.6.1.revertto5.4-3.2 -> 5.6.1.revertto5.4-2.1  x86_64  Main Repository (OSS)  openSUSE
1 Like
73

Tarballs sure look convenient, but I don’t buy into the other arguments they make:

  1. A git clone (normal one, not shallow) of the mainline kernel is around 2.7 GiB and took 1.5 hours to complete IIRC but future checkouts of other branches were near instantaneous. The whole Linux kernel tree is around 10 GiB.
    So caching wise, around 10 GiB total but most likely 3 GiB for the often requested files.
    Whereas there is a binary tarball of around 130 MiB for each minor release, totaling around 26 GiB just for the 6.8 release cycle.
  2. The GNU autotools soup I’m not familiar with at all, but IIUC it makes it easy for the end-user building a package to not have to use the same version of the build tools the developer/maintainer used. This is trivial to solve with the ubiquity of containers?

I don’t know if I’m misconstruing their intentions, but it reads like “everything good with tarballs and obscure build processes, nothing to see here, move on!” :weary:

74

It is common standard for most linux distributions to use tarballs. The build infrastructure from most distros does not allow to download or git clone anything whilst building a package (for obvious security reasons). In the spec file a tarball (offline source) is referenced…

1 Like
75

Let’s be honest, I appreciate all the work going into remediating this but folks just happened to catch this. There are probably any number of subtle compromises worked into libs by state actors currently that we’re unaware of so yeah, while it’s probably worth the major distros re-evaluating their procedures, you’re dealing with very well funded groups who’s entire purpose is to evaluate those procedures and work around them.

Does anyone know what steps are being taken to prevent similar compromises in the future?

2 Likes
76

@butcherbird so if as a user and running unverified sources eg pip then suggest using distrobox to run these applications in. Don’t use sudo, if you need to do admin tasks switch to root user su - perform the task and log out. Don’t use opi/1-click installs from home repositories. Why do folks add development repos, wait for an update to appear in the supported repositories.

I tend to use services and pull direct from git and avoid release tarballs if possible… Source URL’s are verified with upstream on build, if checksums don’t match it bails. Same with source tarballs that’s why in the spec file it requires the full URL to pass the checks. Note these are only for packages that go to development repos and into the release.

2 Likes
77

Are there new information about the malicious payload, especially whether it not uses the known way via sshd?

78

Probably the best resources for that information are the two that are linked earlier in the thread - looks like the jfrog info was updated on the 1st, and the seclists.org thread is ongoing.

79 
5.6.1.revertto5.4-3.2

i note that tw has further reverted xz back from 5.4.6 to 5.4-3.2, which looks lower than the 5.4.5-1.1.x86-64 version i have never exceeded.

has there been any security concern with versions less than 5.6, such that I should worry?

80

Read