Binaries in /usr/local

I am trying to transfer my workload to OpenSuse Aeon and currently have hit a snag - we use Kolide for system authentication and it requires installation in the base OS. At least I ‘think’ it does, as it needs to be able to use osquery to determine that the HDD is encrypted and that various browsers/plugins are installed.

Issue I have been hitting is that the .rpm installs the binaries required to run in /usr/local/kolide-k2. On restart, the service cannot start as these binaries do not exist - at least there is no kolide folder in /usr/local

Running mount I can see that the mount point for /usr/local is set to be BTRFS subvol /usr/local

mount | grep local
/dev/nvme0n1p3 on /usr/local type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=264,subvol=/@/usr/local)

However, if I enter the transactional-update shell - I can see the additional directory in /usr/local. Running the mount command here shows that the BTRFS subvol /usr/local is NOT mounted (expected I assume as it is the ‘current’ image), however, the only drive mount which would contain /usr/local is root.

/dev/nvme0n1p3 on / type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=272,subvol=/@/.snapshots/8/snapshot)

I don’t know how the snapshots map for /usr/local - but it looks like the correct mount point for /usr/local is not being used inside the transactional-update shell? Files are present in the transactional-update shell, but not in the current snapshot.

Should also note, I tried installation of the RPM using transactional-update pkg in prior to dropping into the shell and I undertook a reboot after the install.

I can work around this by updating the service file, however, it looks like there may be something (possibly my understanding) wrong with the mapping of /usr/local?

@2bitstu Hi and welcome to the Forum :smile:
So if you run transactional-update pkg install /path/to/your.rpm and then transactional-update apply this should suffice?

No need to use shell (it’s really for debug), if anything transactional-update run would be the way, eg transactional-install run <some command>.

@malcolmlewis - thanks for the quick response, trying to get involved more :slight_smile:

I installed initially using transactional-update pkg install /local/file.rpm and restarted and this was the behaviour I saw - this is a fresh install of Aeon otherwise. I have just rolled back to initial installation snapshot and started again and still have the same thing showing. I only dived into the shell to try and see what was going on ( I have heeded the warnings from Richard Brown’s presentations! Mostly).

So process I have seen:

  1. Install via transactional-update pkg install
  2. Run transactional-update apply
  3. Reboot
  4. sudo ls /usr/local → Does NOT show the directory `kolide-k2’
  5. transactional-update shell
  6. ls /usr/local → DOES show the directory `kolide-k2’

It just seems like something isn’t updating the correct location of /usr/local, but I don’t understand the snapshot mapping enough to be able to tease out why.

@2bitstu you don’t need to reboot after apply, when you enter shell it’s a new snapshot, so it should be there after apply. After apply, have you run the command which kolide-k2?

Nice to know it doesn’t need a reboot after apply - felt like some belt and braces.

The actual binary is launcher but none of the potential names for it come up with which - I don’t even think the binary gets added to PATH normally - the service file expects it at /usr/local/kolide-k2/launcher.

I have done a bit more digging - output from snapper list shows that I am currently using snapshot #9.

If I list the contents of /usr/local in my current snapshot ls /.snapshots/9/snapshot/usr/local it prints out kolide-k2 as the only directory - assuming this is because it is an overlay.

Still running ls /usr/local itself doesn’t show that directory.

@2bitstu This one

Yes that’s the one

@2bitstu is there a public download of the rpm?

That’s a good point, I can’t see one - mine has it has a secret baked into it so I can’t share that I’m afraid.

@2bitstu ok, so an internal build? If so, do you have access to the src rpm? I’m interested in the spec file used to build…

@malcolmlewis - I only have access to the distributed RPM.
Not sure this helps, but this is all that is inside the RPM I have.

├── etc
│   └── kolide-k2
│       ├── launcher.flags
│       └── secret
├── usr
│   ├── lib
│   │   ├── .build-id
│   │   │   ├── cb
│   │   │   │   └── 130291c70322fe2d39b80b3e8ddda8a8c772be -> ../../../../usr/local/kolide-k2/bin/launcher
│   │   │   └── f8
│   │   │       └── d36129d69697de125dcf02a534e215c21e4bc4 -> ../../../../usr/local/kolide-k2/bin/osqueryd
│   │   └── systemd
│   │       └── system
│   │           └── launcher.kolide-k2.service
│   └── local
│       └── kolide-k2
│           └── bin
│               ├── launcher
│               └── osqueryd
└── var
    └── kolide-k2

@2bitstu when you install the rpm, what does the output look like, all ok?

Yes all goes OK, no warnings errors etc.

@2bitstu So it’s a Centos rpm by the looks… maybe it’s macros are doing something funky…

I think there is an issue with the fstab/mount setup and overlays.

When transactional-update installs the rpm, it installs the /usr/local files into /.snapshots/<next-snapshot>/snapshot/usr/local
I can verify this by seeing it the snapshot.

Looking at the full output of mount (other output chopped):

/dev/nvme0n1p3 on / type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=273,subvol=/@/.snapshots/9/snapshot)
/dev/nvme0n1p3 on /usr/local type btrfs (rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=264,subvol=/@/usr/local)

The root filesystem is mounted correctly, pointing to the correct snapshot, but the @/usr/local is then mounted over the snapshots version of /usr/local. I can show this by doing the following:

# ls /usr/local
> bin  include  lib  lib64  libexec  man  sbin  share  src
# umount /usr/local
# ls /usr/local
> kolide-k2

It looks like the one /usr/local should be overlaid on top of the snapshot (or other way around), but it looks like it is just mounted, which covers what is present in the snapshot?

EDIT: The service for this fires up and it all starts working as soon as I un-mount /usr/local. Only problem being all the system stuff that is no longer accessible at /usr/local

@2bitstu do you use IRC or Matrix?

@malcolmlewis - yes haven’t used either in a while - but am on Matrix

@2bitstu OK, so there is a channel for Aeon, this is Aeon RC2? | Bridged with openSUSE 🦎 and Telegram: Contact @openSUSE_Aeon | Development of the Aeon desktop

1 Like

Cool I’ll head along. Yes Aeon RC2

1 Like

transactional-update only mounts the minimal set of filesystems. What may work - enter shell with transactional-update shell, mount /usr/local and any other required mount point manually, install your package. All this without exiting transactional-update shell session. Do not forget to unmount them before exiting transactional-update shell to avoid any error when completing the task.