Do I have to babysit updates and other questions from a new user

Hello everyone,

I’m quite new to rolling release and RPM based distros having used and administered point-release Debian based ones for over a decade.

I have some questions that are more or less tied to each other.

  1. I see with Fedora (more) and SUSE (less), the package maintainer does not ship /etc configs, if there is one it’s a bare bones minimal one with virtually no comments or explanation. In Debian for example, the maintainer and package manager sets this up to sane defaults with commented out config options and explanations that can be modified relatively easy. To those who haven’t used Debian, a good example I can think of is the difference in /etc/default/grub between Fedora (lean/bare bones) and SUSE/Debian (what I would call normal or full fat).

Is the user expected to create their own /etc configs from reading manpages or from /usr/etc or somewhere else in /usr. How do you handle this?

  1. How does zypper and rpm handle config updates that both the user and the package maintainer modify on distribution upgrade (dup)? For example, as the maintainer ships /etc/default/grub and I modify it, will zypper ask me what to do on dup, as in keep the original (old modified by me), use the new shipped by maintainer or view the diff and make a choice? So far I have not encountered such questions.

  2. How do you handle updates, specifically are you present while doing them or do you non interactively do them with a systemd timer/service every day? I’m afraid of being in this guy’s position of having a broken system, to be fair I’m not sure I would’ve fared better handling that manually.

Thank you,
Pavin.

@pavinjoseph Tumbleweed did a /usr/etc merge some time back where all the defaults are, /etc is for the admin to configure as required to override defaults append etc.

1 Like

I don’t “babysit” updates. I do something else while the update is proceeding.

However, after an update, I do run (as root)

rpmconfigcheck

Take your “/etc/default/grub” as an example.

There could be a file “/etc/default/grub.rpmnew” as a suggested new version. In that case, I use “diff” to examine the changes and modify as needed. And, when done, I remove the “*.rpmnew” version.

Sometimes an update will instead rename the existing config file, and install a replacement. In that case “/etc/default/grub.rpmsave” would be the old one. Again, I diff and update as needed, then remove the “*.rpmsave” version.

The command “rpmconfigcheck” will list any “*.rpmnew” and “*.rpmsave” files that it finds.

2 Likes

Ah okay, that makes sense from an admin perspective.

Great, that’s exactly what I was looking for :slightly_smiling_face:

I do have one more question though, is there an existing package/script that I can use to “stage” an upgrade using a snapshot. If it succeeds, that can be my new root, otherwise it can log the issue and I can either fix the issue myself or file a bug report and wait for a fix. Basically I want to avoid breaking the system and having to do snapper rollback, I’m very well used to this workflow but I and someone else on this forum recently did manage to somehow lose the grub menu snapshot entries during a broken dup. So it cannot be relied upon completely. Besides as they say, isn’t prevention better than a cure? :wink:

I understand the A/B root system used by MicroOS/Aeon et al. but don’t want to deal with the restrictions of an immutable distro. Basically I want A/B root for dup only.

If there isn’t an existing way to do this, I think I will roll this into zypperoni.

When using software “found somewhere”, always ask your self if it is safe to depend on it. E.g. is continuity guaranteed (only one maintainer that has his personal hobbyhorse is not as good for the future as a well know group of developers). How is support organised. Etc., etc.

1 Like

No, to the best of my knowledge, there isn’t. You can choose “Transactional Updates” (I believe the option is) from the Tumbleweed dvd installer, but I don’t actually know what all that entails, and if it would make Tumbleweed behave just like MicroOS or not, as I’ve never tried it.

1 Like

@pavinjoseph I’m interested in these ‘restrictions’ you talk of?

I have a couple of MiniPC’s with MicroOS one with rke2 and and the other with k3s, they update and reboot (in the small hours of the morning), just about zero hand holding…

If your installing just a package or two, there is transactional-update apply and moves to the new snapshot, no reboot required. I use Cockpit to manage as well…

1 Like

This particular problem won’t be an issue with zypperoni as I wrote and maintain it. Quite simple single file python script actually :wink:

Thanks, I saw this option during the installation as well. But it’s for the server role, for desktops there are other options such as Aeon/Gnome though when I tried it in a VM it was horribly broken.

Basically I want access to the root filesystem aside from treating it as a place where you “put packages” via a single tool transactional-update. I don’t doubt this is working well for servers though I am yet to try it. But desktops like Aeon/Gnome was too buggy when I tested it recently.

Anyway, it looks like transactional-update would do just fine for what I am looking to accomplish:

Q: Can I use transactional-update without a read-only root filesystem?
A: Yes, transactional-update will work without a read-only root filesystem. However users who choose this approach should be aware that when they reboot their rootfs will be the one created at the time of the transactional-update . This potentially means losing any custom changes made in the time between the transactional-update and the reboot . Therefore the recommendation would be to immediately reboot after every update if you’re not using a read-only filesystem.
Source: Transactional Updates: Refinements & FAQ - openSUSE Kubic - Retired

An example that just popped into my mind. So today I was experimenting with some of the more user facing stuff in my ongoing migration to Tumbleweed. To install spotify I did opi spotify. Great, only it won’t open because a library it expects isn’t in the same location on TW. A simply ln -s <TW-lib> <path-it-expects> fixed it. How would one go about doing such easy fixes on immutable distros? The answer is always the same, “you should try flatpaks”. :melting_face:

@pavinjoseph if you want to (at some point) borkyour system, then rock on with opi, don’t use it, it’s not recommended… stick with the default (supported) repositories, oss, non-oss and updates. There should be no need to add any more…

Aeon/Kalpa only use flatpaks… or distrobox…

1 Like

I’ll keep that in mind, had no intention of filling up my system with build tools and dependencies anyway.

Distrobox is cool though :hugs:

zypper has grown up, no babysitting or hand holding required:

  1. Solver defaults are annoying. Tweaked them:
erlangen:~ # grep ^solver /etc/zypp/zypp.conf
solver.onlyRequires = true
solver.allowVendorChange = true
solver.dupAllowVendorChange = true
solver.cleandepsOnRemove = true
erlangen:~ # 
  1. Repository priorities for the real world user
    “Use additional repositories with confidence. The topic of additional repositories comes up on a regular basis. The official position is to submit everything to Factory to avoid the issue, but for a large number of reasons this cannot always be the case. As such users living in the real world have to navigate the unsupported landscape. By far the best approach is to use repository priorities available through libzypp, but unfortunately this is not well known.”

  2. Use a systemd service:

erlangen:~ # systemctl cat dup.*
# /etc/systemd/system/dup.service
[Unit]
Description=zypper dist-upgrade
After=purge-kernels.service btrbk.service

[Service]
ExecStartPre=/usr/local/bin/check-opensuse
ExecStart=/usr/bin/zypper --non-interactive dist-upgrade

# /etc/systemd/system/service.d/toplevel-override.conf
[Unit]
OnFailure=failure-notification@%n

# /etc/systemd/system/dup.timer
[Unit]
Description=zypper dist-upgrade

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
erlangen:~ # 
1 Like

Thank you Karl for the detailed reply :slightly_smiling_face:

Does you workflow involve rebooting the machine (hard/soft) according to the output of zypper ps every morning?

I’m guessing /usr/local/bin/check-opensuse refreshes the repos?

Thanks!

For anyone interested, it was quite easy to get transactional-update working on a normal read-write TW install, no setup involved.

# install it
zypper in transactional-update
# perform dup and reboot
sudo transactional-update --drop-if-no-change cleanup reboot dup

It creates a read-write snapshot, runs zypper dup within a chroot environment of that snapshot, sets the new snapshot as the default subvolume, and reboots. Deletes the snapshot if there are no changes or the dup fails. Pretty straightforward :slight_smile:

Here’s my systemd unit files for taking offsite backups and performing transactional update at 3 AM.

# /etc/systemd/system/bkupd.service
[Unit]
Description=Backup and transactional update/reboot

[Service]
Type=oneshot
ExecStartPre=/usr/bin/borgmatic --verbosity 1 --syslog-verbosity 1
ExecStartPre=/usr/bin/snapper -c root create -c number -d 'RO snap before dup'
ExecStartPre=-/usr/bin/zypperoni ref
ExecStartPre=-/usr/bin/zypperoni --no-confirm dup-download
ExecStart=/usr/sbin/transactional-update --drop-if-no-change cleanup reboot dup
# /etc/systemd/system/bkupd.timer
[Unit]
Description=Backup and transactional update/reboot

[Timer]
Unit=bkupd.service
OnCalendar=*-*-* 03:00:00

[Install]
WantedBy=timers.target
# verify and enable
systemd-analyze verify /etc/systemd/system/bkupd.service
systemd-analyze verify /etc/systemd/system/bkupd.timer
systemctl enable bkupd.timer
2 Likes

Thank you for coming back and explaining this, @pavinjoseph

This will probably be very helpful to a few users out there.

1 Like

Most of the time I postpone reboot and continue with the old kernel for some time.

Repos of host erlangen have autorefresh enabled. For many systems ExecStartPre=/usr/bin/nm-online works fine. It broke when enabling wake-on-LAN. I replaced it by:

erlangen:~ # cat /usr/local/bin/check-opensuse
#!/bin/bash 
while true ; do
        curl -I -s cdn.opensuse.org | grep "200 OK" > /dev/null && exit
        sleep 1
        echo sleep
done 
erlangen:~ # 
1 Like

Running critical procedures as a service in the system slice is a great idea.

Backups On infamous host erlangen and its siblings use btrbk, with is extremely easy to use and lightweight.

1 Like

Thanks, your cat looks cute :heart_eyes_cat:
Mine is sleeping off the morning’s work of having two breakfasts :joy_cat:

I’m surprised programs continue to work without being restarted at least by a soft reboot as they’re using old/deleted files.

It’s not the default for a system .service unit file?

By default, service and scope units are placed in system.slice, virtual machines and containers registered with systemd‐machined(8) are found in machine.slice, and user sessions handled by systemd‐logind(8) in user.slice. See systemd.special(7) for more information.
Source: man 5 systemd.slice

Thanks, I will have to give this a try. I use baksnapper on my main workstation for incremental backups to a local drive. But my offsite backups are on an ext4 target, so borg would have to do for now. Tbh, I am not 100% certain all my backups should be on the same FS type, considering btrfs’ spotty history.

When updated while running Firefox prompts for restarting. Sometimes I use Ctrl+Alt+Backspace and login again.

How to Update Tumbleweed says:

" It’s generally safer to logout of your desktop environment (DE) and upgrade from a virtual terminal: This avoids anything related to the DE freezing or dying in the middle of the upgrade."

However this runs in the user slice:

erlangen:~ # systemctl status session-5.scope 
● session-5.scope - Session 5 of User root
     Loaded: loaded (/run/systemd/transient/session-5.scope; transient)
  Transient: yes
     Active: active (running) since Tue 2024-02-13 07:03:40 CET; 12min ago
      Tasks: 2
        CPU: 173ms
     CGroup: /user.slice/user-0.slice/session-5.scope
             ├─ 1242 "login -- root"
             └─24823 -bash

Feb 13 07:03:40 erlangen systemd[1]: Started Session 5 of User root.
Feb 13 07:03:40 erlangen login[1242]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
Feb 13 07:03:40 erlangen login[1242]: ROOT LOGIN ON tty1
erlangen:~ # 

Successfully tested for several months: Best backup format for SSD's - KDE - #12

Your question already contains the answer. Programs continue to work using old versions of files, so you do not benefit from the update until programs are restarted. In the extreme case if program is split into several independent components (like plugins) and interface between them is changed incompatibly, attempt to load updated component by the currently running program will fail in the best case or behave unpredictably in the worst.