MicroOS on Raspberry Pi, want to move /var to spinning disk

[UPPERKEES]

Because an SD card last longer if you don’t write that much to it, I want to move the /var partition to my spinning disk which is connected via USB. /var contains of course log files, but also my container images.

What I’ve done is basically this (based on this blog post):

• Create a btrfs partition with mkfs.btrfs -L spinning /dev/sda
• Then I mounted it under /mnt
• I created a root @ dir with btrfs su create /mnt/@
• Then I created a read-only snapshot of /var with btrfs su create -r /var/ /var/backup
• I sent it to the new disk with btrfs send /var/backup | btrfs receive /mnt/backup
• Then I changed the read-only snapshot to a regular one with btrfs subvolume snapshot /mnt/backup /mnt/@/var
• Removed the old snapshots.
• Edited my fstab, where I substitute the UUID with my LABEL.
• I ran touch /var/old to mark the old /var.
• As a test, when I run mount -a, it indeed mounts it and the /var/old file is gone.
• When I reboot it, it doesn’t mount /var from my USB HDD and I see /var/old.
• When I run mount -a, it mounts the /var from my USB HDD again.

What am I missing here? Is there something else I need to modify? As far as I see, snapper is not using /var. The docs also state this. So I suppose it’s not snapper causing issues. Maybe something in the boot process?

[arvidjaar]

/var contains the writable /etc overlay and is mounted in initrd already. initrd looks for /etc/fstab on real root, not on overlay. When you edited /etc/fstab, you edited it in overlay, not on real root.

I suppose you can use transactional-update shell, unmount /etc and modify /etc/fstab but I have not tested it.

[UPPERKEES]

After doing the transactional-update shell or simply transactional-update pkg install some random package, the system doesn’t boot anymore. I tried it on 2 other Pi’s as well. They ended up dead as well.

I did do a snapper diff and I did see my new line in /etc/fstab.

Is there maybe some documentation about moving a partition onto another disk with MicroOS? Or some other resource to get this sorted?

[geoW]

@UPPERKEES

some documentation about moving a partition onto another disk with MicroOS?

I never did it, but seems it could be worth a try
Portal:MicroOS/Ignition

[UPPERKEES]

I could give that a try.

[malcolmlewis]

@UPPERKEES also look at Fuel Ignition: Config Generator

[UPPERKEES]

I’m familiar with the tool. But custom partitioning is not included in the config generator, right? I’m making the JSON stuff myself now, based on the docs.

[arvidjaar]

Well, I gave it a quick try and it worked just fine.

/             /dev/mapper/cr_root[/@/.snapshots/5/snapshot] btrfs   ro,relatime,seclabel,discard=async,space_cache=v2,subvolid=268,subvol=/@/.snapshots/5/snapshot
├─/srv        /dev/mapper/cr_root[/@/srv]                   btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=262,subvol=/@/srv
├─/.snapshots /dev/mapper/cr_root[/@/.snapshots]            btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=257,subvol=/@/.snapshots
├─/home       /dev/mapper/cr_root[/@/home]                  btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=259,subvol=/@/home
├─/opt        /dev/mapper/cr_root[/@/opt]                   btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=260,subvol=/@/opt
├─/usr/local  /dev/mapper/cr_root[/@/usr/local]             btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=264,subvol=/@/usr/local
├─/root       /dev/mapper/cr_root[/@/root]                  btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=261,subvol=/@/root
├─/var        /dev/vdb1                                     btrfs   rw,relatime,seclabel,discard=async,space_cache=v2,subvolid=5,subvol=/
└─/etc        overlay                                       overlay rw,relatime,seclabel,lowerdir=/sysroot/var/lib/overlay/4/etc:/sysroot/etc,upperdir=/sysroot/var/lib/overlay/5/etc,workdir=/sysroot/var/lib/overlay/5/work-etc,uuid=on

I checked in transactional-update shell and apparently it merges the current writable /etc content into the new snapshot because it already had the right line for new /var. But you must generate new snapshot with new /etc after changing /etc/fstab.

I did not use btrfs send/receive, just tar -c | tar -x. I explicitly enabled SELinux support in tar to make sure SELinux labels are correct (I superficially checked that different files have different labels). It could potentially be a problem, I do not know if btrfs send/receive preserves SELinux labels.

The steps (from memory) were

1. Create new btrfs filesystem
2. Mount it on the temporary mount point /tmp/var
3. Use cd /var; tar -c --selinux . | tar -C /tmp/var -x --selinux
4. Edit /etc/fstab, change /var line
5. Run transactional-update shell. I did not change anything, just checked that /etc/fstab was correct.
6. Reboot.

I did not have anything using /var except systemd-journald and auditd so I did not bother with snapshots.

So it seems to work and you need you troubleshoot why it does not work for you following the normal steps starting with reading logs.

P.S. off-topic, but - this VM is image with systemd-boot and FDE using systemd-pcrlock. When I added the second disk it apparently invalidated TPM2 policy and now I cannot update it. Anyone knows how to reinitialize policies and enroll root again?

[UPPERKEES]

This config generator is great! But it doesn’t include all documented features. Partitioning and mounting stuff isn’t included in that generator. And those not supported additions are removed when loading in my custom config.

And for some reason I cannot re-run my Ignition/Combustion stuff, because the first boot flag is not set as it should be.

/boot/writable/firstboot_happened not created after Ignition/Combustion finished

I’m using a Raspberry Pi with MicroOS and I wanted to force my already setup installation to re-run Ignition/Combustion. In the docs it states that /boot/writable/firstboot_happened is created after the first successful boot. My system has rebooted many times and Ignition/Combustion did install some packages and created my user with my public key on its first boot. So it did run, but it did not leave the flag. Combustion/Ignition also isn’t ran at every boot because of this missing flag, becau…

I’ll have a look again this weekend. Hopefully I can find a working solution. But to reflash my SD card every time makes it a bit tedious.

[arvidjaar]

@arvidjaar

I did not use btrfs send/receive, just tar -c | tar -x.

Well, I repeated with btrfs send/receive and it worked just fine. I checked that SELinux labels were correct on /var clone.

[arvidjaar]

@UPPERKEES

I tried it on 2 other Pi’s as well. They ended up dead as well.

I suspect initrd does not include drivers for your HDD controller.

[UPPERKEES]

Yeah, I have to run this with a monitor attached. The config should be fine, since the Combustion validator command doesn’t complain. It may be indeed a missing driver or something like that. Or some other variable that messes this up. But I think the best way is to set this up with Combustion/Ignition at boot time. The partitioning of the USB HDD and then the change to the partition table and where /var should be mounted. Then it’s also nicely reproducible.

[UPPERKEES]

I’ll do the the attached monitor thing next week, in the meantime I think I’ll configure my Ignition config in yaml and use the Butane tool to convert it to json since the OpenSUSE Ignition generator looks nice, but hasn’t implemented all options.