How to install kubeadm so that the immutability does not break his operation

English Virtualization
1

I tried to setup Kubernetes like here Kubernetes: The cluster-info ConfigMap does not yet contain a JWS signature for token ID "462w80", will try again
but after checking the database

kube-system 37s Warning FailedMount pod/kube-controller-manager-m0.k8b.raymwm.mwm MountVolume.SetUp failed for volume flexvolume-dir : mkdir /usr/libexec/kubernetes: read-only file system

happens that the inmutability of /usr is breaking kubeadm setup, so how can I solve this?

2

@Misekayek0l transactional-update shell probably, I’m not a kubeadm user, I just use k3s and rke…

3

I used it in all aperently required steps, for this suggestion to be more meaningful do you meant that I should run kudeadm init inside the transactional shell?

4

Yes, you get a mutable version of the root fs inside TU shell.

5

Please do not use transactional-update shell for system management. It bypasses many of the safety checks and can potentially lead to problems.

If you need to run kubeadm init in the system root, use transactional-update run kube init

shell is primarily only intended for testing/debugging, and using it for system administration can lead to unsupported and/or broken configurations.

1 Like
6

I doubt it would work anyway, it requires that kubelet service is running.

7

Services should be handled just like any other service would on Tumbleweed or Leap, when you’re using Leap Micro/MicroOS.

systemctl start servicename You generally shouldn’t need to touch transactional-update for that. Unless there’s something screwy in the packaging.

1 Like
8

That is not the problem.

9

@Misekayek0l if it’s just that one command, then as @sfalken says, just use run. But I’m not sure why your just not doing everything via combustion before the system goes read-only…

10

Because before setting the combustion script I want to be sure that it works

11

As expected transactional-update run kubeadm init does not work:

nsactional-update run kubeadm init
Checking for newer version.
New version found - updating...
Loading repository data...
Reading installed packages...
Retrieving: transactional-update-4.1.6-150500.3.3.1.x86_64 (SLE Micro Update Repository)                         (1/1),  83.5 KiB
Retrieving: transactional-update-4.1.6-150500.3.3.1.x86_64.rpm ..................................................[done (213.9 KiB/s)]
(1/1) /tmp/transactional-update.8Ry1zwLMgr/repo-sle-update/x86_64/transactional-update-4.1.6-150500.3.3.1.x86_64.rpm ..........[done]
Loading repository data...
Reading installed packages...
Retrieving: libtukit4-4.1.6-150500.3.3.1.x86_64 (SLE Micro Update Repository)                                    (1/2), 174.4 KiB
Retrieving: libtukit4-4.1.6-150500.3.3.1.x86_64.rpm .............................................................[done (849.3 KiB/s)]
(1/2) /tmp/transactional-update.8Ry1zwLMgr/repo-sle-update/x86_64/libtukit4-4.1.6-150500.3.3.1.x86_64.rpm .....................[done]
Retrieving: tukit-4.1.6-150500.3.3.1.x86_64 (SLE Micro Update Repository)                                        (2/2),  79.8 KiB
Retrieving: tukit-4.1.6-150500.3.3.1.x86_64.rpm .................................................................[done (171.2 KiB/s)]
(2/2) /tmp/transactional-update.8Ry1zwLMgr/repo-sle-update/x86_64/tukit-4.1.6-150500.3.3.1.x86_64.rpm .........................[done]
transactional-update 4.1.6 started
Options: run kubeadm init
Separate /var detected.
2024-02-20 20:29:09 tukit 4.1.6 started
2024-02-20 20:29:09 Options: -c3 open
2024-02-20 20:29:12 Using snapshot 3 as base for new snapshot 4.
2024-02-20 20:29:12 /var/lib/overlay/3/etc
2024-02-20 20:29:12 Syncing /etc of previous snapshot 2 as base into new snapshot "/.snapshots/4/snapshot"
2024-02-20 20:29:12 SELinux is enabled.
ID: 4
2024-02-20 20:29:27 Transaction completed.
2024-02-20 20:29:27 tukit 4.1.6 started
2024-02-20 20:29:27 Options: call 4 kubeadm init
2024-02-20 20:29:29 Executing `kubeadm init`:
2024-02-20 20:29:29 Discarding snapshot 4.
Failure (dbus fatal exception).
Cannot delete snapshot 4 since it is the currently mounted snapshot.
ERROR: Calling kubeadm failed: No such file or directory
2024-02-20 20:29:30 Application returned with exit status 1.
2024-02-20 20:29:34 tukit 4.1.6 started
2024-02-20 20:29:34 Options: close 4
2024-02-20 20:29:44 New default snapshot is #4 (/.snapshots/4/snapshot).
2024-02-20 20:29:44 Transaction completed.

Please reboot your machine to activate the changes and avoid data loss.
New default snapshot is #4 (/.snapshots/4/snapshot).
transactional-update finished
12

@Misekayek0l and why would you expect a Tumbleweed rpm to work on Leap Micro?

13

It is not a rpm, but the package manager-less installation from Installing kubeadm | Kubernetes

here is my secondstage.sh script:

#!/bin/bash

echo Schrift 1
#transactional-update --non-interactive --continue up
echo Schrift 2
transactional-update --non-interactive --continue pkg install --no-recommends containerd conntrack-tools socat ethtool

CNI_PLUGINS_VERSION="v1.3.0"
ARCH="amd64"
DEST="/opt/cni/bin"
 mkdir -p "$DEST"
curl -L "https://github.com/containernetworking/plugins/releases/download/${CNI_PLUGINS_VERSION}/cni-plugins-linux-${ARCH}-${CNI_PLUGINS_VERSION}.tgz" |  tar -C "$DEST" -xz

DOWNLOAD_DIR="/usr/local/bin"
 mkdir -p "$DOWNLOAD_DIR"

CRICTL_VERSION="v1.28.0"
ARCH="amd64"
curl -L "https://github.com/kubernetes-sigs/cri-tools/releases/download/${CRICTL_VERSION}/crictl-${CRICTL_VERSION}-linux-${ARCH}.tar.gz" |  tar -C $DOWNLOAD_DIR -xz

RELEASE="$(curl -sSL https://dl.k8s.io/release/stable.txt)"
ARCH="amd64"
cd $DOWNLOAD_DIR
 curl -L --remote-name-all https://dl.k8s.io/release/${RELEASE}/bin/linux/${ARCH}/{kubeadm,kubelet}
 chmod +x {kubeadm,kubelet}

RELEASE_VERSION="v0.16.2"
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubelet/kubelet.service" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" |  tee /etc/systemd/system/kubelet.service
 mkdir -p /etc/systemd/system/kubelet.service.d
curl -sSL "https://raw.githubusercontent.com/kubernetes/release/${RELEASE_VERSION}/cmd/krel/templates/latest/kubeadm/10-kubeadm.conf" | sed "s:/usr/bin:${DOWNLOAD_DIR}:g" |  tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

mkdir -p /etc/modules-load.d

 tee /etc/modules-load.d/containerd.conf << EOF
overlay
br_netfilter
EOF

cat <<EOF |  tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

mkdir -p /etc/sysctl.d
cat <<EOF |  tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

systemctl enable containerd
systemctl enable kubelet
14

@Misekayek0l So investigate/fix the issue Failure (dbus fatal exception).?

15

It work apparently perfect until you try to join a node, then you realize that kube-controller-manager has been in creating for days, then you check the logs and it was trying to create a file in /usr/libexec I have not the exact details, because did not take note, thus why I wanted to use packages because may bypass this problem.

16

So I have already investigated the issue

17

@Misekayek0l so have you moved this process back to the Combustion script before the system goes read-only? Else use transactional-update shell to investigate further.

18

kubeadm init requires the existence of dbus running, so is dbus for when the combustion script is running?

19

@Misekayek0l If it’s installed, use systemctl to start it…

20

But dbus only can be requested by a dependency and looks like my tow services don’t request it:

m0:~ # cat /etc/systemd/system/multi-user.target.wants/containerd.service
# Copyright The containerd Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Modifications by SUSE LLC under the same license

[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
Conflicts=docker.service

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/sbin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
LimitNOFILE=1048576
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
m0:~ # cat /etc/systemd/system/multi-user.target.wants/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target