Which log should I look at?
symptoms
My tumbleweed vm stops.
The cursor is stopped without an error.
TW doesn’t respond to ping.
Even if I shut down in proxmox, it won’t be turn off.
Only reset or stop work.
This symptoms has been happening for about two weeks.
Proxmox has not been updated recently, and only tumbleweed is supposed to be updated automatically, so tumbleweed is suspected.
But I have no idea which log to look at.
would use
journalctl -b
and
dmsg
And go to the man pages of those.
Dmesg doesn’t show the previous log.
It only shows after booting.
I haven’t tried journal -b, but I’ll try it.
journal -b is still not visible before booting.
Can anyone help me?
Edit the boot line and remove splash=silent and quiet.
Being a VM I would just set up a new Tumbleweed VM.
@cola16 to see the previous boot log in reverse (as in when you had to force off) use journalctl -r -b -1 or -2 etc to go back to others…
The machine stopped at 3:14 again today.
I do a health check every minute. Therefore, it worked at 3:13.
7월 15 03:13:46 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:13:46+09:00" level=info msg="Server 192.168.10.33:9345@PREFERRED->HEALTHY from successful health check"
7월 15 03:13:46 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:13:46+09:00" level=info msg="Server 192.168.10.33:6443@PREFERRED->HEALTHY from successful health check"
7월 15 03:13:28 agent2.k8s.rack.cola16 sshd-session[329111]: Connection closed by 10.42.7.91 port 57304
7월 15 03:13:08 agent2.k8s.rack.cola16 sshd-session[328117]: Connection closed by 10.42.7.91 port 54826
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDEND (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDOUT (rtt min/avg/max/mdev = 3.082/3.281/3.481/0.199 ms)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDOUT (2 packets transmitted, 2 received, 0% packet loss, time 251ms)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDOUT (--- 1.1.1.1 ping statistics ---)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDOUT ()
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327680]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327680]: (root) CMDOUT (user)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327681]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327679]: (root) CMDOUT (PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327680]: (root) CMDOUT (system)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327681]: (root) CMDOUT (user)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327681]: (root) CMDOUT (system)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327680]: (root) CMDOUT (ssd)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327681]: (root) CMDOUT (ssd)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327684]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327683]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:13:00 agent2.k8s.rack.cola16 CRON[327682]: (root) CMD (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:12:48 agent2.k8s.rack.cola16 sshd-session[327089]: Connection closed by 10.42.7.91 port 42174
7월 15 03:12:46 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:46+09:00" level=info msg="Server 192.168.10.33:9345@RECOVERING->PREFERRED from successful health check"
7월 15 03:12:46 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:46+09:00" level=info msg="Server 192.168.10.33:6443@RECOVERING->PREFERRED from successful health check"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Server 192.168.10.33:9345@UNCHECKED->RECOVERING from successful health check"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Server 192.168.10.33:6443@UNCHECKED->RECOVERING from successful health check"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Remotedialer connected to proxy" url="wss://192.168.10.33:9345/v1-rke2/connect"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Connecting to proxy" url="wss://192.168.10.33:9345/v1-rke2/connect"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Started tunnel to 192.168.10.33:9345"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Updated load balancer rke2-agent-load-balancer server addresses -> [192.168.10.31:9345 192.168.10.32:9345 192.168.10.33:9345] [default: server-k8s-rack.cola16.dev:9345]"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Adding server to load balancer rke2-agent-load-balancer: 192.168.10.33:9345"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer server addresses -> [192.168.10.31:6443 192.168.10.32:6443 192.168.10.33:6443] [default: server-k8s-rack.cola16.dev:6443]"
7월 15 03:12:45 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:45+09:00" level=info msg="Adding server to load balancer rke2-api-server-agent-load-balancer: 192.168.10.33:6443"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Proxy done" err="context canceled" url="wss://192.168.10.33:9345/v1-rke2/connect"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Stopped tunnel to 192.168.10.33:9345"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Updated load balancer rke2-agent-load-balancer server addresses -> [192.168.10.31:9345 192.168.10.32:9345] [default: server-k8s-rack.cola16.dev:9345]"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Removing server from load balancer rke2-agent-load-balancer: 192.168.10.33:9345"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Updated load balancer rke2-api-server-agent-load-balancer server addresses -> [192.168.10.31:6443 192.168.10.32:6443] [default: server-k8s-rack.cola16.dev:6443]"
7월 15 03:12:34 agent2.k8s.rack.cola16 rke2[1575]: time="2025-07-15T03:12:34+09:00" level=info msg="Removing server from load balancer rke2-api-server-agent-load-balancer: 192.168.10.33:6443"
7월 15 03:12:28 agent2.k8s.rack.cola16 sshd-session[326093]: Connection closed by 10.42.7.91 port 58422
7월 15 03:12:08 agent2.k8s.rack.cola16 sshd-session[325096]: Connection closed by 10.42.7.91 port 40872
7월 15 03:12:01 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDEND (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:12:01 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDOUT (rtt min/avg/max/mdev = 3.349/3.516/3.683/0.167 ms)
7월 15 03:12:01 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDOUT (2 packets transmitted, 2 received, 0% packet loss, time 251ms)
7월 15 03:12:01 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDOUT (--- 1.1.1.1 ping statistics ---)
7월 15 03:12:01 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDOUT ()
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324713]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324712]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324713]: (root) CMDOUT (user)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324713]: (root) CMDOUT (system)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324712]: (root) CMDOUT (user)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324712]: (root) CMDOUT (system)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324711]: (root) CMDOUT (PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324713]: (root) CMDOUT (ssd)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324712]: (root) CMDOUT (ssd)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324716]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324715]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:12:00 agent2.k8s.rack.cola16 CRON[324714]: (root) CMD (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:11:48 agent2.k8s.rack.cola16 sshd-session[324063]: Connection closed by 10.42.7.91 port 59560
7월 15 03:11:28 agent2.k8s.rack.cola16 sshd-session[323064]: Connection closed by 10.42.7.91 port 52592
7월 15 03:11:08 agent2.k8s.rack.cola16 sshd-session[322063]: Connection closed by 10.42.7.91 port 52320
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDEND (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDOUT (rtt min/avg/max/mdev = 3.118/3.184/3.250/0.066 ms)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDOUT (2 packets transmitted, 2 received, 0% packet loss, time 250ms)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDOUT (--- 1.1.1.1 ping statistics ---)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDOUT ()
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321665]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321666]: (root) CMDEND (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321665]: (root) CMDOUT (user)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321666]: (root) CMDOUT (user)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321664]: (root) CMDOUT (PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321665]: (root) CMDOUT (system)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321666]: (root) CMDOUT (system)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321666]: (root) CMDOUT (ssd)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321665]: (root) CMDOUT (ssd)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321668]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ls -al /_DATA | grep $i| grep -q '?'; then echo remount $i; umount -f /_DATA/$i; mount /_DATA/$i; mount -o remount /_DATA/$i; mount -a; fi; done)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321669]: (root) CMD (for i in `cat /etc/fstab | grep ceph | awk '{print $2}'| cut -d/ -f3`; do echo $i; if ! mount | grep -q /_DATA/$i; then echo mount $i; mount /_DATA/$i; mount -a; fi; done)
7월 15 03:11:00 agent2.k8s.rack.cola16 CRON[321667]: (root) CMD (/usr/bin/fish -c 'if not begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1 ; or not begin; sleep 5; end; or begin; ping -c 2 -W 1 -i 0.25 -q 1.1.1.1; end; end; echo "restart netbird" | mail -s (date)(hostname -f)": cron: restart netbird" root; systemctl restart netbird; end')
7월 15 03:10:48 agent2.k8s.rack.cola16 sshd-session[321028]: Connection closed by 10.42.7.91 port 42352
7월 15 03:10:28 agent2.k8s.rack.cola16 sshd-session[320027]: Connection closed by 10.42.7.91 port 45520
lines 1-80
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.