Using SUDO makes / partition RO

Hello Geekos!

I’ve been using openSUSE for quite a while and I’m happy about it, esp. Tubleweed giving me exciting stuff :slight_smile: But… something went wrong a couple of hours ago and I’m trying to fix it. I’m using a DELL E6410 and I plugged in my power supply and then stuff started crashing. I rebooted and managed to watch a movie fine. But now I wanted to do some work and discovered, that my / partition is RO!

I did my homework:

https://forums.opensuse.org/showthread.php/389635-Root-Mounting-Read-Only?highlight=root+read-only
https://forums.opensuse.org/showthread.php/478909-Advisory-June-2013-New-Users-beware-of-using-the-BTRFS-filesystem/page7
https://forums.opensuse.org/showthread.php/475020-Read-Only-Root-disk?highlight=root+read-only

http://ubuntuforums.org/showthread.php?t=2136601
http://unix.stackexchange.com/questions/35963/how-to-switch-back-to-writable-mode-without-reboot

and I now know, that the kernel will mount a volume as ro, if it sees possible problems brewing. Fair enough. But what is interesting, I managed to fix this by rebooting and fiddling with grub. I’ve read that all I have to do is to use grub, and posbbily try usng a different kernel.

But, it gets better: all goes back to being ro right after I issue sudo. I have some terminal listings:

As you can see, my /etc/mtab tells me all is fine.

td@linux-7qtq:~> cat /etc/mtab
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
devtmpfs /dev devtmpfs rw,nosuid,size=1912504k,nr_inodes=478126,mode=755 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
/dev/sda1 / btrfs rw,relatime,ssd,space_cache,subvolid=258,subvol=/.snapshots/1/snapshot 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
tmpfs /var/run tmpfs rw,nosuid,nodev,mode=755 0 0
/dev/sda1 /var/log btrfs rw,relatime,ssd,space_cache,subvolid=272,subvol=/var/log 0 0
/dev/sda1 /var/lib/mariadb btrfs rw,relatime,ssd,space_cache,subvolid=269,subvol=/var/lib/mariadb 0 0
/dev/sda2 /home xfs rw,relatime,attr2,inode64,noquota 0 0
/dev/sda1 /var/lib/mailman btrfs rw,relatime,ssd,space_cache,subvolid=268,subvol=/var/lib/mailman 0 0
/dev/sda1 /boot/grub2/i386-pc btrfs rw,relatime,ssd,space_cache,subvolid=259,subvol=/boot/grub2/i386-pc 0 0
/dev/sda1 /var/lib/pgsql btrfs rw,relatime,ssd,space_cache,subvolid=271,subvol=/var/lib/pgsql 0 0
/dev/sda1 /boot/grub2/x86_64-efi btrfs rw,relatime,ssd,space_cache,subvolid=260,subvol=/boot/grub2/x86_64-efi 0 0
/dev/sda1 /var/spool btrfs rw,relatime,ssd,space_cache,subvolid=274,subvol=/var/spool 0 0                                                                        
/dev/sda1 /opt btrfs rw,relatime,ssd,space_cache,subvolid=262,subvol=/opt 0 0                                                                                    
/dev/sda1 /.snapshots btrfs rw,relatime,ssd,space_cache,subvolid=257,subvol=/.snapshots 0 0                                                                      
/dev/sda1 /var/tmp btrfs rw,relatime,ssd,space_cache,subvolid=275,subvol=/var/tmp 0 0                                                                            
/dev/sda1 /usr/local btrfs rw,relatime,ssd,space_cache,subvolid=265,subvol=/usr/local 0 0                                                                        
/dev/sda1 /var/lib/libvirt/images btrfs rw,relatime,ssd,space_cache,subvolid=267,subvol=/var/lib/libvirt/images 0 0                                              
/dev/sda1 /var/opt btrfs rw,relatime,ssd,space_cache,subvolid=273,subvol=/var/opt 0 0                                                                            
/dev/sda1 /tmp btrfs rw,relatime,ssd,space_cache,subvolid=264,subvol=/tmp 0 0                                                                                    
/dev/sda1 /var/crash btrfs rw,relatime,ssd,space_cache,subvolid=266,subvol=/var/crash 0 0                                                                        
/dev/sda1 /var/lib/named btrfs rw,relatime,ssd,space_cache,subvolid=270,subvol=/var/lib/named 0 0                                                                
/dev/sda1 /srv btrfs rw,relatime,ssd,space_cache,subvolid=263,subvol=/srv 0 0                                                                                    
/dev/sdb1 /aux xfs rw,relatime,attr2,inode64,noquota 0 0                                                                                                         
tmpfs /run/user/473 tmpfs rw,nosuid,nodev,relatime,size=384176k,mode=700,uid=473,gid=474 0 0                                                                     
tmpfs /var/run/user/473 tmpfs rw,nosuid,nodev,relatime,size=384176k,mode=700,uid=473,gid=474 0 0                                                                 
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=384176k,mode=700,uid=1000,gid=100 0 0                                                                   
tmpfs /var/run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=384176k,mode=700,uid=1000,gid=100 0 0                                                               
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0                                                                                                         
gvfsd-fuse /run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0                                                            
gvfsd-fuse /var/run/user/1000/gvfs fuse.gvfsd-fuse rw,nosuid,nodev,relatime,user_id=1000,group_id=100 0 0                                                        
tracefs /sys/kernel/debug/tracing tracefs rw,relatime 0 0                                                                                                        

I check, just to be sure, if there are any ro volumes:


td@linux-7qtq:~> cat /etc/mtab | grep " ro"                                                                                                                      
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0

And the /var/log, we’ll see in a moment why:


td@linux-7qtq:~> cat /etc/mtab | grep "/var/log"
/dev/sda1 /var/log btrfs rw,relatime,ssd,space_cache,subvolid=272,subvol=/var/log 0 0

Now, I run yast to do some administration. The ncurses interface gets cut by many black lines with errors when I try actually navigating yast. The errors are in Polish, “filesystem is read-only”.


td@linux-7qtq:~> sudo yast
root's password:
y2log: Error opening logfile '/var/log/YaST2/y2log': System plików wyłącznie do odczytu.
[redacted for brevity]
Błąd we-wy (mkdir failed errno:30 (Read-only file system)).
y2log: Error opening logfile '/var/log/YaST2/y2log': System plików wyłącznie do odczytu.
[redacted for brevity]

Now, let’s get back to /etc/mtab


td@linux-7qtq:~> cat /etc/mtab | grep "/var/log"
/dev/sda1 /var/log btrfs ro,relatime,ssd,space_cache,subvolid=272,subvol=/var/log 0 0


td@linux-7qtq:~> cat /etc/mtab | grep " ro"     
tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0
/dev/sda1 / btrfs ro,relatime,ssd,space_cache,subvolid=258,subvol=/.snapshots/1/snapshot 0 0
/dev/sda1 /var/log btrfs ro,relatime,ssd,space_cache,subvolid=272,subvol=/var/log 0 0
/dev/sda1 /var/lib/mariadb btrfs ro,relatime,ssd,space_cache,subvolid=269,subvol=/var/lib/mariadb 0 0
/dev/sda1 /var/lib/mailman btrfs ro,relatime,ssd,space_cache,subvolid=268,subvol=/var/lib/mailman 0 0
/dev/sda1 /boot/grub2/i386-pc btrfs ro,relatime,ssd,space_cache,subvolid=259,subvol=/boot/grub2/i386-pc 0 0
/dev/sda1 /var/lib/pgsql btrfs ro,relatime,ssd,space_cache,subvolid=271,subvol=/var/lib/pgsql 0 0
/dev/sda1 /boot/grub2/x86_64-efi btrfs ro,relatime,ssd,space_cache,subvolid=260,subvol=/boot/grub2/x86_64-efi 0 0
/dev/sda1 /var/spool btrfs ro,relatime,ssd,space_cache,subvolid=274,subvol=/var/spool 0 0
/dev/sda1 /opt btrfs ro,relatime,ssd,space_cache,subvolid=262,subvol=/opt 0 0
/dev/sda1 /.snapshots btrfs ro,relatime,ssd,space_cache,subvolid=257,subvol=/.snapshots 0 0
/dev/sda1 /var/tmp btrfs ro,relatime,ssd,space_cache,subvolid=275,subvol=/var/tmp 0 0
/dev/sda1 /usr/local btrfs ro,relatime,ssd,space_cache,subvolid=265,subvol=/usr/local 0 0
/dev/sda1 /var/lib/libvirt/images btrfs ro,relatime,ssd,space_cache,subvolid=267,subvol=/var/lib/libvirt/images 0 0
/dev/sda1 /var/opt btrfs ro,relatime,ssd,space_cache,subvolid=273,subvol=/var/opt 0 0
/dev/sda1 /tmp btrfs ro,relatime,ssd,space_cache,subvolid=264,subvol=/tmp 0 0
/dev/sda1 /var/crash btrfs ro,relatime,ssd,space_cache,subvolid=266,subvol=/var/crash 0 0
/dev/sda1 /var/lib/named btrfs ro,relatime,ssd,space_cache,subvolid=270,subvol=/var/lib/named 0 0
/dev/sda1 /srv btrfs ro,relatime,ssd,space_cache,subvolid=263,subvol=/srv 0 0

And mount


td@linux-7qtq:~> mount | grep "ro,"
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
/dev/sda1 on / type btrfs (ro,relatime,ssd,space_cache,subvolid=258,subvol=/.snapshots/1/snapshot)
/dev/sda1 on /var/log type btrfs (ro,relatime,ssd,space_cache,subvolid=272,subvol=/var/log)
/dev/sda1 on /var/lib/mariadb type btrfs (ro,relatime,ssd,space_cache,subvolid=269,subvol=/var/lib/mariadb)
/dev/sda1 on /var/lib/mailman type btrfs (ro,relatime,ssd,space_cache,subvolid=268,subvol=/var/lib/mailman)
/dev/sda1 on /boot/grub2/i386-pc type btrfs (ro,relatime,ssd,space_cache,subvolid=259,subvol=/boot/grub2/i386-pc)
/dev/sda1 on /var/lib/pgsql type btrfs (ro,relatime,ssd,space_cache,subvolid=271,subvol=/var/lib/pgsql)
/dev/sda1 on /boot/grub2/x86_64-efi type btrfs (ro,relatime,ssd,space_cache,subvolid=260,subvol=/boot/grub2/x86_64-efi)
/dev/sda1 on /var/spool type btrfs (ro,relatime,ssd,space_cache,subvolid=274,subvol=/var/spool)
/dev/sda1 on /opt type btrfs (ro,relatime,ssd,space_cache,subvolid=262,subvol=/opt)
/dev/sda1 on /.snapshots type btrfs (ro,relatime,ssd,space_cache,subvolid=257,subvol=/.snapshots)
/dev/sda1 on /var/tmp type btrfs (ro,relatime,ssd,space_cache,subvolid=275,subvol=/var/tmp)
/dev/sda1 on /usr/local type btrfs (ro,relatime,ssd,space_cache,subvolid=265,subvol=/usr/local)
/dev/sda1 on /var/lib/libvirt/images type btrfs (ro,relatime,ssd,space_cache,subvolid=267,subvol=/var/lib/libvirt/images)
/dev/sda1 on /var/opt type btrfs (ro,relatime,ssd,space_cache,subvolid=273,subvol=/var/opt)
/dev/sda1 on /tmp type btrfs (ro,relatime,ssd,space_cache,subvolid=264,subvol=/tmp)
/dev/sda1 on /var/crash type btrfs (ro,relatime,ssd,space_cache,subvolid=266,subvol=/var/crash)
/dev/sda1 on /var/lib/named type btrfs (ro,relatime,ssd,space_cache,subvolid=270,subvol=/var/lib/named)
/dev/sda1 on /srv type btrfs (ro,relatime,ssd,space_cache,subvolid=263,subvol=/srv)
td@linux-7qtq:~> 

I tried doing an smartctl but I’m ro (yes, I can use a live DVD, but today I’m not even seeing straight anymore). I’m considering an fsck, but maybe you have a better idea? I’m a self-starter, so my Linux knowledge is a bit uneven :wink: This is an up to date Tumbleweed.

Thanks in advance,
Bartosz

It may be hardware problem (hard disk, controller) after which filesystem is remounted read-only. I am not sure what btrfs does in out of space conditions. Check dmesg output if there are any errors right after you see this problem.

Hello again and thank you arvidjaar for a quick reply :slight_smile:

I’ll describe everything in order, step by step.

Closing the OS, multiple errors occur

systemd-journal[496]: Cannot truncate file to its own size - filesystem is read-only
  1. Cold boot;
  2. GRUB;
  3. Start boot-loader from a read only snapshot;
  4. openSUSE Tumbleweed (4.30-1,2015-11-10T20:07,post,zypp(y2base));
  5. Bootable snapshot #71;
  6. If OK run ‘snapper rollback’ and reboot;
  7. I cannot do anything, so I use [ESC] to go back one time;
  8. I select openSUSE Tumbleweed
  9. KDE logon screen, login successful;
  10. KWallet, credentials correct;
  11. I see my mtab with cat, as described before, the volumes are rw;
  12. I try running yast and boom, I get an error;
  13. My dmesg follows.

  202.459446] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
 shortened for brevity ]
  460.222358] SFW2-INext-DROP-DEFLT IN=wlp2s0 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=94 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=54 
  510.403249] IPv4: martian source 255.255.255.255 from 192.168.51.101, on dev wlp2s0
  510.403257] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  510.710388] IPv4: martian source 192.168.51.255 from 192.168.51.101, on dev wlp2s0
  510.710393] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  511.427206] IPv4: martian source 192.168.51.255 from 192.168.51.101, on dev wlp2s0
  511.427217] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  512.246401] IPv4: martian source 192.168.51.255 from 192.168.51.101, on dev wlp2s0
  512.246407] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  527.573991] SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=108 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=68 
 shortened for brevity ]
  757.621994] SFW2-INext-DROP-DEFLT IN=wlp2s0 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=94 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=54 
  761.407357] IPv4: martian source 255.255.255.255 from 192.168.51.101, on dev wlp2s0
  761.407367] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  761.713738] IPv4: martian source 192.168.51.255 from 192.168.51.101, on dev wlp2s0
 shortened for brevity ]
  802.572389] IPv4: martian source 192.168.51.255 from 192.168.51.101, on dev wlp2s0
  802.572397] ll header: 00000000: ff ff ff ff ff ff e0 2a 82 19 a9 29 08 00        .......*...)..
  825.415649] SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=108 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=68 
 shortened for brevity ]
  856.541961] SFW2-INext-DROP-DEFLT IN=wlp2s0 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=94 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=54 
  887.232556] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.232802] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.234056] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.234314] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.234567] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.234835] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.235097] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.235356] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.235611] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.235865] BTRFS: sda1 checksum verify failed on 7313555456 wanted C32E6C57 found E4C76A7A level 0
  887.244017] BTRFS: error (device sda1) in btrfs_drop_snapshot:8695: errno=-5 IO failure
  887.244024] BTRFS info (device sda1): forced readonly
  887.246142] systemd-journald[519]: Failed to truncate file to its own size: Read-only file system
 shortened for brevity ]
  915.074926] systemd-journald[519]: Failed to truncate file to its own size: Read-only file system
  925.695685] SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=108 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=68 
 shortened for brevity ]
 1024.482823] SFW2-INext-DROP-DEFLT IN=em1 OUT= MAC=33:33:00:00:00:fb:60:03:08:a1:38:e8:86:dd SRC=fe80:0000:0000:0000:6203:08ff:fea1:38e8 DST=ff02:0000:0000:0000:0000:0000:0000:00fb LEN=144 TC=0 HOPLIMIT=255 FLOWLBL=429615 PROTO=UDP SPT=5353 DPT=5353 LEN=104 
 1024.482947] systemd-journald[519]: Failed to truncate file to its own size: Read-only file system

I highlighted to entries in red. The “forced readonly” line was also red in the dmesg output. So, looks like the FS gets all defensive because some checksums don’t add up. As you probably assumed already, sda1 is my main volume. This does suggest FS corruption and/or failing drive. This is a Kingston SSD, I have it for less than a year but I have to make a confession. In my almost 10 years handling PCs and laptops, from Windows 98 to various Linux distros I’ve never had a “drive accident”. Until recently: I dropped the poor guy :frowning:

However: everything was running smooth for about a week before this mess started, so thought it was all good. Maybe this whole power thing lead to FS corruption? It’s more than likely I had yast opened with sudo when the situation occurred. I’m not an FS expert, so I’m only theorizing here.

Thanks,
Bartosz

Well, at this point you may want to post to btrfs mailing list (linux-btrfs@vger.kernel.org) for guidance how to recover filesystem.

Hello!

Thank you for all the help, but I had to do a fresh install. And that failed as well, so looks like a hardware error. I had to fall back to an HDD for the time being. Well, lesson learned. When tinkering - be focused.