No space left on device, uncommon cause?

orsos · November 16, 2017, 4:23pm

Hi OpenSUSE community,
Although I am noob, it has been two years that I enjoy LEAP without troubleshouting, usually I always find the answer on forum.
Here I am stuck on a more serious problem (than usual): my GUI does not start from today: “no space left on device”.
I check with

df -Th

and indeed my root partition of 40GB (btrfs filesystem) is full.

https://image.ibb.co/frrtDR/20171116_155211_Small.jpg

Here are my snapshots:

snapper -c root list

https://image.ibb.co/ktcN7m/20171116_155241_Small.jpg

Before that, I had a problem while installing the package “kernel-devel”: I received an error which I ignored (unwise!) resulting on a problem with my packages, no matter what I tried with YaST or zypper I was receiving an error message “failed to move cache to final destination” (my guess is a mismatch between the the version of “kernel-devel” and my kernel version, but could not inquire further…).
Once I restarted my laptop and lunch YaST, it reinstalled all the packages. Ever since, when I restart, I run into the error.

At this point I prefer to ask for help before to deteriorate the situation even more.
I see two options:

Erase some files in the root partition, but which? I should need snapshots to recover the system before I tried to install “kernel-devel”, right?
Save personal file on usb stick and full installation of openSUSE.
Thanks!

jdd · November 16, 2017, 4:50pm

Le 16/11/2017 à 16:26, orsos a écrit :
>
> Hi OpenSUSE community,
> Although I am noob, it has been two years that I enjoy LEAP without

two years? 42.1??

anyway your problem is very common specially on older installs (some
progress have been made on newer defaults)

the bible is here, first link on google for “opensuse btrfs”

https://en.opensuse.org/SDB:BTRFS

The simpler solution is removing older snapshots and editing snappers
config files to have less snapshots

by the way df do not gives relevant infos on btrfs

jdd (that was also hit some month ago

dcurtisfra · November 16, 2017, 5:30pm

Unfortunately, the openSUSE Btrfs SDB doesn’t mention the cron jobs which run if the system is “always on” but, the advice in the sections “Disk Space Full Because of Snapper” and “How to repair a broken/unmountable btrfs filesystem” is valid and, may be the only way you can initially repair the system sufficiently to allow the default Btrfs maintenance cron jobs to be executed.

Once you have the system in a state which has some free space in the Btrfs system partition, manually execute the Btrfs maintenance scripts located in /etc/cron.monthly/ and /etc/cron.weekly/.

After the system has booted as far as it can, you’ll need to:

activate the first VT (tty1) (<Ctrl-Alt-F1>);
login as the user “root”;
shut the system down to “single-user mode” with: systemctl rescue
type in the password of the user “root”;
‘cd’ to /etc/cron.monthly/
and execute the ‘btrfs-scrub’ script in that directory; 1. ‘cd’ to /etc/cron.weekly/
and execute the ‘btrfs-balance’ and ‘btrfs-trim’ scripts in that directory; 1. wait for each script to execute – each script will take more than a few minutes to complete;
reboot: systemctl reboot

Once the system has been repaired and is running normally, check the following (with a “normal” user’s CLI):systemctl list-unit-files | grep -i 'btrfs’

– if it doesn’t indicate that, the systemd service is enabled, enable it (with the user “root” or, by means of ‘sudo’):sudo systemctl enable btrfsmaintenance-refresh.service

orsos · November 16, 2017, 5:35pm

Thanks for the reply jdd.
Yes 42.1 at the beginning, now running on LEAP 42.3.
Yes I thought about deleting snapshots, but since I need to restore from previous was not sure.
Say I want to restore to this morning, so # 1026.
Can I use the following code?

snapper delete 901-1025

and then

snapper undochange 1026..1047

jdd · November 16, 2017, 6:02pm

Le 16/11/2017 à 17:36, orsos a écrit :
>
> Thanks for the reply jjd.
> Yes 42.1 at the beginning, now running on LEAP 42.3.
> Yes I thought about deleting snapshots, but since I need to restore from
> previous was not sure.

are you sure you need it?

first, list the snapshots with snapper, then remove the older ones, one
at a time. When sufficient space is available, try rebooting. If the
default do not boot, try rebooting the previous one in the grub menu

do revert only if the boot is effective

jdd

jdd · November 16, 2017, 6:05pm

Le 16/11/2017 à 17:36, dcurtisfra a écrit :
>
> Unfortunately, the openSUSE Btrfs SDB doesn’t mention the cron jobs
> which run if the system is “-always on-”

the howto you give seems very good, can you add it to the wiki?

just a warning, balance can be extremely long. I tried this on a (very
old) test computer, had to leave 2 hours later it was not finished.
Stopping the computer killed the install. I never could recover it. Was
TW and nothing important on this test machine.

thanks
jdd

>
> After the system has booted as far as it can, you’ll need to:
>
>
>
> - activate the first VT (tty1) (<Ctrl-Alt-F1>);
> - login as the user “root”;
> - shut the system down to “single-user mode” with: systemctl rescue
> - type in the password of the user “root”;
> - ‘cd’ to /etc/cron.monthly/ and execute the ‘btrfs-scrub’ script
> in that directory;
> - ‘cd’ to /etc/cron.weekly/ and execute the ‘btrfs-balance’ and
> ‘btrfs-trim’ scripts in that directory;
> - wait for each script to execute – each script will take more than a
> few minutes to complete;
> - reboot: systemctl reboot
>
>
> Once the system has been repaired and is running normally, check the
> following (with a “normal” user’s CLI):
> systemctl
> list-unit-files | grep -i ‘btrfs’
>
> – if it doesn’t indicate that, the systemd service is enabled, enable
> it (with the user “root” or, by means of ‘sudo’):
> sudo systemctl
> enable btrfsmaintenance-refresh.service
>
>

dcurtisfra · November 16, 2017, 6:31pm

I’ll check with the maintainer – RBrown: SUSE – openSUSE manager – responsible person . . .It’s OK – I’ve met and talked to Richard a couple of times at the openSUSE conferences and, he’s located only about 20 km from where I live.

knurpht · November 16, 2017, 6:42pm

Use snapper to remove snapshots

orsos · November 16, 2017, 8:46pm

Thanks both of you for your replies.
I will try your proposition.
Just something that bother me, all along you’re assuming the excess number of snapshots is causing my root partition to be full, but my system was fine until I try to install the “kernel-devel” this morning. I am not sure to understand the link here, or at least we can exclude the age of my installation as a cause.

jdd · November 16, 2017, 9:16pm

Le 16/11/2017 à 20:56, orsos a écrit :

> I am not sure to understand the link here, or at least we can exclude
> the age of my installation as a cause.
>
>
list the snapshots, you will know

jdd

orsos · November 16, 2017, 9:38pm

So I deleted some soft from /opt/ and some old snapshots and could get to my GUI system.
Now I am facing another problem with the package manager.
Happens since I try to install “kernel-devel” package this morning.

For example if I enter

rpm -qa --last

my console freezes…

I did not run dcurtisfra Btrfs maintenance proposition yet as I am not sure to understand whether it will fix this issue.
What option seems the most appropriate?
1- Anything specific considering “kernel-devel”?
2- system rollback?
3- Btrfs maintenance?

Thanks again

orsos · November 16, 2017, 9:41pm

Here we are:

orsos@linux-pkwf:~> sudo snapper -c root list
[sudo] password for root:  
Type   | #    | Pre # | Date                             | User | Cleanup | Description                | Userdata      
-------+------+-------+----------------------------------+------+---------+----------------------------+--------------
single | 0    |       |                                  | root |         | current                    |               
single | 1    |       | Mon 11 Apr 2016 02:07:21 AM CEST | root |         | first root filesystem      |               
pre    | 1022 |       | Thu 09 Nov 2017 10:24:39 AM CET  | root | number  | zypp(packagekitd)          | important=no  
post   | 1023 | 1022  | Thu 09 Nov 2017 10:26:20 AM CET  | root | number  |                            | important=no  
pre    | 1024 |       | Tue 14 Nov 2017 03:15:10 PM CET  | root | number  | zypp(packagekitd)          | important=no  
post   | 1025 | 1024  | Tue 14 Nov 2017 03:16:25 PM CET  | root | number  |                            | important=no  
pre    | 1026 |       | Thu 16 Nov 2017 09:23:14 AM CET  | root | number  | zypp(packagekitd)          | important=no  
post   | 1027 | 1026  | Thu 16 Nov 2017 09:23:34 AM CET  | root | number  |                            | important=no  
pre    | 1028 |       | Thu 16 Nov 2017 10:33:25 AM CET  | root | number  | yast sw_single             |               
pre    | 1029 |       | Thu 16 Nov 2017 10:34:49 AM CET  | root | number  | zypp(ruby)                 | important=yes
post   | 1031 | 1028  | Thu 16 Nov 2017 10:35:43 AM CET  | root | number  |                            |               
pre    | 1032 |       | Thu 16 Nov 2017 10:36:33 AM CET  | root | number  | yast sw_single             |               
pre    | 1039 |       | Thu 16 Nov 2017 01:35:31 PM CET  | root | number  | yast sw_single             |               
post   | 1040 | 1039  | Thu 16 Nov 2017 01:38:13 PM CET  | root | number  |                            |               
pre    | 1041 |       | Thu 16 Nov 2017 01:38:54 PM CET  | root | number  | yast sw_single             |               
pre    | 1042 |       | Thu 16 Nov 2017 01:40:04 PM CET  | root | number  | yast OneClickInstallWorker |               
post   | 1043 | 1042  | Thu 16 Nov 2017 01:41:01 PM CET  | root | number  |                            |               
post   | 1044 | 1041  | Thu 16 Nov 2017 01:41:20 PM CET  | root | number  |                            |               
pre    | 1045 |       | Thu 16 Nov 2017 01:41:41 PM CET  | root | number  | yast OneClickInstallWorker |               
pre    | 1046 |       | Thu 16 Nov 2017 01:41:51 PM CET  | root | number  | zypp(ruby)                 | important=no  
post   | 1047 | 1045  | Thu 16 Nov 2017 01:42:09 PM CET  | root | number  |                            |               
pre    | 1048 |       | Thu 16 Nov 2017 08:45:43 PM CET  | root | number  | yast snapper               |               
post   | 1049 | 1048  | Thu 16 Nov 2017 08:59:38 PM CET  | root | number  |                            |               
pre    | 1050 |       | Thu 16 Nov 2017 08:59:46 PM CET  | root | number  | yast sw_single             |               
post   | 1051 | 1050  | Thu 16 Nov 2017 09:01:31 PM CET  | root | number  |                            |               
pre    | 1052 |       | Thu 16 Nov 2017 09:13:48 PM CET  | root | number  | yast snapper               |               
post   | 1053 | 1052  | Thu 16 Nov 2017 09:16:00 PM CET  | root | number  |                            |

I just realized that the system is creating a lot of snapshots… So I am almost certain it is due to my problem of package installation.

orsos · November 16, 2017, 11:16pm

Since the problem seems related to the package manager, I opened a new threat https://forums.opensuse.org/showthread.php/528145-Package-installation-failed-cannot-open-Packages-index-anymore!?p=2845137#post2845137

malcolmlewis · November 16, 2017, 11:24pm

Hi
I think your chasing your tail, if you don’t clean up disk space, then nothing will install… it can’t complete something like installing kernel-development because lack of space…

You need to run the snapper and btrfs maintenance programs to remove snapshots and recover allocated space…

Step 1: Change the configuration of /etc/snapper/configs/root number cleanup to;


# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="2-3"
NUMBER_LIMIT_IMPORTANT="3-4"

Run the daily cronjob manually;


/etc/cron.daily/suse.de-snapper
snapper list

Now run the btrfs balance manually a couple of time (and let it finish, it may take some time);


/etc/cron.weekly/btrfs-balance
/etc/cron.weekly/btrfs-balance

Then post the output from;


btrfs fi usage /

jdd · November 17, 2017, 12:03am

Le 16/11/2017 à 21:46, orsos a écrit :

> I did not run dcurtisfra Btrfs maintenance proposition yet as I am not
> sure to understand whether it will fix this issue.

may be not, but it shouldn’t do any harm

jdd

orsos · November 17, 2017, 12:59pm

Thanks for your answers!
I changed the config of in /etc/snapper/configs/root as suggested.
For the daily cronjob, I obtain:

orsos@linux-pkwf:/etc/cron.daily> sudo ./suse.de-snapper  
quota not working (qgroup not set)

Should I set a qgroup?

Here is the snapper list

orsos@linux-pkwf:/etc/cron.daily> sudo snapper list
Type   | #    | Pre # | Date                             | User | Cleanup | Description           | Userdata      
-------+------+-------+----------------------------------+------+---------+-----------------------+--------------
single | 0    |       |                                  | root |         | current               |               
single | 1    |       | Mon 11 Apr 2016 02:07:21 AM CEST | root |         | first root filesystem |               
pre    | 1029 |       | Thu 16 Nov 2017 10:34:49 AM CET  | root | number  | zypp(ruby)            | important=yes
pre    | 1048 |       | Thu 16 Nov 2017 08:45:43 PM CET  | root | number  | yast snapper          |               
post   | 1049 | 1048  | Thu 16 Nov 2017 08:59:38 PM CET  | root | number  |                       |

I obtain an error for btrfs-maintenance (anything to do with the fact I have a SSD?):

orsos@linux-pkwf:/etc/cron.weekly> ./btrfs-balance  
Before balance of /
Data, single: total=38.47GiB, used=16.52GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.50GiB, used=482.56MiB
GlobalReserve, single: total=176.00MiB, used=0.00B
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        43G   19G   24G  44% /
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=10
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=20
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=30
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=40
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=50
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=1
  SYSTEM (flags 0x2): balancing, usage=1
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=5
  SYSTEM (flags 0x2): balancing, usage=5
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=10
  SYSTEM (flags 0x2): balancing, usage=10
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=20
  SYSTEM (flags 0x2): balancing, usage=20
ERROR: error during balancing '/': Operation not permitted
There may be more info in syslog - try dmesg | tail
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=30
  SYSTEM (flags 0x2): balancing, usage=30
After balance of /
Data, single: total=38.47GiB, used=16.52GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.50GiB, used=482.56MiB
GlobalReserve, single: total=176.00MiB, used=0.00B
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        43G   19G   24G  44% /

Here is the usage:


orsos@linux-pkwf:/etc/cron.weekly> sudo btrfs fi usage /
Overall:
    Device size:                  40.00GiB
    Device allocated:             40.00GiB
    Device unallocated:            1.00MiB
    Device missing:                  0.00B
    Used:                         16.99GiB
    Free (estimated):             21.95GiB      (min: 21.95GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              176.00MiB      (used: 0.00B)

Data,single: Size:38.47GiB, Used:16.52GiB
   /dev/sda2      38.47GiB

Metadata,single: Size:1.50GiB, Used:482.56MiB
   /dev/sda2       1.50GiB

System,single: Size:32.00MiB, Used:16.00KiB
   /dev/sda2      32.00MiB

Unallocated:
   /dev/sda2       1.00MiB

malcolmlewis · November 17, 2017, 1:29pm

Hi
So all looks good;


Used: 16.99GiB
Free (estimated): 21.95GiB (min: 21.95GiB)

So now to clean up the allocated/unallocated space have a read ;
https://btrfs.wiki.kernel.org/index.php/FAQ

Section: Help! Btrfs claims I’m out of space, but it looks like I should have lots left! and part “if your device is large (>16GiB)”

ndc33 · November 17, 2017, 2:03pm

> I obtain an error for btrfs-maintenance
just in case - these need to be executed with elevated privilege

orsos · November 17, 2017, 4:08pm

Yes indeed… Here we are


orsos@linux-pkwf:/etc/cron.weekly> sudo ./btrfs-balance  
[sudo] password for root:  
Before balance of /
Data, single: total=38.47GiB, used=16.52GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=1.50GiB, used=482.61MiB
GlobalReserve, single: total=176.00MiB, used=0.00B
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        43G   19G   24G  44% /
Done, had to relocate 0 out of 54 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=1
Done, had to relocate 0 out of 54 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=5
Done, had to relocate 5 out of 54 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=10
Done, had to relocate 1 out of 50 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=20
Done, had to relocate 4 out of 50 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=30
Done, had to relocate 7 out of 47 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=40
Done, had to relocate 6 out of 41 chunks
Dumping filters: flags 0x1, state 0x0, force is off
  DATA (flags 0x2): balancing, usage=50
Done, had to relocate 3 out of 36 chunks
Done, had to relocate 0 out of 34 chunks
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=1
  SYSTEM (flags 0x2): balancing, usage=1
Done, had to relocate 1 out of 34 chunks
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=5                                                                                                                   
  SYSTEM (flags 0x2): balancing, usage=5
Done, had to relocate 1 out of 34 chunks
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=10
  SYSTEM (flags 0x2): balancing, usage=10
Done, had to relocate 1 out of 34 chunks
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=20
  SYSTEM (flags 0x2): balancing, usage=20
Done, had to relocate 1 out of 34 chunks
Dumping filters: flags 0x6, state 0x0, force is off
  METADATA (flags 0x2): balancing, usage=30
  SYSTEM (flags 0x2): balancing, usage=30
Done, had to relocate 4 out of 34 chunks
After balance of /
Data, single: total=21.50GiB, used=16.52GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=768.00MiB, used=478.41MiB
GlobalReserve, single: total=160.00MiB, used=0.00B
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        43G   19G   25G  44% /

orsos · November 18, 2017, 9:52am

Thank you all. I think the problem of the thread is fixed. Now I have a problem with my RPM database, but I opened a new thread for that.