Too many open files

English Install/Boot/Login
#1

After upgrading to OpenSUSE 12.1 from 11.4, I keep getting the “Too many open files” error after the machine have been running for a few days.

I have seen this on almost all my machines running opensuse 12.1, I can confirm it on 3 servers, and atleast 2 workstations. I had no problems with 11.4.

dbus-daemon have around 140 processes running. I can solve the problem without a reboot by killing all dbus-daemon processes. Most processes are owned by root.

When investigating /proc for the total number of files that are open I found that dbus-daemon is owning 1158 of total 1981 open fd’s.

I tried to increase the maximum number of open files on the machines, but that didn’t help.

Here is /etc/security/limits.conf

  • hard nofile 65535
  • soft nofile 65535

echoes:/home/jonas # cat /proc/sys/fs/file-max
204876


root      1133  0.0  0.0   2876   380 ?        Ss   Jan19   0:00 /bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
#2

If you really did an upgrade and did not do a clean installation of openSUSE 12.1, then I suggest that this has created the problem that you are having. For instance, dbus-1 is at 1.4 in openSUSE 11.4 and 1.5 in openSUSE 12.1 and if not properly updated, it could create such a problem, though it is only a guess. I suggest that you do a clean install, do a custom partition, elect to mount all existing partitions as created before, except you will format the root / partition and you WILL NOT format the /home partition and only mount it. Thus keeping all personnel settings and requiring only that you reinstall your old applications again. Doing that upgrade to openSUSE 12.1 is allowed and most often does conclude with a new system that can boot and startup. However, you can end up with mixed versions and orphaned applications when doing an upgrade that can cause a lot of odd issues. Consider that clean install without formatting your /home partition and lets see if that helps.

Thank You,

#3

I noticed that every time I login via SSH or ‘su root’ on the shell dbus-daemon forks a new process, but the processes are never free’d.

#4

The problem is not related to upgrading. One of the machines is a new installation and have the exact same problem, just had to do ‘killall -u root dbus-daemon’ 1 hour ago. On this machine I recently had a disk crash so I had reinstalled 12.1 on a new HDD. Runs out of file descriptors, increasing number of dbus-daemon processes.

Another problem in 12.1 is that libsystemd-daemon.so.0 and libsystemd-daemon.so.0.0.0 is in /usr/lib64 and not /lib64 so if /usr is on a seperate partition than / it fails to find it when rebooting after /usr has been umounted.

#5

So I have four installs with openSUSE 12.1 (though one is in a VM) and not seen the behavior. Since we are creatures of habit, its possible your partitioning or other such thing has created this issue. If we can, lets look at your clean install and gives us more information. What desktop did you install? Have you done a complete update of all software since the install? Can you do the following in terminal and post the results?

su - 
password: 
cat /etc/fstab 
cat /boot/grub/device.map 
cat /boot/grub/menu.lst 
fdisk -l 
free

Post the results of these in a forum message for us to view. I have a couple of other bash script that can provide even more information:

H.I. Hardware Information - A Bash script to install and run inxi with default options! - Blogs - openSUSE Forums

S.K.I.M. - SuSE Kernel Installed Modules - A lsmod replacement- Creates Alphabetized Module Listing - Blogs - openSUSE Forums

S.L.A.V.E. - SuSE Logfile Automated Viewer Engine - Version 2.55 - Blogs - openSUSE Forums

openSUSE 12.1 uses systemd for startup. Some have switched back to using System V at startup to see if it helps. In the grub OS selection menu, press F5 and select System V and see if that changes the problem. For anyone using systemd, here are a couple of other links you should see:

systemd and using the after.local script in openSUSE 12.1 - Blogs - openSUSE Forums

openSUSE 12.1 and clearing /tmp at boot doesn’t work is systemd - Blogs - openSUSE Forums

As is normal, I throw out lots of stuff to see if any sticks.

Thank You,

#6

Do you run KDE or gnome? If you don’t have the problem, can you check this: If you ssh to the machine and type exit, does that leave an extra dbus-daemon process alive after exit?

#7 
linuxtv:/home/jonas # cat /etc/fstab
/dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN-part1 /boot                ext4       acl,user_xattr,noatime        1 2
/dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN-part5 /                    ext4       acl,user_xattr,noatime        1 1
/dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN-part6 /usr                 ext4       acl,user_xattr,noatime        1 2
/dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN-part7 /opt                 ext4       acl,user_xattr,noatime        1 2
/dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN-part8 /usr/local           ext4       acl,user_xattr,noatime        1 2

/dev/disk/by-id/ata-SAMSUNG_HD103SJ_S246JDWZ209023-part1 /var                 ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-SAMSUNG_HD103SJ_S246JDWZ209023-part2 swap                 swap       defaults              0 0
/dev/disk/by-id/ata-SAMSUNG_HD103SJ_S246JDWZ209023-part3 /home                ext4       acl,user_xattr        1 2
/dev/disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1KQC05172-part1 /mnt/data            ext4       defaults              1 2
/dev/disk/by-id/ata-SAMSUNG_HD103SJ_S246JDWZ209023-part4 /mnt/store           xfs        defaults              1 2

/dev/disk/by-id/usb-SAMSUNG_HD105SI_259BB4047828-0:0-part1 /mnt/flacbackup1     auto     defaults,user,noauto  1 2
/dev/disk/by-id/usb-SAMSUNG_HD204UI_FAFFFFF0FDF2FFF1BF8421-0:0-part1 /mnt/flacbackup2   auto     defaults,user,noauto  1 2

tmpfs                /tmp                 tmpfs      size=900M,noatime     0 0

proc                 /proc                proc       defaults              0 0
sysfs                /sys                 sysfs      noauto                0 0
debugfs              /sys/kernel/debug    debugfs    noauto                0 0
usbfs                /proc/bus/usb        usbfs      noauto                0 0
devpts               /dev/pts             devpts     mode=0620,gid=5       0 0


linuxtv:/home/jonas # cat /boot/grub/device.map 
(hd1)   /dev/disk/by-id/ata-SAMSUNG_HD103SJ_S246JDWZ209023
(hd2)   /dev/disk/by-id/ata-SAMSUNG_HD753LJ_S13UJ1KQC05172
(hd0)   /dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR135105BV080BGN




linuxtv:/home/jonas # fdisk -l 

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1  1953525167   976762583+  ee  GPT

Disk /dev/sdc: 750.2 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders, total 1465149168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00069ba4

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1465144064   732572001   83  Linux

Disk /dev/sda: 80.0 GB, 80026361856 bytes
255 heads, 63 sectors/track, 9729 cylinders, total 156301488 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00023c05

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1230847      614400   83  Linux
/dev/sda2         1230848   156301311    77535232    5  Extended
/dev/sda5         1232896    11718655     5242880   83  Linux
/dev/sda6        11720704    74635263    31457280   83  Linux
/dev/sda7        74637312    78831615     2097152   83  Linux
/dev/sda8        78833664    83027967     2097152   83  Linux

Disk /dev/sdd: 2000.4 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders, total 3907029168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xee109927

   Device Boot      Start         End      Blocks   Id  System
/dev/sdd1            2048  3907028991  1953513472   83  Linux




linuxtv:/home/jonas # free
             total       used       free     shared    buffers     cached
Mem:       2055412    1967856      87556          0      24620     966796
-/+ buffers/cache:     976440    1078972
Swap:      4194300     458548    3735752
#8

I am not seeing the problem.

On my main desktop, I see:


102       1261     1  0 Jan22 ?        00:00:17 /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation
rickert   3318     1  0 Jan22 ?        00:00:00 dbus-launch --sh-syntax --exit-with-session
rickert   3321     1  0 Jan22 ?        00:00:01 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

So that’s two dbus daemons running as me, and one running as user 102. There are none running as root. I last booted this on Sunday, but have only logged into the desktop once, though I have done an ssh in on several occasions.

On another system, I last rebooted on Friday. I have logged into KDE twice, and logged out again. I have also accessed via ssh on several occasions. There, I see:


102        757     1  0 Jan20 ?        00:00:03 /bin/dbus-daemon --system --address=systemd: --nofork --systemd-activation

That’s just the one dbus process, running as user 102. I was logged in via ssh when listing that.

From what I am seeing, it looks as if dbus processes started by my KDE login are automatically terminated when I logout, and it looks as if an ssh into the system does not start a dbus process.

You are probably doing more complex things when you are logged in, and presumably they are related to the problem. So maybe you need to experiment with what causes these dbus processes to start. The process start time might help you track them down.

#9

On 2012-01-24 01:46, jdmcdaniel3 wrote:
>
> If you really did an upgrade and did not do a clean installation of
> openSUSE 12.1, then I suggest that this has created the problem that you
> are having.

You should not think that upgrading is always the cause of all problems. In
this case, it is not, as the OP has the same problem on freshly installed
machines.

> For instance, dbus-1 is at 1.4 in openSUSE 11.4 and 1.5 in
> openSUSE 12.1 and if not properly updated, it could create such a
> problem, though it is only a guess. I suggest that you do a clean
> install,

Before doing that, you should prove that your hypothesis is correct, by
verifying the version of the suspect package.

> Thus keeping all
> personnel settings and requiring only that you reinstall your old
> applications again.

And destroying all system configurations and settings.

> However, you can end up with mixed versions and orphaned applications
> when doing an upgrade that can cause a lot of odd issues.

This is not true.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

#10

So using ssh to log in and out does not create such a situation on my setup though I don’t use it much at home, but you would need to be logging in and out a lot to make any difference I would think. But, I have two quick comments based on what I see. First, you are running out of memory on this setup. Once you start using SWAP, you have topped out on main memory. It may be time to add more to this system. Perhaps using ssh under that circumstance could create an issue? Second, you have an 80 GB ssd broken up into five different partitions. Why do you need such a setup? SSD are already small, you have all of the partitions on the same disk and using multiple partitions spread your free space all over the place and creating partitions unto themselves wastes space as well. I have the same size SSD on one PC and I dedicated it all to the root / as a single partition including all you have broken up and even so, I must watch the free space like a hawk. I do lots of kernel compiles that can really use up disk space. So, are you sure you have free space on all partitions of the SSD? And thanks for the detail as I will keep looking and consider info from the other apps as well. By the way, what I have found in the past is that since we have a way doing things, we repeat the same setup over and over. Splitting up an SSD into lots of partitions perhaps, or keeping memory to a minimum. I am not sure, but its something you do on every system and trust me, I have found my own repeatable issues before as well, only I was was the person causing the issues to repeat.

Thank You,

#11

Since the intent of posting most messages here is to provide help to the user, why not do so and stop not wasting his time in a disagreement of doing an upgrade verse an install post. Upgrades have came up several times as a issue and does not straighten out all application dependencies that can come up. So, in this case, what is your suggestion to help the OP with his problem or do think suggesting what I say is wrong is helping in some way? What is the purpose of such a post?

Thank You,

#12

On 2012-01-24 02:16, jonaski wrote:
>
> The problem is not related to upgrading. One of the machines is a new
> installation and have the exact same problem, just had to do ‘killall -u
> root dbus-daemon’ 1 hour ago. On this machine I recently had a disk
> crash so I had reinstalled 12.1 on a new HDD. Runs out of file
> descriptors, increasing number of dbus-daemon processes.

It smells of bugs, but one idea is if you are installing on your machines
some non typical software. Or hardware.

> Another problem in 12.1 is that libsystemd-daemon.so.0 and
> libsystemd-daemon.so.0.0.0 is in /usr/lib64 and not /lib64 so if /usr is
> on a seperate partition than / it fails to find it when rebooting after
> /usr has been umounted.

I was told that issues like this had been taken care of. :-?

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

#13

On 01/23/2012 07:46 PM, jonaski wrote:
>
> Do you run KDE or gnome? If you don’t have the problem, can you check
> this: If you ssh to the machine and type exit, does that leave an extra
> dbus-daemon process alive after exit?

I use KDE. After I ssh to the box and exit, the same dbus-related processes are
running that were running before the ssh.

#14

On 2012-01-24 03:16, jdmcdaniel3 wrote:

> So, in this case, what is your
> suggestion to help the OP with his problem or do think suggesting what I
> say is wrong is helping in some way? What is the purpose of such a
> post?

To avoid the OP going the road of reinstalling fresh when it is useless and
a waste of time.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

#15

On 2012-01-24 03:08, Carlos E. R. wrote:
> On 2012-01-24 02:16, jonaski wrote:

Another thing you can try is login with a new user. See if the problem is
related to your home profile.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

#16

Is it maybe this bug: https://bugs.kde.org/show_bug.cgi?id=261180

It seems that is fixed so check if there is an update already.

For the server I think one possible workaround would be to run it on level 3 instead of 5. Check if this helps.

Cheers.

#17

Finally figured this one out. It was the following line in .bashrc: export $(dbus-launch)
Had the same thing an all my machines. It worked fine in 11.4.
Removed it and now dbus-daemon shuts down on logout. Will just use ‘su -’ instead of ‘su’ to launch X applications from the shell.

#18

If you ‘su’ on the shell, enter “export $(dbus-launch)” and exit, you will see that dbus-daemon does not quit.

#19

On 01/24/2012 05:06 AM, jonaski wrote:
>
> Finally figured this one out. It was the following line in .bashrc:
> export $(dbus-launch)
> Had the same thing an all my machines. It worked fine in 11.4.
> Removed it and now dbus-daemon shuts down on logout. Will just use ‘su
> -’ instead of ‘su’ to launch X applications from the shell.

no! do not use either “su” or “su -” to “launch X applications from the
shell” instead use one of these, depending on your desktop environment:

kdesu [app executable]
gnomesu [app executable]
/usr/bin/xdg-su -c [app executable]

because “export $(dbus-launch)” was a BAD solution to a non-problem that
is avoided if (in openSUSE) one follows the correct way here and does
not launch root powered X apps from a user terminal incorrectly…

hmmmm, another way of saying that is: i have no problems whatsoever
launching root powered apps from a user terminal in 11.4 and i have
(just checked) no ‘export $(dbus-launch)’ in any of

/home/[me]/.bashrc
/etc/bash.bashrc
/etc/bash.bashrc/local

and, i have no /root/.bashrc (because there should not be one)

using bad practice and then solving non-problems is not a good thing.

yes, i know some other distros work fine with these from a user terminal:

sudo [app executable]
su [app executable]
su - [app executable]

but openSUSE varies from (for example) Windows, Mac and other Linux
distros–vive la différence!

–
DD
openSUSE®, the “German Engineered Automobiles” of operating systems!

#20

This is an example of a problem on different computers, loaded in different ways, yet with the same problem. As I did say, often such issues exist due to us being creatures of habit, performing the same task or function over and over in each different computer, thus extending the issue across all within our reach. It is good to hear you found the problem and hopefully you will formulate a better solution than the one you were using.

Thank You,