not able to connect via openVPN

r-cane · December 13, 2011, 10:21am

Hello,

after running openVPN for a while, users cannot connect anymore. The only work-around so far is to restart openVPN. Unfortunately, if many users are connecting the file handles are exhausted in less than a day. Therefore I would like to have a more permanent solution for this.

/var/log/messages shows the following errors:


Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_env.so): /lib/security/pam_env.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_env.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_gnome_keyring.so): /lib/security/pam_gnome_keyring.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_gnome_keyring.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_unix2.so): /lib/security/pam_unix2.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_unix2.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_winbind.so): /lib/security/pam_winbind.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_winbind.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_localuser.so): /lib/security/pam_localuser.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_localuser.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_pwcheck.so): /lib/security/pam_pwcheck.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_pwcheck.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_mkhomedir.so): /lib/security/pam_mkhomedir.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_mkhomedir.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_limits.so): /lib/security/pam_limits.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_limits.so
Dec 12 18:34:24 a-gate openvpn[26072]: PAM unable to dlopen(/lib/security/pam_apparmor.so): /lib/security/pam_apparmor.so: cannot open shared object file: Too many open files
Dec 12 18:34:24 a-gate openvpn[26072]: PAM adding faulty module: /lib/security/pam_apparmor.so

I had a look at the files open by openVPN (lsof -p 26072) and found that one or more new sockets are created every time a user logs on. The sockets are never released:


COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
openvpn 6303 root    0u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    1u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    2u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    3u  unix 0xf3173800      0t0 1002734187 socket
openvpn 6303 root    5u  unix 0xf315b300      0t0 1002733627 socket
openvpn 6303 root    6u  unix 0xf7345800      0t0 1002734192 socket
openvpn 6303 root    7u  unix 0xf359b580      0t0 1002734374 socket
openvpn 6303 root    8u  unix 0xf30f2d00      0t0 1002737606 socket
openvpn 6303 root    9u  unix 0xf30f2300      0t0 1002737716 socket
openvpn 6303 root   10u  unix 0xf30f2580      0t0 1002737782 socket
openvpn 6303 root   11u  unix 0xf34bd300      0t0 1002737845 socket
openvpn 6303 root   12u  unix 0xf6fd0580      0t0 1002737955 socket
openvpn 6303 root  cwd    DIR        8,2      648          2 /
openvpn 6303 root  mem    REG        8,2     9688     110153 /lib/security/pam_ck_connector.so
openvpn 6303 root  mem    REG        8,2    13732      21753 /lib/libgpg-error.so.0.7.0
openvpn 6303 root  mem    REG        8,2    13844     110173 /usr/lib/libck-connector.so.0.0.0
openvpn 6303 root  mem    REG        8,2    13864      73704 /usr/lib/libplds4.so
openvpn 6303 root  mem    REG        8,2    17392      30309 /lib/libdl-2.11.3.so
openvpn 6303 root  mem    REG        8,2    17680     116462 /etc/openvpn/openvpn-auth-pam.so
openvpn 6303 root  mem    REG        8,2    17992      52101 /usr/lib/libplc4.so
openvpn 6303 root  mem    REG        8,2    26172     308087 /lib/libnss_winbind.so.2
openvpn 6303 root  mem    REG        8,2    36040      30325 /lib/libnss_compat-2.11.3.so
openvpn 6303 root  mem    REG        8,2    43341      36295 /lib/librt-2.11.3.so
openvpn 6303 root  mem    REG        8,2    49683      30401 /lib/libnss_nis-2.11.3.so
openvpn 6303 root  mem    REG        8,2    55044      26741 /lib/libpam.so.0.83.0
openvpn 6303 root  mem    REG        8,2    58657      30330 /lib/libnss_files-2.11.3.so
openvpn 6303 root  mem    REG        8,2    71324      42906 /usr/lib/libtasn1.so.3.1.5
openvpn 6303 root  mem    REG        8,2    87752      36361 /lib/libz.so.1.2.5
openvpn 6303 root  mem    REG        8,2    91520     359931 /lib/libaudit.so.1.0.0
openvpn 6303 root  mem    REG        8,2   107282      30318 /lib/libnsl-2.11.3.so
openvpn 6303 root  mem    REG        8,2   108452      47709 /usr/lib/libnssutil3.so
openvpn 6303 root  mem    REG        8,2   112512      47017 /usr/lib/libpkcs11-helper.so.1.0.0
openvpn 6303 root  mem    REG        8,2   120868      21857 /lib/libselinux.so.1
openvpn 6303 root  mem    REG        8,2   125115      30534 /lib/libpthread-2.11.3.so
openvpn 6303 root  mem    REG        8,2   140796      22174 /lib/liblzo2.so.2.0.0
openvpn 6303 root  mem    REG        8,2   143979      23140 /lib/ld-2.11.3.so
openvpn 6303 root  mem    REG        8,2   170960      56859 /usr/lib/libsmime3.so
openvpn 6303 root  mem    REG        8,2   241720      32437 /usr/lib/libnspr4.so
openvpn 6303 root  mem    REG        8,2   301312     294437 /lib/libdbus-1.so.3.5.3
openvpn 6303 root  mem    REG        8,2   355552     490732 /lib/libssl.so.1.0.0
openvpn 6303 root  mem    REG        8,2   524484       1240 /lib/libgcrypt.so.11.6.0
openvpn 6303 root  mem    REG        8,2   671196     396438 /usr/lib/libgnutls.so.26.14.12
openvpn 6303 root  mem    REG        8,2  1328988      47708 /usr/lib/libnss3.so
openvpn 6303 root  mem    REG        8,2  1683935      24495 /lib/libc-2.11.3.so
openvpn 6303 root  mem    REG        8,2  1693496     490731 /lib/libcrypto.so.1.0.0
openvpn 6303 root  rtd    DIR        8,2      648          2 /
openvpn 6303 root  txt    REG        8,2   654360    2207194 /usr/sbin/openvpn

This is the configuration of the openVPN server:


local 192.168.2.10
port 1195
proto udp
dev tap0
ca <ca.crt>
cert <cert.crt>
key <cert.key>  # This file should be kept secret
dh keys/dh2048.pem
ifconfig-pool-persist ipp_dsl_tap0.txt
server-bridge 172.20.10.34 255.255.0.0 172.20.10.201 172.20.10.221
push "dhcp-option DNS 172.20.10.10"
push "dhcp-option DNS 172.20.10.11"
push "dhcp-option DOMAIN <domain>"
client-to-client
keepalive 10 120
comp-lzo
user nobody
persist-key
persist-tun
status openvpn-status_dsl_tap0.log
verb 3
plugin /etc/openvpn/openvpn-auth-pam.so login
client-cert-not-required
username-as-common-name

ab · December 13, 2011, 2:32pm

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Is this a complete listing of the lsof output? That is really not many
files to have open by one process. Also did you change your command or
restart OpenVPN at some point or something? The PID you mentioned in
the lsof command does not match the lsof output… probably just because
of ongoing work on your side and that’s fine.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJO51QLAAoJEF+XTK08PnB5heIP/R0dLO8CFYFj/7CZhl17ik/1
65dudGo+/CgKmv6C3iebkTsS/IrRNSAKDhYJAqvRfthnTgfb+mHl8SHzJORngFsQ
qenQwCUAwvMes/oJPp3b7wMhb2Q75fpb4WYStW1IdjE3fJh41B4KR6ljst7WWnSj
5CligrC9rw3KDFfuMN2qBgemEABIcExG7UyVBmDsvKS8wORLvc/u477kt6Ux11kz
5/4atrM8P5OHbMbCGzp5RaKb3Vr1z33rZgd7YD90+pNTGBwCcfHoZDlSvTNf4GLD
ZnYHZydby8LGRZ3RoNL5fZHx3Ox73t99MJKYRSqORg1IT83+ZWwuEWybBULLWSSC
Ie/yQQj3Suvo+4s6uX2xYShfhSAH4UwJjRxNJD2VfnSJ9tVZ5bDcxghgW2Wuu6Dd
+s0OwVs5tV5T1TmaHIsAnYdejF5KqipAQ8sxRJAFXSm+TzkbgsHR9Jv9WPbu0xpu
mXXVlLrhoGKfr1Az5Tjwa+pQKwJU9KMYwsEeudfIlGbM5XRJqrPbFSVKjaZXR0C2
j3bBA3vo/N/7O2a8+BEOjZZ8CiWnz12IaYA4/GCxU54mJq0KCQWuWTjIspd6bOJu
uqgL+9d5OJ/pT96SAjop0niALbPHWQ9ATVYtJK4eCidqZxAL33C6uDot+djXPBM8
tkkDqqBwOpnZ2apIwQhg
=q7eh
-----END PGP SIGNATURE-----

r-cane · December 13, 2011, 10:35pm

OpenVPN was indeed restarted. That’s why the PID is different and the number of files is small. This is the lsof output from this morning:


COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
openvpn 6303 root    0u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    1u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    2u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    3u  unix 0xf3173800      0t0 1002734187 socket
openvpn 6303 root    5u  unix 0xf315b300      0t0 1002733627 socket
openvpn 6303 root    6u  unix 0xf7345800      0t0 1002734192 socket
openvpn 6303 root    7u  unix 0xf359b580      0t0 1002734374 socket
openvpn 6303 root    8u  unix 0xf30f2d00      0t0 1002737606 socket
openvpn 6303 root    9u  unix 0xf30f2300      0t0 1002737716 socket
openvpn 6303 root   10u  unix 0xf30f2580      0t0 1002737782 socket
openvpn 6303 root   11u  unix 0xf34bd300      0t0 1002737845 socket
openvpn 6303 root   12u  unix 0xf6fd0580      0t0 1002737955 socket
openvpn 6303 root  cwd    DIR        8,2      648          2 /
openvpn 6303 root  mem    REG        8,2     9688     110153 /lib/security/pam_ck_conn
openvpn 6303 root  mem    REG        8,2    13732      21753 /lib/libgpg-error.so.0.7.
openvpn 6303 root  mem    REG        8,2    13844     110173 /usr/lib/libck-connector.
openvpn 6303 root  mem    REG        8,2    13864      73704 /usr/lib/libplds4.so
openvpn 6303 root  mem    REG        8,2    17392      30309 /lib/libdl-2.11.3.so
openvpn 6303 root  mem    REG        8,2    17680     116462 /etc/openvpn/openvpn-auth
openvpn 6303 root  mem    REG        8,2    17992      52101 /usr/lib/libplc4.so
openvpn 6303 root  mem    REG        8,2    26172     308087 /lib/libnss_winbind.so.2
openvpn 6303 root  mem    REG        8,2    36040      30325 /lib/libnss_compat-2.11.3
openvpn 6303 root  mem    REG        8,2    43341      36295 /lib/librt-2.11.3.so
openvpn 6303 root  mem    REG        8,2    49683      30401 /lib/libnss_nis-2.11.3.so
openvpn 6303 root  mem    REG        8,2    55044      26741 /lib/libpam.so.0.83.0
openvpn 6303 root  mem    REG        8,2    58657      30330 /lib/libnss_files-2.11.3.
openvpn 6303 root  mem    REG        8,2    71324      42906 /usr/lib/libtasn1.so.3.1.
openvpn 6303 root  mem    REG        8,2    87752      36361 /lib/libz.so.1.2.5
openvpn 6303 root  mem    REG        8,2    91520     359931 /lib/libaudit.so.1.0.0
openvpn 6303 root  mem    REG        8,2   107282      30318 /lib/libnsl-2.11.3.so
openvpn 6303 root  mem    REG        8,2   108452      47709 /usr/lib/libnssutil3.so
openvpn 6303 root  mem    REG        8,2   112512      47017 /usr/lib/libpkcs11-helper
openvpn 6303 root  mem    REG        8,2   120868      21857 /lib/libselinux.so.1
openvpn 6303 root  mem    REG        8,2   125115      30534 /lib/libpthread-2.11.3.so
openvpn 6303 root  mem    REG        8,2   140796      22174 /lib/liblzo2.so.2.0.0
openvpn 6303 root  mem    REG        8,2   143979      23140 /lib/ld-2.11.3.so
openvpn 6303 root  mem    REG        8,2   170960      56859 /usr/lib/libsmime3.so
openvpn 6303 root  mem    REG        8,2   241720      32437 /usr/lib/libnspr4.so
openvpn 6303 root  mem    REG        8,2   301312     294437 /lib/libdbus-1.so.3.5.3
openvpn 6303 root  mem    REG        8,2   355552     490732 /lib/libssl.so.1.0.0
openvpn 6303 root  mem    REG        8,2   524484       1240 /lib/libgcrypt.so.11.6.0
openvpn 6303 root  mem    REG        8,2   671196     396438 /usr/lib/libgnutls.so.26.
openvpn 6303 root  mem    REG        8,2  1328988      47708 /usr/lib/libnss3.so
openvpn 6303 root  mem    REG        8,2  1683935      24495 /lib/libc-2.11.3.so
openvpn 6303 root  mem    REG        8,2  1693496     490731 /lib/libcrypto.so.1.0.0
openvpn 6303 root  rtd    DIR        8,2      648          2 /
openvpn 6303 root  txt    REG        8,2   654360    2207194 /usr/sbin/openvpn

This is the output from half an hour ago:


COMMAND  PID USER   FD   TYPE     DEVICE SIZE/OFF       NODE NAME
openvpn 6303 root    0u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    1u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    2u   CHR        1,3      0t0        858 /dev/null
openvpn 6303 root    3u  unix 0xf3173800      0t0 1002734187 socket
openvpn 6303 root    5u  unix 0xf315b300      0t0 1002733627 socket
openvpn 6303 root    6u  unix 0xf7345800      0t0 1002734192 socket
openvpn 6303 root    7u  unix 0xf359b580      0t0 1002734374 socket
openvpn 6303 root    8u  unix 0xf30f2d00      0t0 1002737606 socket
openvpn 6303 root    9u  unix 0xf30f2300      0t0 1002737716 socket
openvpn 6303 root   10u  unix 0xf30f2580      0t0 1002737782 socket
openvpn 6303 root   11u  unix 0xf34bd300      0t0 1002737845 socket
openvpn 6303 root   12u  unix 0xf6fd0580      0t0 1002737955 socket
openvpn 6303 root   13u  unix 0xf31ce580      0t0 1002843549 socket
openvpn 6303 root   14u  unix 0xf31ced00      0t0 1002843655 socket
openvpn 6303 root   15u  unix 0xf6c0da80      0t0 1002843765 socket
openvpn 6303 root   16u  unix 0xf37dc580      0t0 1002933565 socket
openvpn 6303 root   17u  unix 0xf37dc300      0t0 1002933718 socket
openvpn 6303 root   18u  unix 0xf1567300      0t0 1002933781 socket
openvpn 6303 root   19u  unix 0xf6f20a80      0t0 1003011920 socket
openvpn 6303 root   20u  unix 0xf3173580      0t0 1003026294 socket
openvpn 6303 root   21u  unix 0xf3173a80      0t0 1003026359 socket
openvpn 6303 root   22u  unix 0xf1751300      0t0 1003026422 socket
openvpn 6303 root   23u  unix 0xf15a9a80      0t0 1003117243 socket
openvpn 6303 root   24u  unix 0xf7345d00      0t0 1003117349 socket
openvpn 6303 root   25u  unix 0xf15a9300      0t0 1003117412 socket
openvpn 6303 root   26u  unix 0xf3629a80      0t0 1003205790 socket
openvpn 6303 root   27u  unix 0xf3629d00      0t0 1003205895 socket
openvpn 6303 root   28u  unix 0xf3629800      0t0 1003205959 socket
openvpn 6303 root   29u  unix 0xf3390d00      0t0 1003288439 socket
openvpn 6303 root   30u  unix 0xf3390300      0t0 1003288588 socket
openvpn 6303 root   31u  unix 0xf6fcf800      0t0 1003288652 socket
openvpn 6303 root   32u  unix 0xf7360080      0t0 1003349326 socket
openvpn 6303 root   33u  unix 0xf6c64080      0t0 1003370937 socket
openvpn 6303 root   34u  unix 0xf6e03300      0t0 1003371047 socket
openvpn 6303 root   35u  unix 0xf1628580      0t0 1003432581 socket
openvpn 6303 root   36u  unix 0xf16a1800      0t0 1003453911 socket
openvpn 6303 root   37u  unix 0xf37c1080      0t0 1003453974 socket
openvpn 6303 root   38u  unix 0xcbecc300      0t0 1003512277 socket
openvpn 6303 root   39u  unix 0xf7111d00      0t0 1003533460 socket
openvpn 6303 root   40u  unix 0xf1713d00      0t0 1003612451 socket
openvpn 6303 root   41u  unix 0xf358c080      0t0 1003626275 socket
openvpn 6303 root   42u  unix 0xf35faa80      0t0 1003704040 socket
openvpn 6303 root  cwd    DIR        8,2      648          2 /
openvpn 6303 root  mem    REG        8,2     9688     110153 /lib/security/pam_ck_conn
openvpn 6303 root  mem    REG        8,2    13732      21753 /lib/libgpg-error.so.0.7.
openvpn 6303 root  mem    REG        8,2    13844     110173 /usr/lib/libck-connector.
openvpn 6303 root  mem    REG        8,2    13864      73704 /usr/lib/libplds4.so
openvpn 6303 root  mem    REG        8,2    17392      30309 /lib/libdl-2.11.3.so
openvpn 6303 root  mem    REG        8,2    17680     116462 /etc/openvpn/openvpn-auth
openvpn 6303 root  mem    REG        8,2    17992      52101 /usr/lib/libplc4.so
openvpn 6303 root  mem    REG        8,2    26172     308087 /lib/libnss_winbind.so.2
openvpn 6303 root  mem    REG        8,2    36040      30325 /lib/libnss_compat-2.11.3
openvpn 6303 root  mem    REG        8,2    43341      36295 /lib/librt-2.11.3.so
openvpn 6303 root  mem    REG        8,2    49683      30401 /lib/libnss_nis-2.11.3.so
openvpn 6303 root  mem    REG        8,2    55044      26741 /lib/libpam.so.0.83.0
openvpn 6303 root  mem    REG        8,2    58657      30330 /lib/libnss_files-2.11.3.
openvpn 6303 root  mem    REG        8,2    71324      42906 /usr/lib/libtasn1.so.3.1.
openvpn 6303 root  mem    REG        8,2    87752      36361 /lib/libz.so.1.2.5
openvpn 6303 root  mem    REG        8,2    91520     359931 /lib/libaudit.so.1.0.0
openvpn 6303 root  mem    REG        8,2   107282      30318 /lib/libnsl-2.11.3.so
openvpn 6303 root  mem    REG        8,2   108452      47709 /usr/lib/libnssutil3.so
openvpn 6303 root  mem    REG        8,2   112512      47017 /usr/lib/libpkcs11-helper
openvpn 6303 root  mem    REG        8,2   120868      21857 /lib/libselinux.so.1
openvpn 6303 root  mem    REG        8,2   125115      30534 /lib/libpthread-2.11.3.so
openvpn 6303 root  mem    REG        8,2   140796      22174 /lib/liblzo2.so.2.0.0
openvpn 6303 root  mem    REG        8,2   143979      23140 /lib/ld-2.11.3.so
openvpn 6303 root  mem    REG        8,2   170960      56859 /usr/lib/libsmime3.so
openvpn 6303 root  mem    REG        8,2   241720      32437 /usr/lib/libnspr4.so
openvpn 6303 root  mem    REG        8,2   301312     294437 /lib/libdbus-1.so.3.5.3
openvpn 6303 root  mem    REG        8,2   355552     490732 /lib/libssl.so.1.0.0
openvpn 6303 root  mem    REG        8,2   524484       1240 /lib/libgcrypt.so.11.6.0
openvpn 6303 root  mem    REG        8,2   671196     396438 /usr/lib/libgnutls.so.26.
openvpn 6303 root  mem    REG        8,2  1328988      47708 /usr/lib/libnss3.so
openvpn 6303 root  mem    REG        8,2  1683935      24495 /lib/libc-2.11.3.so
openvpn 6303 root  mem    REG        8,2  1693496     490731 /lib/libcrypto.so.1.0.0
openvpn 6303 root  rtd    DIR        8,2      648          2 /
openvpn 6303 root  txt    REG        8,2   654360    2207194 /usr/sbin/openvpn

As you can see, the number of open sockets has grown considerably. Can I somehow prevent this or close the open (and not needed) sockets?