/etc/sysctl.conf not read on boot, no longer usable

Although I had noticed that the /etc/sysctl.conf file has not been read for awhile now in openSUSE, I did not consider the problem beyond the optional customizations I’ve been implementing. I now understand this to be a major issue with far reaching consequences.

Issue:
This configuration file which was commonly used prior to systemd changes no longer is usable at all. There is a bugzilla thread discussing how to handle this problem, but for many Users, they cannot wait… They need a solution now (see below).
**
Scope of issue:**
For Upgraders, this file almost certainly contains entries. Apparently upgrading does <not> properly parse this file and convert entries so these entries remain but are invalid after upgrade.
For Everyone, this file ordinarily should be avoided for any customizations. There is a very large body of support documents on the Internet which describes making entries into this file, but those “Help” documents should now be considered invalid without modification.
Important Note: There is <something> that is currently being run whenever I run “zypper up” which likely is relying on settings in sysctl.conf, my current most likely candidate is libvirt (vm manager, vm install, etc). And, if this can happen in this specific situation of course it can happen in any other install/configure routine that might be calling sysctl.conf

Issue which led me to discover this is a significant problem:
Troubleshooting erratic virtual network bridge devices and the virtual networks they support, they would work for only about a day and then become useless.

  • Verified bridge devices exist
  • Verified bridge devices were placed in SUSE FW internal zone
  • Verified bridge-nf scripts were set to disable
  • Verified no other FW software was running despite a generic warning that popped up whenever running SUSE FW from YAST.
  • verified IPTABLES rule enabling IP_FORWARDING is set correctly

But, discovered
The following set to zero

/proc/sys/net/ipv4/ip_forward

And, further discovery the following lines in my sysctl.conf

kernel.sysrq = 0                                                                                                                                               
net.ipv4.ip_forward = 1                                                                                                                                        
net.ipv4.tcp_syncookies = 1                                                                                                                                    
net.ipv6.conf.all.forwarding = 0   

Apparent Result:
Whenever a new bridge device is created by libvirt or YAST, typically it will work initially. But, sometimes with the next “zypper up” and almost certainly with the next boot, the bridge device will no longer forward packets. The possibilities are myriad on my machine what might be happening so it’s very difficult to pinpoint exactly all possible causes although the current strong candidate is something related to libvirt(but notice that the consequences affects everything, not just libvirt).

Solution:
Regardless of the possible causes, there seems to be a reliable solution!
For the past 3 days now, the fix I’ve implemented seems to be holding, am still testing the scope of the fix.

**1. Go “old school” **- Sysctl read and wrote to other files, it was simply a centralized management tool which preserved original settings while implementing custom and provided some fault tolerance(mistakes simply defaulted to original settings). With this method, although the original, default settings are lost, you can write into the original files. An example follows for the above settings which will likely be found in many if not all openSUSE:

The follwing script copies the original files to another location (you can modify) for backup

mkdir /backup_original_before_sysctl
cp /proc/sys/kernel/sysrq /backup_original_before_sysctl/
cp /proc/sys/net/ipv4/ip_forward /backup_original_before_sysctl/
cp /proc/sys/net/ipv4/tcp_syncookies /backup_original_before_sysctl
cp /proc/sys/net/ipv6/conf/all/forwarding /backup_original_before_sysctl

The following script writes the sysctl.conf values into the original config files

echo 0 > /proc/sys/kernel/sysrq
echo 1 > /proc/sys/ipv4/ip_forward
echo 1 > /proc/sys/ipv4/tcp_syncookies
echo 0 > /proc/sys/ipv6/conf/all/forwarding
  1. The likely preferred method going forward is to use the systemd Unit systemd-sysctl.service
    Unfortunately at this time I can’t find any existing templates or documentation how to properly write the dependency Units and the documented locations are empty. If anyone knows where this documentaiton can be found, pls post to this thread.

Will be submitting bugzilla(s) on this, but still considering how to frame this problem(s) because it involves many different components, and although the basic issue is the sysctl.conf file, the consequences are varied and far.

TSU

Yes. Just enable it and /etc/sysctl.conf is applied on boot:

sudo systemctl enable systemd-sysctl.service

Oh and it is enabled here although I never explicitely enabled it. I upgraded from 12.2 if that matters. There was a bug which would disable services if you upgraded from an earlier version to 12.3: https://bugzilla.novell.com/show_bug.cgi?id=809695

This is the systemd-sysctl.service text file, disabled by default and is located in the folder /usr/lib/systemd/system:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Apply Kernel Variables
Documentation=man:systemd-sysctl.service(8) man:sysctl.d(5)
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service
After=systemd-modules-load.service
Before=sysinit.target shutdown.target
ConditionPathIsReadWrite=/proc/sys/
ConditionPathExists=|/etc/sysctl.conf
ConditionDirectoryNotEmpty=|/lib/sysctl.d
ConditionDirectoryNotEmpty=|/usr/lib/sysctl.d
ConditionDirectoryNotEmpty=|/usr/local/lib/sysctl.d
ConditionDirectoryNotEmpty=|/etc/sysctl.d
ConditionDirectoryNotEmpty=|/run/sysctl.d

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/lib/systemd/systemd-sysctl

I have a bash script that you can use to find, edit and enable or disable the service file as you desire you can find here: SysdCmd - systemd Command Help/Config Editor - Blogs - openSUSE Forums

Thank You,

@jdmcdaniel3
Thx. good stuff I’ll dig deeper to verify all is configured correctly on my machine, although since I haven’t touched this file I’d expect mine to look the same way. On my machine, the service has always been running (status ative(exited)).

@wolfi323
Good points, but don’t seem to apply in my case. Something (possibly the upgrade?) enabled the service by default so it has always been running. The bug you referenced doesn’t apply in this case. And yet, unless I’m missing something, as I described in my original post there is verified and definite problem implementing the /etc/sysctl.conf entries.

Of course, I can’t know for sure if this problem affects just myself, all machines which are setup the same way (eg This is an upgrade from 12.1), or all machines without exception.

But, how many people really test their /etc/sysctl.conf configurations?

If anyone wanted to test their setup,

The “Optimizing your Network Connections” guide I wrote awhile back contains <many> modifications which are made using /etc/sysctl.conf, including how to test whether the modification is working and monitoring in realtime. Try just any one,. several or all.
https://sites.google.com/site/4techsecrets/optimize-and-fix-your-network-connection

And, for many (especially those who connect using wireless of any type), at least the method describing configuring the TCP/IP Congestion Control Algorithm should be permanently helpful.

The settings in the Guide can be tested not only on bare metal but in virtual machines, I would guess the only likely modification from what is in the guide is that “sysctl -p” likely won’t work, you’d have to reboot to apply changes.

As for myself,
I’d say that the fix I described in the original post of this thread continues to “take” so it appears that it fixes what I described… “ip forward” <should> have been applied with the sysctl.conf file, but it wasn’t. It’s less clear what is actually happening because as I described, a newly configured bridge device works properly which implies that it’s configured with default settings which are correct but different settings are applied at some point that alters and renders it ineffective.

Further testing,
I’ve also manually entered the settings I describe in the Guide above and now see the effects on my machine… unlike before with default settings, I can “feel” greater resources allocated when I’m using bittorrent - My downloads fill the available bandwidth more consistently and TCP/IP windows “feel” like they’re more long-lived.

TSU

I remember now what I had concluded when I first read this Unit file awhile back…

From this file, the only thing that can be verified is the conditional statement testing that the path to /etc/sysctl.conf exists.

Since systemd-sysctl is a binary now, it’s not possible to realy know what it does and if it’s working properly.

TSU

Did not occur to me before, but it seems to me that the “systemd equivalent” for “sysctl -p” should be the following (I don’t have anything to test for the moment)

systemctl restart systemd-sysctl.service

TSU

I just tried this:

  • set “net.ipv4.ip_forward = 1” in /etc/sysctl.conf (was set to 0 before)
  • run “systemctl restart systemd-sysctl.service”
    After that /proc/sys/net/ipv4/ip_forward was 1 as expected. :wink:

I ran the same test and it’s working for me, too.
How 'bout that. Cool.

But then, I tested changing the tcp_congestion_control to veno and that didn’t work.
I then checked to see whether the tcp buffer modifications I recommend <for special circumstances> in my guide were applied by sysctl.conf and they weren’t either.

So, the mystery deepens…
Could be that sysctl.conf is now being read, but curiously not applied consistently which is odd. I wonder if someone is in the process of fixing these things one at a time when there might be a more fundamental problem somewhere. Part of the problem is that ip_forward is a <standard> setting widely used in sysctl.conf for all distros, but the other entries I describe which are not default but widely described might not have been accounted for in systemd-sysctl.service (speculation but significant circumstantial evidence).

TSU

Well, I just tried that and it worked as well:

wolfi@amiga:~> cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
wolfi@amiga:~> sudo vim /etc/sysctl.conf #added the line: net.ipv4.tcp_congestion_control = veno
wolfi@amiga:~> sudo systemctl restart systemd-sysctl.service 
wolfi@amiga:~> cat /proc/sys/net/ipv4/tcp_congestion_control
veno
wolfi@amiga:~>

Maybe you had a typo somewhere?

The following should automatically add the custom entries I’ve been using in /etc/sysctl.conf (paste into a root console)

cat >> /etc/sysctl.conf <<EOF
# Custom TCP Buffer settings
 sysctl -w net.core.wmem_max=12582912
 sysctl -w net.core.rmem_max=12582912
 sysctl -w net.ipv4.tcp_rmem=10240 87380 1258291
 sysctl -w net.ipv4.tcp_wmem=10240 87380 12582912
 sysctl -w net.core.netdev_max_backlog =5000
# Change TCP Congestion Control
 sysctl -w net.ipv4.tcp_congestion_control = veno

net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-arptables=0
EOF

Then, as discussed

systemctl restart systemd-sysctl.conf

Then each can be tested, eg

cat /proc/sys/net/ipv4/tcp_congestion_control
cat /proc/sys/net/core/wmem_max
cat /proc/sys/net/bridge/bridge-nf-call-iptables

Returning the system to its original state can be done simply by either commenting or removing the lines of code at the end of /etc/sysctl.conf and re-starting systemd-sysctl.conf

TSU

No! This is wrong and can’t work, this wouldn’t even have worked before systemd!

You have to leave out "sysctl -w " in /etc/sysctl.conf.
sysctl.conf is a config file, no shell script.

Correct would be:

cat >> /etc/sysctl.conf <<EOF
# Custom TCP Buffer settings
net.core.wmem_max=12582912
net.core.rmem_max=12582912
net.ipv4.tcp_rmem=10240 87380 1258291
net.ipv4.tcp_wmem=10240 87380 12582912
net.core.netdev_max_backlog =5000
# Change TCP Congestion Control
net.ipv4.tcp_congestion_control = veno

net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-arptables=0
EOF

This should work. Do you see the difference? :wink:

Oh, and it seems to me there is a ‘2’ missing in the bolded line.
Shouldn’t that be:

net.ipv4.tcp_rmem=10240 87380 12582912

?

And another thing:

sysctl still works fine! At least on 12.3…

See here for proof:

amiga:~ # cat /proc/sys/net/ipv4/tcp_congestion_control
cubic
amiga:~ # sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic
amiga:~ # sysctl -w net.ipv4.tcp_congestion_control=veno
net.ipv4.tcp_congestion_control = veno
amiga:~ # sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = veno
amiga:~ # cat /proc/sys/net/ipv4/tcp_congestion_control
veno

sysctl.conf seems to be working properly on my machine now(been running stable for 3 days now), so the** original reason for reading this FAQ is no longer necessary.**

Although I can’t say for sure whether this has been because of User error or a fix/update, I had noticed this problem on my system for a very long time, and consistently. Regardless, it’s still nice when things suddenly start working properly.

I have returned my system to default values where there are modifications in sysctl.conf, then re-run “systemctl restart systemd-sysctl.conf” and values are being updated properly.

  • Upon closer inspection, I agree that “sysctl -w” in my sysctl.conf were redundant, and should not have been invoked that way, but I did verify that it seems to work (I double-checked before removing). Although it seems to be a non-critical error, of course should not be recommended because calling sysctl twice makes little sense(and now it looks like sysctl itself also works).
  • Thx for the incorrect value for tcp_rmem, I had noticed before but didn’t think about it in depth and agree it was an incorrect value.
  • Although I’m running systemd version 202 on this system, I doubt that should make a difference, the same Unit code for “systemd-sysctl.service” should exist in v195 (the default systemd for 12.3).

Am providing the following script for anyone who wrote values directly into the config files described in this thread and wishes to re-install default values (recommended) now that sysctl.conf should be working

Script to restore original, default settings (12.1-12.3)

echo 212992 > /proc/sys/net/core/wmeme_max
echo 212992 > /proc/sys/net/core/rmeme_max
echo 4096 87380 4133088 > /proc/sys/net/ipv4/tcp_rmem
echo 4096 16384 4133088 > /proc/sys/net/ipv4/tcp_wmem 
echo 1000 > /proc/sys/net/core/netdev_max_backlog
echo veno > /proc/sys/net/ipv4/tcp_congestion_control

As noted earlier in this thread and in my Guide,
I highly recommend the revised TCP Buffer settings if your machine is utilized as a more than lightly used Server, eg a Server with at least moderate traffic and many <simultaneous> connections, or applications like bittorrent which commonly open many half-open connections and over-ride normal TCP settings.

And, I highly recommend the veno congestion control algorithm if your WiFi connection isn’t optimal (many competing connections, RF interference, etc).

TSU

No! Having “sysctl -w” in the sysctl.conf is not redundant, it is INVALID! And I just tested this on 11.4 where there is no systemd yet!
Just try to run “sysctl -p” in a shell and you will get the following critical error message:

/proc/sys/sysctl -w net/ipv4/tcp_congestion_control: No such file or directory

You don’t really need this, since if you use “echo > /proc/sys/net/xxx” or “sysctl -w xxx” this isn’t saved anywhere, and when you reboot the values are reset to the defaults (if they are not set in /etc/sysctl.conf, of course).

Those (/proc/sys/net/ipv4/tcp_congestion_control f.e.) are no config files, but runtime kernel variables!

Not true.
If you execute the “echo 1 > /proc/sys/net/xxx” you’re writing to the file making hard changes. That is why in my original post this method was used, when the “virtual” sysctl method which over-rides the hard values wasn’t working, I resorted to making hard changes as last resort. That is also why I posted the script to return the hard values to their original <if sysctl can be made to work>.

It is also why using sysctl.conf should be the recommended way to make changes (instead of making hard changes). When you make hard changes,

  1. If you make a nonsensical mistake, it could possibly bork your system. If there is a mistake in using sysctl, then if the value could potentially bork your system, the system is supposed to disregard the sysctl value and default to the hard value.

  2. When you make hard changes, no automatic backup is done. You either should make backups as I described in my original post, copy the values to a file for safeguarding or be willing to do research looking for the original values.

TSU

Wrong. Those are no files. Just believe me.
Ever heard of the procfs filesystem?

It’s my understanding that the files represent values in the kernel.
Although I admit I have not tried, are you saying then that if the /proc/sys/* tree were removed that the kernel would run with original default values? It’s my understanding that they don’t, when you write a change directly to /proc/sys/* the changes are persistent which to me is a “hard change.”

Which is unlike how sysctl works. sysctl values are not persistent unless entered into sysctl.conf which is read on bootup. And, AFAIK there is no similar method of storing /proc/sys/* custom values during bootup over-riding kernel defaults.

Admittedly am leaning on stuff I had read years ago, and could have misunderstood what I read then,
TSU

I guess you just can’t remove them. Can’t try right now.

But they are no files on your harddisk. If the kernel reboots, the values are lost.

Just try booting a LiveCD and have a look in the /proc directory on your harddisk.
It will be empty!
What does that tell you?

And to learn more about procfs you could start your reading here:
procfs - Wikipedia, the free encyclopedia