OpenSuSE 11.3 (but this is not likely a distro specific issue)
Kernel 2.6.34.7, but likely applies to a number of previous
Am close to launching a more extensive webpage describing in detail why and how this configuration is important, but I’m missing up to date information on how to persist the config. By default, changing the TCP Congestion Control algorithm only modifies the current system session. Docs I’ve read recommend that the command resetting the algorithm be written to
/etc/rc.d/rc.d
Am confused with all the different config files, but I suspect either an existing file should be modified or an additional file should be modified in
/etc/rc.d/rc5.d/
Question:
Need a recommendation. Should I just choose to modify a config file which might have remote relevance to kernel networking modules or should I just create a new file in rc5 something like
/etc/rc.d/rc5.d/K07networkkernel
Or, should something else be done?
BTW - As an example, the command to be implemented if implementing the veno algorithm which specially analyzes for wireless packet loss:
On 01/27/2011 04:36 PM, tsu2 wrote:
>
> OpenSuSE 11.3 (but this is not likely a distro specific issue)
> Kernel 2.6.34.7, but likely applies to a number of previous
>
> Am close to launching a more extensive webpage describing in detail why
> and how this configuration is important, but I’m missing up to date
> information on how to persist the config. By default, changing the TCP
> Congestion Control algorithm only modifies the current system session.
> Docs I’ve read recommend that the command resetting the algorithm be
> written to
>
>
> Code:
> --------------------
> /etc/rc.d/rc.d
> --------------------
>
>
> Am confused with all the different config files, but I suspect either
> an existing file should be modified or an additional file should be
> modified in
>
>
> Code:
> --------------------
> /etc/rc.d/rc5.d/
> --------------------
>
>
> Question:
> Need a recommendation. Should I just choose to modify a config file
> which might have remote relevance to kernel networking modules or should
> I just create a new file in rc5 something like
>
>
> Code:
> --------------------
> /etc/rc.d/rc5.d/K07networkkernel
> --------------------
>
>
> Or, should something else be done?
>
> BTW - As an example, the command to be implemented if implementing the
> veno algorithm which specially analyzes for wireless packet loss:
>
> echo “veno” > /proc/sys/net/ipv4/tcp_congestion_control
>
> TIA,
> Tony
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
Thx, Abe, but your post doesn’t actually address what I’m asking for…
By default, current Linux kernels default to the cubic congestion control algorithm, but it’s a very consservative algorithm that assumes small to medium file transfers with excellent network connectivity, most likely using a wired connection.
The reason why I want to change to veno is becuase that algorithm better analyzes for wireless packet loss (I usually connect using an 802.11 connection), and instead of re-compiling the kernel today it’s possible to take advantage of the kernel architecture’s “pluggable kernel modules.”
The command I posted is the proper command to invoke the veno congestion control module, but because it’s not part of the base kernel compilation, the invocation is only session-specific and is lost with the next reboot.
As I’ve posted, docs have suggested invoking the command with a config file in rc.d, but I’m asking for more specific guidance to avoid possible future issues with other configs.
If it turns out in the end that no one with specific knowledge of the kernel pluggable module architecture can recommend “best practice” I may just do whatever works, post the practice and wait for a reaction.
Per your suggestions I re-read the sysctl man pages (both) and re-considered whether Abe’s original suggestion might actually work.
If I read the Man correctly, Abe’s code does write to sysctl, so I think that is the recommended method to do this. I also found a web page on the Internet that suggested that this command is supposed to “make kernel changes at runtime.” I also opened up sysctl.conf in a text editor but didn’t notice anything that makes me comfortable doing a direct edit (although maybe eventually I may try that also unless someone says to just go do it).
Unfortunately, the results are no different than my original methods. Changes are not persisted across reboots. If anyone wants to verify this is so, here are the commands (they’re safe for anyone to use)
# Determine current module loaded, default is cubic
cat /proc/sys/net/ipv4/tcp_congestion_control
# Running Abe's code writing a change to sysctl
sudo /sbin/sysctl -w net.ipv4.tcp_congestion_control=veno
# Verify module has changed to new module
cat /proc/sys/net/ipv4/tcp_congestion_control
# Reboot now.
# Determine which congestion control module is loaded
cat /proc/sys/net/ipv4/tcp_congestion_control
Am speculating that although writing to sysctl may be appropriate for many LKM (Loadable Kernel Modules), the pluggable modules for congestion control may be a special case where the recommended approach isn’t to load the appropriate module at bootup, but to modify the the module after bootup. That would probably explain the recommendation I ran into recommending changes to rc.d.
Hmm… a SLES 11 server behaved similarly. I was able to make it
persist by simply adding the same parameter to the /etc/sysctl.conf file
which is probably a bad practice since it is owned by the procps package.
Still tinkering, but that is working properly.
The line I added:
net.ipv4.tcp_congestion_control = reno
Good luck.
On 01/31/2011 09:36 AM, tsu2 wrote:
>
> ken_yap;2284495 Wrote:
>> Edit /etc/sysctl.conf. Read /etc/init.d/boot.sysctl and the sysctl man
>> page and you will understand.
>
> Thx Ken (and Abe again),
>
> Unfortunately, still no joy.
>
> Per your suggestions I re-read the sysctl man pages (both) and
> re-considered whether Abe’s original suggestion might actually work.
>
> If I read the Man correctly, Abe’s code does write to sysctl, so I
> think that is the recommended method to do this. I also found a web page
> on the Internet that suggested that this command is supposed to “make
> kernel changes at runtime.” I also opened up sysctl.conf in a text
> editor but didn’t notice anything that makes me comfortable doing a
> direct edit (although maybe eventually I may try that also unless
> someone says to just go do it).
>
> Unfortunately, the results are no different than my original methods.
> Changes are not persisted across reboots. If anyone wants to verify this
> is so, here are the commands (they’re safe for anyone to use)
>
>
> Code:
> --------------------
> # Determine current module loaded, default is cubic
> cat /proc/sys/net/ipv4/tcp_congestion_control
> # Running Abe’s code writing a change to sysctl
> sudo /sbin/sysctl -w net.ipv4.tcp_congestion_control=veno
> # Verify module has changed to new module
> cat /proc/sys/net/ipv4/tcp_congestion_control
> # Reboot now.
> # Determine which congestion control module is loaded
> cat /proc/sys/net/ipv4/tcp_congestion_control
> --------------------
>
>
> Am speculating that although writing to sysctl may be appropriate for
> many LKM (Loadable Kernel Modules), the pluggable modules for congestion
> control may be a special case where the recommended approach isn’t to
> load the appropriate module at bootup, but to modify the the module
> after bootup. That would probably explain the recommendation I ran into
> recommending changes to rc.d.
>
> Thx,
> Tony
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
Before I recommend a direct edit to sysctl.conf, unless I hear from someone else I’ll probably start experimenting with modifying rc.d which to me is a more “elegant” and in my mind likely more appropriate solution if I can get it to work properly.
I did a test using the sysctl.conf file and then upgraded the package and
the change survived the upgrade which is interesting. That would make me
think it is safe to go that route which, to me, is cleaner than using an
init script. Your call, though.
Good luck.
On 01/31/2011 10:36 AM, tsu2 wrote:
>
> Thx Abe for the update.
>
> Before I recommend a direct edit to sysctl.conf, unless I hear from
> someone else I’ll probably start experimenting with modifying rc.d which
> to me is a more “elegant” and in my mind likely more appropriate
> solution if I can get it to work properly.
>
> Tony
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
And now I wonder if my results were simply because the sysctl.conf wasn’t
changed in the new RPM… not sure on that one since it was via an update
channel.
It is for reasons like these that having the sysconfig side of things or
special directories for extension of conf files (as done with Apache, or
ld.so.conf, or whatever) is nice since it keeps the benefits of packages
but without the drawbacks of packages.
Good luck.
On 01/31/2011 11:40 AM, ab@novell.com wrote:
> I did a test using the sysctl.conf file and then upgraded the package and
> the change survived the upgrade which is interesting. That would make me
> think it is safe to go that route which, to me, is cleaner than using an
> init script. Your call, though.
>
> Good luck.
>
>
>
>
> On 01/31/2011 10:36 AM, tsu2 wrote:
>
>> Thx Abe for the update.
>
>> Before I recommend a direct edit to sysctl.conf, unless I hear from
>> someone else I’ll probably start experimenting with modifying rc.d which
>> to me is a more “elegant” and in my mind likely more appropriate
>> solution if I can get it to work properly.
>
>> Tony
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
After skimming /etc/sysconfig/kernel, I’m uncomfortable with that approach… because
The kernel has already been compiled (original compilation) to load the cubic pluggable kernel module.
The commands I saw in the /etc/sysconfig/kernel file look like they only load, don’t unload and load, or take a loaded module and modify it. As it stands right now, I’m envisioning a race situation where simply installing a new congestion control module either could cause a conflict or if loaded before the default “cubic” module be modified back to the default.
Am pretty sure all the modules loaded in /etc/sysconfig/kernel are simply additions to the default compiled kernel, not altering an existing.
Am pretty sure the purpose of the idea of implementing an “rc.d” solution is that it ensures that the default module is loaded first as compiled, and only then when starting to invoke daemons to modify the module.
sysconfig/kernel only loads modules, perhaps with parameters, never configures them. That’s later for sysctl.
When you compile a loadable module, it’s not part of the booted kernel, it’s loaded on demand just like provided modules. If you try to load a module twice, it just ignores the second try. So there is no race condition.
I don’t know what happens if you try to load two modules that do the same thing. However you have exactly the same problem doing it in a rc script. Also it’s not clear if you have two modules or just one module with an algorithm parameter.
All that sysconfig/kernel does is load modules at boot time. It’s no different from doing it yourself later on in the boot sequence, just earlier.
The more I think about this, I tend to agree. When I opened up sysctl.config and saw references to initrd, I wasn’t sure whether that might mean a possible race condition if an alternate algorithm was invoked but the more I think about it, initrd isn’t a kernel being compiled, it’s an image of a pre-compiled kernel and likely already with the cubic algorithm embedded and enabled… So invoking a different module at <any> time thereafter should have the same effect.
The way the Loadable Kernel Modules has been described in the articles I’ve read is that they are each completely separate modules, not one module with an algorithm parameter, and a file at /proc/… points to any module that’s pre-compiled and available, optionally enabling the module. So, in this case, the value in “tcp_congestion_control” enables the specified Congestion Control module.
Your idea raising the possibility of a single module with algorithm parameter might make sense only in this case because every module manages the same packet variables, but other types of LKM aren’t so similar to each other.
Given all that,
It seems to me then that invoking a working command just about anytime during bootup would function fine, it’s now a matter of determining “best practice” placing the command in the proper place, likely along with other similar functions or commands. IMO this argues against editing sysctl.config and /etc/sysconfig/kernel which look like files which should probably be left alone and I would guess over-written with the next upgrade.
Initrd doesn’t contain a kernel. It contains an initial root filesystem for bootstrapping. The main use of it is to load modules that are needed to make it possible to mount the working root filesystem, e.g. SCSI drivers, filesystem drivers, but other actions that must be done before the working root filesystem is mounted can be done from initrd.
sysconfig and sysctl.conf should be safe against modification by updates because they hold working config info and normal practice is not to break a working config when a package is updated. If new settings are introduced, normal practice is to mark it in the RPM so that a foo.rpmnew is created if foo has been modified by the sysadmin and then the sysadmin must go and see if any reconciliation is needed to bring in new settings.
I have confirmed this as well. SUSE’s policy is to NOT revert changes in
config files by using the %config(noreplace) option in the SPEC file. As
a result if you modify /etc/sysctl.conf it will never be changed by the
RPM again and, if it does get changed, that’s a bug (but you can check the
SPEC file yourself to confirm things are set properly if you’d like). As
a result, /etc/sysctl.conf is the officially-correct place to make this
change.
Good luck.
On 02/01/2011 05:36 PM, ken yap wrote:
>
> Initrd doesn’t contain a kernel. It contains an initial root filesystem
> for bootstrapping. The main use of it is to load modules that are needed
> to make it possible to mount the working root filesystem, e.g. SCSI
> drivers, filesystem drivers, but other actions that must be done before
> the working root filesystem is mounted can be done from initrd.
>
> sysconfig and sysctl.conf should be safe against modification by
> updates because they hold working config info and normal practice is not
> to break a working config when a package is updated. If new settings are
> introduced, normal practice is to mark it in the RPM so that a
> foo.rpmnew is created if foo has been modified by the sysadmin and then
> the sysadmin must go and see if any reconciliation is needed to bring in
> new settings.
>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
Hmmm, I thought that to enable bootstrapping initrd needs to contain the kernel? I’ll have to re-read info on that.
In any case, here is my latest update.
After considering the options and doing a lot more research, I now understand the relationship between sysctl and the /proc/sys/… tree, the later is the actual layout which includes all info about what is available, what is enabled and how they are configured relating to the processor. Although it’s possible to write directly to a file using the “echo” method I originally posted, the sysctl interface may be the preferred way to interact with /proc/sys/… because it implements some error-checking and is less likely to allow a potentially critical mistake.
I also was able to find more references that describe how to persist sysctl changes across reboots, mentioning both direct edits to sysctl.conf and implementing scripts in the rc.d directory.
I’ve now tried all possible ways to persist sysctl changes and it seems that editing the sysctl.conf is the only way which works. I tried creating new rc.d scripts using both the “echo” and “sysctl -w” commands, but neither works.
What you state about preserving package information relating to rpms make sense, but the sysctl.cfg doesn’t do that, it stores information related to the installed kernel… so can we assume when a new kernel is installed with an upgrade (not update), will the sysctl.config be preserved?
Yes, I can see why an update might be OK, but would an OS upgrade be different?
The bootloader loads two things: the kernel and the initrd. The initrd is optional for very simple setups.
sysctl.conf stores settings that you want to be activated for this machine. It’s not tied to a particular kernel package.
If you do an OS (as opposed to a kernel) upgrade all bets are off. The package procps which sysctl.conf belongs to may replace it or may tell you there’s an updated version. You should backup sysctl.conf just like you would backup any other file in /etc and examine it later.
Did some more reading and review, the bootloader boots both a “basic” kernel image and the initrd image to provide an “early” root file system, but it’s not an empty root file system… In most distros, it seems that LKM intended for early kernel functionality is stored in the initrd and is loaded into the kernel. By the looks of sysctl.config, OpenSUSE does exactly that, there are different sections where at different stages LKM can be loaded and because the TCP Congestion Control module is not named to me that lends further credence that the CUBIC module is pre-compiled into the base kernel.
So, to me the bottom line becomes clearer that no race condition should exist, invoking another Congestion Control module at any time should be OK.
Thx for confirming my suspicion the likelihood a custom sysctl.conf would not survive an upgrade (since 11.4 is only a month away as of this post it looks like a test is coming up). And, unfortunately some further attempts at getting an rc.d script to work are failures.
Maybe someone might notice something I’m missing. This is the latest of what I’ve tried
I don’t think that the script is bad, I can run the following in a terminal which verifies the script works
# Verify existing algorithm which should return cubic
cat proc/sys/net/ipv4/tcp_congestion_control
# execute new custom script
/etc/rc.d/rc5.d/K03network-tcpconfgestioncontrol
# Verify that script has switched system to new algorithm
cat proc/sys/net/ipv4/tcp_congestion_control
Unlike other distros, it doesn’t work in SUSE to just place a script in a runlevel directory. You have to provide the metadata given in the example /etc/init.d/skeleton template and to insserv the script.
Thx.
Looks like a fairly involved process which will require not just decyphering the structure and parameters, it also looks like every custom entry also requires researching available and possibly submitting new namespaces to make the entry “universal” and conform to standards.
Due to time constraints, I’ll have to set aside implementing rc.d scripts for awhile…
