linux 5.3.18-59.16-default x86_64
I found this in the syslog today, multiple entries:
2021-09-06T09:33:11-0700 sma-server3 kernel: Route cache is full: consider increasing sysctl net.ipv[4|6].route.max_size.
And the results are:
$ sysctl net.ipv4.route.max_size
net.ipv4.route.max_size = 2147483647
$ sudo sysctl net.ipv6.route.max_size
net.ipv6.route.max_size = 4096
Such an interesting suggestion has never appeared there before.
Would someone explain what it is regarding?
No.
Can you show us the output of “ip route list” ?
Yes.
$ ip route list
default via 192.168.69.1 dev eth0
192.168.69.0/24 dev eth0 proto kernel scope link src 192.168.69.246
Hi
Are you using any ipv6 at all? If not then suggest disabling (YaST -> Network Settings -> General tab), if you do then might want to tweak postfixes main.cf inet_protocol to only be ipv4 rather than all.
IPv6 is enabled for the hosts in our network. We have an assigned IPv6 block (fd2f:4760:521f:3f3c::/64), and have given each host an address. That is the extent of the explicit use of IPv6.
I discovered the IPv6 command for the route listing:
$ ip -6 route list
::1 dev lo proto kernel metric 256 pref medium
fd2f:4760:521f:3f3c::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd2f:4760:521f:3f3c::c0a8:4501 dev eth0 metric 1024 pref medium
default via fe80::2eb8:edff:fe5a:9d44 dev eth0 proto ra metric 1024 expires 1717sec hoplimit 64 pref medium
I do not understand why you suggested changing the configuration for postfix.
This is cache of per-destination entries. When Linux kernel resolves route to destination it puts it in cache for future use. AFAIK it is used only for IPv6 today, no more for IPv4. This case is unrelated to “routing between interfaces”. Output of
cat /proc/net/rt6_stats
ip -6 route show cache
when this happens would be interesting.
$ sudo sysctl net.ipv6.route.max_size net.ipv6.route.max_size = 4096
Pragmatic answer - if you hit this you may want to increase this limit. But of course it would be useful to (at least, try to) understand why you have so many entries. What does this host do? Is it Web/Mail or like server? Are you using torrents (or some other peer to peer solutions)? How may concurrent connections do you usually have?
$ cat /proc/net/rt6_stats
001a 000f 1c2ffe2 0011 0000 000e 0020
$ ip -6 route show cache
$
Pragmatic answer - if you hit this you may want to increase this limit.
I did that, to 8192. Although this warning had never occurred before, and has not since.
What does this host do? Is it Web/Mail or like server?
It provides mail services for our business.
It offers web services for the local network. A separate service handles the public website.
It offers DNS services locally and for our corporate domain.
Are you using torrents (or some other peer to peer solutions)?
No.
How may concurrent connections do you usually have?
I do not know. Not many? It is a small business.
Probably good to dump the route cache and see what is in it:
ip -s route show cache
What is “sudo sysctl net.ipv4.ip_forward” giving you?
$ ip -s route show cache
$
What is “sudo sysctl net.ipv4.ip_forward” giving you?
$ sudo sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 0
$
No forwarding enabled and the cache is also empty.
Of coarse the command is not given at the moment the “kernel: Route cache is full” was logged so not that surprising.
Searching further I read the route cache was removed in Linux 3.6, so that makes the message even more puzzling,
The route lookup performance papers published by Vincent Bernat are even more interesting – <https://vincent.bernat.ch/en/blog/2017-performance-progression-ipv4-route-lookup-linux>, <https://vincent.bernat.ch/en/blog/2017-performance-progression-ipv6-route-lookup-linux>.
How small – or large – is this machine?
It seems that, the network interface has enough capacity for the traffic but, the internal machine buses may be constrained by a memory limitation …
The third entry is how many times function that allocates dst_entry was called since boot (the error message comes from this function). This happens for every remote destination after kernel has determined route to it. 0x1c2ffe2 == 29556706 which is quite a lot. Of course it depends on uptime. When system was booted (who -b)?
The second last entry is the actual number of dst_entries currently allocated. It is small (0xe == 14), but it is possible that you have some burst of activity at one point.
$ ip -6 route show cache $
That is probably red herring. As far as I can tell those cached entries are created only in two cases - on ICMP route redirect and to override Path MTU value. Neither should happen very often.
The fact is that when this error message is output kernel hits limit on the number of destination entries it maintains. If you do not see it anymore it was probably caused by sudden burst of activity from many different remote systems. We likely never know.
You determined it from the single error message that was provided? Amazing …
Let’s face it –
[INDENT=2]mail services
web services – including a public server
DNS services[/INDENT]
If that message is appearing then, the physical network connection is not the bottleneck.
What’s left?
Given a standard Ethernet cable, the maximum amount of routing requests per second is –
[HR][/HR]If, the CPU is weak and/or there ain’t enough memory for the Kernel’s buffers, it will not be able to serve the number of Ethernet packets per second a 1 Gb/s Ethernet cable is capable of delivering …