What is a "route cache?"

linux 5.3.18-59.16-default x86_64

I found this in the syslog today, multiple entries:

2021-09-06T09:33:11-0700 sma-server3 kernel: Route cache is full: consider increasing sysctl net.ipv[4|6].route.max_size. 

And the results are:

$ sysctl net.ipv4.route.max_size 
net.ipv4.route.max_size = 2147483647

$ sudo sysctl net.ipv6.route.max_size 
net.ipv6.route.max_size = 4096

Such an interesting suggestion has never appeared there before.
Would someone explain what it is regarding?

@jimoe666:

Are you routing between two interfaces?

  • Can you show us the output of “ip route list
    ” ?

No.

Can you show us the output of “ip route list” ?

Yes.

$ ip route list 
default via 192.168.69.1 dev eth0  
192.168.69.0/24 dev eth0 proto kernel scope link src 192.168.69.246

Hi
Are you using any ipv6 at all? If not then suggest disabling (YaST -> Network Settings -> General tab), if you do then might want to tweak postfixes main.cf inet_protocol to only be ipv4 rather than all.

I note that:

IPv6 is enabled for the hosts in our network. We have an assigned IPv6 block (fd2f:4760:521f:3f3c::/64), and have given each host an address. That is the extent of the explicit use of IPv6.

I discovered the IPv6 command for the route listing:

$ ip -6 route list 
::1 dev lo proto kernel metric 256 pref medium 
fd2f:4760:521f:3f3c::/64 dev eth0 proto kernel metric 256 pref medium 
fe80::/64 dev eth0 proto kernel metric 256 pref medium 
default via fd2f:4760:521f:3f3c::c0a8:4501 dev eth0 metric 1024 pref medium 
default via fe80::2eb8:edff:fe5a:9d44 dev eth0 proto ra metric 1024 expires 1717sec hoplimit 64 pref medium 

I do not understand why you suggested changing the configuration for postfix.

This is cache of per-destination entries. When Linux kernel resolves route to destination it puts it in cache for future use. AFAIK it is used only for IPv6 today, no more for IPv4. This case is unrelated to “routing between interfaces”. Output of

cat /proc/net/rt6_stats
ip -6 route show cache

when this happens would be interesting.

$ sudo sysctl net.ipv6.route.max_size 
net.ipv6.route.max_size = 4096

Pragmatic answer - if you hit this you may want to increase this limit. But of course it would be useful to (at least, try to) understand why you have so many entries. What does this host do? Is it Web/Mail or like server? Are you using torrents (or some other peer to peer solutions)? How may concurrent connections do you usually have?

IPv6 route lookup on Linux

$ cat /proc/net/rt6_stats 
001a 000f 1c2ffe2 0011 0000 000e 0020 

$ ip -6 route show cache 
$  

Pragmatic answer - if you hit this you may want to increase this limit.

I did that, to 8192. Although this warning had never occurred before, and has not since.

What does this host do? Is it Web/Mail or like server?

It provides mail services for our business.
It offers web services for the local network. A separate service handles the public website.
It offers DNS services locally and for our corporate domain.

Are you using torrents (or some other peer to peer solutions)?

No.

How may concurrent connections do you usually have?
I do not know. Not many? It is a small business.

Probably good to dump the route cache and see what is in it:

ip -s route show cache

What is “sudo sysctl net.ipv4.ip_forward” giving you?

$  ip -s route show cache
$

What is “sudo sysctl net.ipv4.ip_forward” giving you?

$  sudo sysctl net.ipv4.ip_forward 
net.ipv4.ip_forward = 0
$

No forwarding enabled and the cache is also empty.

Of coarse the command is not given at the moment the “kernel: Route cache is full” was logged so not that surprising.

Searching further I read the route cache was removed in Linux 3.6, so that makes the message even more puzzling,

The route lookup performance papers published by Vincent Bernat are even more interesting – <https://vincent.bernat.ch/en/blog/2017-performance-progression-ipv4-route-lookup-linux&gt;, <https://vincent.bernat.ch/en/blog/2017-performance-progression-ipv6-route-lookup-linux&gt;.

How small – or large – is this machine?

  • In particular, how much memory does the thing have?

It seems that, the network interface has enough capacity for the traffic but, the internal machine buses may be constrained by a memory limitation …

The third entry is how many times function that allocates dst_entry was called since boot (the error message comes from this function). This happens for every remote destination after kernel has determined route to it. 0x1c2ffe2 == 29556706 which is quite a lot. Of course it depends on uptime. When system was booted (who -b)?

The second last entry is the actual number of dst_entries currently allocated. It is small (0xe == 14), but it is possible that you have some burst of activity at one point.

$ ip -6 route show cache 
$  

That is probably red herring. As far as I can tell those cached entries are created only in two cases - on ICMP route redirect and to override Path MTU value. Neither should happen very often.

The fact is that when this error message is output kernel hits limit on the number of destination entries it maintains. If you do not see it anymore it was probably caused by sudden burst of activity from many different remote systems. We likely never know.

You determined it from the single error message that was provided? Amazing …

Let’s face it –

  • The machine is supporting –

[INDENT=2]mail services
web services – including a public server
DNS services[/INDENT]

  • It has only one network interface.

If that message is appearing then, the physical network connection is not the bottleneck.

What’s left?

  • Nothing – the routing being performed by the Kernel is hitting a limit which can only be somewhere in the system queues.
  • The figures provided by Vincent Bernat are showing that, the Kernel’s lookup time of IPv4 routing is somewhat less than 40 ns and, the Kernel’s lookup time of IPv6 routing is less than 500 ns – given enough hardware …
  • In other words, the Kernel can handle 25 million IPv4 routing lookups per second and, 2 million IPv6 routing lookups per second.

Given a standard Ethernet cable, the maximum amount of routing requests per second is –

  • 1 Gb/s == 125 million bytes per second == (given 84 bytes per Ethernet packet) 1.488095 million Ethernet packets per second

[HR][/HR]If, the CPU is weak and/or there ain’t enough memory for the Kernel’s buffers, it will not be able to serve the number of Ethernet packets per second a 1 Gb/s Ethernet cable is capable of delivering …