Varnish 3.x / 4.x 503 / backend_fail [Help for those who run into it]

This isn’t really a help request more like “This is what bay be causing the problem and how to fix it”:

I’ve been running Varnish for quite some time now and on openSUSE 13.x or similar systems where systemd is used, the global limits set in /etc/security/limits.conf no longer apply.

So on a busy Varnish frontend server with hundreds to thousands (or more) connections per second you will run into the issue of file limits, changing them the way they used to be done by editing limits.conf and setting soft/hard nofile xxxxx will have no effect and the end result is that you’ll run into an issue where your Varnish installation will start raining 503’s and they will show up as “backend fail” in your logs / varnishstat.

Now, to get around this issue you’ll need to edit: /usr/lib/systemd/system/varnish.service and in the [Service] part add: LimitNOFILE=32768 , this would raise the Varnish limit to 32768 files (which is a lot). This alone may not be enough, so raising the fs.file-max value for the OS itself may be needed, you can do this by adding fs.file-max=1000000 to /etc/sysctl.conf and issuing sysctl -p

Note: If you have hundreds or thousands of Apache processes running (like I do) you will need to apply the same systemd fix to your apache2.service.

Remember to restart services after you’ve applied the fix.