Redis Sentinel "Read-only file system" in tumbleweed

At some point on my tumbleweed/apache server with Redis Sentinel stopped working. Redis works fine (I’ve monitored it), but sentinel logs shows:


Sentinel config file /etc/redis/sentinel-redis-xyz.conf is not writable: Read-only file system. Exiting...

However, my conf permissions file is as follows:


-rw-rw---- 1 root redis 13958 May  3 11:03 sentinel-redis-xyz.conf

systemd fails with:


redis-sentinel@redis-xyz.service: Main process exited, code=exited, status=1/FAILURE
redis-sentinel@redis-xyz.service: Failed with result 'exit-code'.
Failed to start Redis Sentinel instance: redis-xyz.

And the systemd service is as follows:


[Unit]
Description=Redis Sentinel instance: %i
After=network.target
PartOf=redis-sentinel.target


[Service]
Type=notify
User=redis
Group=redis
PrivateTmp=true
# added automatically, for details please see
# https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
ProtectSystem=full
ProtectHome=true
PrivateDevices=true
ProtectHostname=true
ProtectClock=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
RestrictRealtime=true
# end of automatic additions 
PIDFile=/run/redis/sentinel-%i.pid
ExecStart=/usr/sbin/redis-sentinel /etc/redis/sentinel-%i.conf
LimitNOFILE=10240
Restart=on-failure


[Install]
WantedBy=multi-user.target redis.target

So, I have no idea how to get it back up and working. I kind of suspect it is related to recent systemd hardening, but I have no idea how to troubleshoot beyond what I have already done. I haven’t changed the config, but I have been updating with zypper dup regularly.

Can anyone suggest where to start?

I have no idea what Redis Sentinel is, but above you get a message about a read-only file system and then you check the permissions of a file.

IMO should check if you have a read-only file system (that contains that file)

mount

O, and btw, please also include the prompt/command line when you copy/paste. We now have output about the permissions of a file, but we do not know what you did to get it, nor e.g. what the working directory is. Maybe not that important for this time, but better do it always, it may matter much next time.

Sorry, the file permission confirmation was just “ls -l” in the /etc/redis/ directory. I can make new files in this directory so I don’t think the file system is ro. I’ve checked mount and nothing is ro.

If you read “man systemd.exec” and search for ProtectSystem you will see

If set to "**full**", **the /etc/ directory is mounted read-only**, too.

If your application really needs write access to this file/directory, you need to open bug report on openSUSE bugzilla so unit definition is fixed. https://bugzilla.opensuse.org, same user/password as here.

So…

I just commented out all the hardening changes in the redis-sentinel systemd service config, and now it works. So, now the question is which of these changes is responsible and is it needed?


[Unit]
Description=Redis Sentinel instance: %i
After=network.target
PartOf=redis-sentinel.target


[Service]
Type=notify
User=redis
Group=redis
PrivateTmp=true
# added automatically, for details please see
# https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
#ProtectSystem=full
#ProtectHome=true
#PrivateDevices=true
#ProtectHostname=true
#ProtectClock=true
#ProtectKernelTunables=true
#ProtectKernelModules=true
#ProtectKernelLogs=true
#ProtectControlGroups=true
#RestrictRealtime=true
# end of automatic additions 
PIDFile=/run/redis/sentinel-%i.pid
ExecStart=/usr/sbin/redis-sentinel /etc/redis/sentinel-%i.conf
LimitNOFILE=10240
Restart=on-failure


[Install]
WantedBy=multi-user.target redis.target

Guess that answers my second post while I was posting again…

That part of the config is auto generated, but I guess it breaks redis-sentinel because the file is written to during service startup. Is this a really important security issue?

This is misunderstanding. This part of config has been generated once. From now on it is up to package maintainer to adjust it. It is not as if it will be added to unit definition every time new package is built.

Submitted.