Results 1 to 8 of 8

Thread: Redis Sentinel "Read-only file system" in tumbleweed

  1. #1

    Default Redis Sentinel "Read-only file system" in tumbleweed

    At some point on my tumbleweed/apache server with Redis Sentinel stopped working. Redis works fine (I've monitored it), but sentinel logs shows:
    Code:
    Sentinel config file /etc/redis/sentinel-redis-xyz.conf is not writable: Read-only file system. Exiting...
    However, my conf permissions file is as follows:
    Code:
    -rw-rw---- 1 root redis 13958 May  3 11:03 sentinel-redis-xyz.conf
    systemd fails with:
    Code:
    redis-sentinel@redis-xyz.service: Main process exited, code=exited, status=1/FAILURE
    redis-sentinel@redis-xyz.service: Failed with result 'exit-code'.
    Failed to start Redis Sentinel instance: redis-xyz.
    And the systemd service is as follows:
    Code:
    [Unit]
    Description=Redis Sentinel instance: %i
    After=network.target
    PartOf=redis-sentinel.target
    
    
    [Service]
    Type=notify
    User=redis
    Group=redis
    PrivateTmp=true
    # added automatically, for details please see
    # https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
    ProtectSystem=full
    ProtectHome=true
    PrivateDevices=true
    ProtectHostname=true
    ProtectClock=true
    ProtectKernelTunables=true
    ProtectKernelModules=true
    ProtectKernelLogs=true
    ProtectControlGroups=true
    RestrictRealtime=true
    # end of automatic additions 
    PIDFile=/run/redis/sentinel-%i.pid
    ExecStart=/usr/sbin/redis-sentinel /etc/redis/sentinel-%i.conf
    LimitNOFILE=10240
    Restart=on-failure
    
    
    [Install]
    WantedBy=multi-user.target redis.target
    So, I have no idea how to get it back up and working. I kind of suspect it is related to recent systemd hardening, but I have no idea how to troubleshoot beyond what I have already done. I haven't changed the config, but I have been updating with zypper dup regularly.

    Can anyone suggest where to start?

  2. #2
    Join Date
    Jun 2008
    Location
    Netherlands
    Posts
    30,705

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Quote Originally Posted by sunscape View Post
    At some point on my tumbleweed/apache server with Redis Sentinel stopped working. Redis works fine (I've monitored it), but sentinel logs shows:
    Code:
    Sentinel config file /etc/redis/sentinel-redis-xyz.conf is not writable: Read-only file system. Exiting...
    However, my conf permissions file is as follows:
    Code:
    -rw-rw---- 1 root redis 13958 May  3 11:03 sentinel-redis-xyz.conf
    I have no idea what Redis Sentinel is, but above you get a message about a read-only file system and then you check the permissions of a file.

    IMO should check if you have a read-only file system (that contains that file)
    Code:
    mount
    O, and btw, please also include the prompt/command line when you copy/paste. We now have output about the permissions of a file, but we do not know what you did to get it, nor e.g. what the working directory is. Maybe not that important for this time, but better do it always, it may matter much next time.
    Last edited by hcvv; 04-May-2022 at 02:37.
    Henk van Velden

  3. #3

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Sorry, the file permission confirmation was just "ls -l" in the /etc/redis/ directory. I can make new files in this directory so I don't think the file system is ro. I've checked mount and nothing is ro.

  4. #4
    Join Date
    Sep 2012
    Posts
    7,690

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Quote Originally Posted by sunscape View Post
    Code:
    Sentinel config file /etc/redis/sentinel-redis-xyz.conf is not writable: Read-only file system. Exiting...
    ...
    Code:
    [Unit]
    Description=Redis Sentinel instance: %i
    After=network.target
    PartOf=redis-sentinel.target
    
    
    [Service]
    Type=notify
    User=redis
    Group=redis
    PrivateTmp=true
    # added automatically, for details please see
    # https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
    ProtectSystem=full
    If you read "man systemd.exec" and search for ProtectSystem you will see
    Code:
    If set to "full", the /etc/ directory is mounted read-only, too.
    If your application really needs write access to this file/directory, you need to open bug report on openSUSE bugzilla so unit definition is fixed. https://bugzilla.opensuse.org, same user/password as here.

  5. #5

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    So...

    I just commented out all the hardening changes in the redis-sentinel systemd service config, and now it works. So, now the question is which of these changes is responsible and is it needed?

    Code:
    [Unit]
    Description=Redis Sentinel instance: %i
    After=network.target
    PartOf=redis-sentinel.target
    
    
    [Service]
    Type=notify
    User=redis
    Group=redis
    PrivateTmp=true
    # added automatically, for details please see
    # https://en.opensuse.org/openSUSE:Security_Features#Systemd_hardening_effort
    #ProtectSystem=full
    #ProtectHome=true
    #PrivateDevices=true
    #ProtectHostname=true
    #ProtectClock=true
    #ProtectKernelTunables=true
    #ProtectKernelModules=true
    #ProtectKernelLogs=true
    #ProtectControlGroups=true
    #RestrictRealtime=true
    # end of automatic additions 
    PIDFile=/run/redis/sentinel-%i.pid
    ExecStart=/usr/sbin/redis-sentinel /etc/redis/sentinel-%i.conf
    LimitNOFILE=10240
    Restart=on-failure
    
    
    [Install]
    WantedBy=multi-user.target redis.target

  6. #6

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Quote Originally Posted by arvidjaar View Post
    If you read "man systemd.exec" and search for ProtectSystem you will see
    Code:
    If set to "full", the /etc/ directory is mounted read-only, too.
    If your application really needs write access to this file/directory, you need to open bug report on openSUSE bugzilla so unit definition is fixed. https://bugzilla.opensuse.org, same user/password as here.
    Guess that answers my second post while I was posting again...

    That part of the config is auto generated, but I guess it breaks redis-sentinel because the file is written to during service startup. Is this a really important security issue?

  7. #7
    Join Date
    Sep 2012
    Posts
    7,690

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Quote Originally Posted by sunscape View Post
    That part of the config is auto generated
    This is misunderstanding. This part of config has been generated once. From now on it is up to package maintainer to adjust it. It is not as if it will be added to unit definition every time new package is built.

  8. #8

    Default Re: Redis Sentinel "Read-only file system" in tumbleweed

    Quote Originally Posted by arvidjaar View Post
    If you read "man systemd.exec" and search for ProtectSystem you will see
    Code:
    If set to "full", the /etc/ directory is mounted read-only, too.
    If your application really needs write access to this file/directory, you need to open bug report on openSUSE bugzilla so unit definition is fixed. https://bugzilla.opensuse.org, same user/password as here.
    Submitted.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •