Nginx won't start after upgrade from leap 15.3 to 15.4,

Hi,

Just updated leap from 15.3 to 15.4, now nginx won’t start.

#systemctl status nginx.service

× nginx.service - The nginx HTTP and reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; disabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2023-06-21 21:25:56 AEST; 1min 4s ago
    Process: 20713 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=1/FAILURE)

Jun 21 21:25:55 XxXxXxXxXx systemd[1]: Starting The nginx HTTP and reverse proxy server...
Jun 21 21:25:56 XxXxXxXxXx nginx[20713]: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
Jun 21 21:25:56 XxXxXxXxXx nginx[20713]: nginx: [emerg] open() "/usr//on" failed (30: Read-only file system)
Jun 21 21:25:56 XxXxXxXxXx nginx[20713]: nginx: configuration file /etc/nginx/nginx.conf test failed
Jun 21 21:25:56 XxXxXxXxXx systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 21 21:25:56 XxXxXxXxXx systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 21 21:25:56 XxXxXxXxXx systemd[1]: Failed to start The nginx HTTP and reverse proxy server.

From what I gather - nginx: [emerg] open() “/usr//on” failed (30: Read-only file system), is the issue. I have read the notes on OpenSUSE’ page, and the notes from nginx, but I don’t know what actions I need to take. Obviously, I need to change either the nginx settings, or file permissions, but the notes don’t give the information to make a informed decision.

Can anyone help?

Kind regards
Pedro

I would say the very strange filename /usr//on is an issue.

As was already explained in the other old topic that you have read, nginx.service has ProtectSystem=full which makes /usr read-only for any process belonging to this service. I personally do not see any reason for nginx to write to /usr so it sounds more like a bug in your ngnix configuration.

Thank you for your reply #arvidjaar.

Could you please point to that thread?

I found this on github, it implies that “/usr” is hard coded into nginx, and not something I can configure…

Also, running “nginx -t” reports the configuration is okay. My configuration has been fine for a long time, that the upgrade to 15.4 has affect it…

I have no further idea about the subject. But such an approval only tells you that it is syntactically correct. It does not tell you if you have configured on or off features you do like or not, or it is pointing to non-existing files, or files that are impossible to create because of permissions and a lot of other semantics. That is of course up to you.

You cannot remember where you posted several hours ago?

The /usr dir is not hard coded in nginx it is hardcoded into the systemd system. The default configuration of nginx and the systemd start unit for nginx in Leap 15.3/4 does not contain any referrence to the /usr directory. But in the past you or an application that you installed has added something to the nginx settings on this machine that needs read/write permissions in /usr/on. And this is not possible anymore with the enhanced security settings in the nginx.service unit that comes wirh Leap 15.4.
In my opinion you have 2 options:

  1. Find the reason why read/write access to /usr/on is needed in your local config and change this to another directory that is not affected form the security settings. We do not know anything of your local nginx settings so we cannot help you with this option.

  2. Add a drop in to nginx.service unit via
    # systemctl edit nginx
    that is adding /usr/on to option ReadWritePaths as shown below:

### Editing /etc/systemd/system/nginx.service.d/override.conf
### Anything between here and the comment below will become the new contents of the file

[Service]
ReadWritePaths=/usr/on

### Lines below this comment will be discarded
.....
1 Like

Thank you very much for your VERY informative reply @rawar.

I did find in the .conf of a test site I have, the malformed directive:

error_log on;
error_log /something/something/error.log;

Nginx is working now :grinning:

When I have the other issues with 15.4 fixed, I’ll have a look into the new nginx / opensuse security directives…

Kind regards
Pedro