Strange library search path issue

Issue:

$ sudo cp systemd-journal-remote /usr/lib/systemd/systemd-journal-remote
$ cd /usr/lib/systemd
$ ./systemd-journal-remote --help
./systemd-journal-remote: error while loading shared libraries: libsystemd-shared-249.so: cannot open shared object file: No such file or directory

Analysis (blank lines inserted between commands for better readability):

$ ls -al libsystemd-shared-249.so
-rwxr-xr-x 1 root root 2967160 Feb 14 12:58 libsystemd-shared-249.so

$  ls -al systemd-journal-remote
-rwxr-xr-x 1 root root 188744 Apr 23 21:18 systemd-journal-remote

$ file systemd-journal-remote
systemd-journal-remote: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=60cdeb73cd86a000a4f73ade46ebf6bfee739e6f, for GNU/Linux 3.2.0, with debug_info, not stripped

$ readelf -d systemd-journal-remote | egrep 'R.*PATH'
 0x000000000000001d (RUNPATH)            Library runpath: [/lib/systemd]

$ echo \<$LD_LIBRARY_PATH\>
<>

$ sudo ldconfig -p | grep libsystemd-shared
(no output, library is not in the file /etc/ld.so.cache)

$ LD_DEBUG=libs ./systemd-journal-remote --help
     51003:     find library=libsystemd-shared-249.so [0]; searching
     51003:      search path=/lib/systemd/tls/x86_64/x86_64:/lib/systemd/tls/x86_64:/lib/systemd/tls/x86_64:/lib/systemd/tls:/lib/systemd/x86_64/x86_64:/lib/systemd/x86_64:/lib/systemd/x86_64:/lib/systemd           (RUNPATH from file ./systemd-journal-remote)
     51003:       trying file=/lib/systemd/tls/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/tls/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib/systemd/libsystemd-shared-249.so
     51003:      search cache=/etc/ld.so.cache
     51003:      search path=/lib64/tls/x86_64/x86_64:/lib64/tls/x86_64:/lib64/tls/x86_64:/lib64/tls:/lib64/x86_64/x86_64:/lib64/x86_64:/lib64/x86_64:/lib64:/usr/lib64/tls/x86_64/x86_64:/usr/lib64/tls/x86_64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64/x86_64:/usr/lib64/x86_64:/usr/lib64/x86_64:/usr/lib64          (system search path)
     51003:       trying file=/lib64/tls/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/tls/libsystemd-shared-249.so
     51003:       trying file=/lib64/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/lib64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/tls/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/tls/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/tls/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/x86_64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/x86_64/libsystemd-shared-249.so
     51003:       trying file=/usr/lib64/libsystemd-shared-249.so
     51003:
./systemd-journal-remote: error while loading shared libraries: libsystemd-shared-249.so: cannot open shared object file: No such file or directory

Question:
How does the loader get these strange search paths, where do they come from?

Note:
It is understood that the real issue most likely is with RUNPATH, this should be /usr/lib/systemd, not /lib/systemd. However, the same strange search path is applied for binaries contained in the distribution, just run an existing command like systemd-makefs with LD_DEBUG=libs prepended to see.

@duise:

I’m having more than a few issues understanding what you’re doing:

  • The “systemd-journal-remote” package provides a systemd service to achieve the following:

This extends the journal functionality to keep a copy of logs on a remote server by providing programs to forward journal entries over the network, using encrypted HTTP, and to write journal files from serialized journal contents.

Looking at the man page and the serverfault note, I fail to understand why, you feel that, you have to copy the executable file named “systemd-journal-remote” to any place other than that where, the package had placed the binary file …
<https://www.man7.org/linux/man-pages/man8/systemd-journal-remote.8.html>
<https://serverfault.com/questions/758244/how-to-configure-systemd-journal-remote>


Perhaps, you could detail exactly what you’re hoping to achieve –

  • Please …

I think this one:

$ ./systemd-journal-remote --help
./systemd-journal-remote: error while loading shared libraries: libsystemd-shared-249.so: cannot open shared object file: No such file or directory

Same when starting for example audacity the first time after reboot or system start.
I do then:

/sbin/ldconfig -v /usr/lib64/audacity

Dumping some random commands without a single comment is not the best way to ask questions. You are expected to at least describe why you are using these commands and what result you wanted to achieve.

You just replaced file provided by openSUSE package with something. You never explained where this something (your systemd-journal-remote) comes from, why you are copying it and what you expect from these actions. But the obvious answer is - you have to build your program using the correct options so it can find what it needs.

Try find / -name systemd-journal-remote on your system and you’ll see. This systemd program which is needed for remote journal transfer is missing in Leap 15.4. BTW, the same applies for lots of other parts of systemd. systemd-journal-remote is only one of 18 programs missing in the directory /usr/lib/systemd:

systemd-coredump
systemd-export
systemd-homed
systemd-homework
systemd-import
systemd-importd
systemd-import-fs
systemd-journal-gatewayd
systemd-journal-remote
systemd-journal-upload
systemd-machined
systemd-networkd
systemd-networkd-wait-online
systemd-oomd
systemd-portabled
systemd-pull
systemd-userdbd
systemd-userwork

None of these programs is there, they are all missing.

So I thought it would be a good idea to download the tarball from Github and compile the whole systemd distribution to get all the missing pieces.

But the question that I raised here is completely independent. As pointed out, you can do the very same search path analyis on an existing program and you’ll see that the loader is always showing this (IMO) crappy behaviour.

search path=/lib/systemd/tls/x86_64/x86_64:/lib/systemd/tls/x86_64 \
:/lib/systemd/tls/x86_64:/lib/systemd/tls:/lib/systemd/x86_64/x86_64 \
:/lib/systemd/x86_64:/lib/systemd/x86_64:/lib/systemd

Most of these directories are not existing!

LD_DEBUG=libs program name

on any existing program in /usr/lib/systemd will output:

search path=/usr/lib/systemd/tls/x86_64/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls:/usr/lib/systemd/x86_64/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd

Most of these directories are not existing!

$ ls /usr/lib/systemd/tls
ls: cannot access '/usr/lib/systemd/tls': No such file or directory
$ ls /usr/lib/systemd/x86_64
-bash: /usr/lib/systemd/x86_64: No such file or directory

THAT was the issue that I reported and questioned, not the missing systemd parts.

Nope. systemd-journal-remote is available in the Leap 15.4 OSS repo. Did you even try to use yast software or zypper to search the package? The “find” command is not suitable to search for packages which are not installed…
So you try to find a package (with the wrong tool) which you did not even have installed…ok…

OK, you are right. But this has nothing to do with the issue I reported.

By compiling the systemd stack yourself you broke your system in some way as can be seen at the output of the first command. On an untinkered Leap 15.4 with the builtin systemd packages your commands show valid output…

leap@leaptest:/usr/lib/systemd> ./systemd-journal-remote --help
systemd-journal-remote [OPTIONS...] {FILE|-}...

Write external journal events to journal file(s).

  -h --help                 Show this help
     --version              Show package version
     --url=URL              Read events from systemd-journal-gatewayd at URL
     --getter=COMMAND       Read events from the output of COMMAND
     --listen-raw=ADDR      Listen for connections at ADDR
     --listen-http=ADDR     Listen for HTTP connections at ADDR
     --listen-https=ADDR    Listen for HTTPS connections at ADDR
  -o --output=FILE|DIR      Write output to FILE or DIR/external-*.journal
     --compress[=BOOL]      Use compression in the output journal (default: yes)
     --seal[=BOOL]          Use event sealing (default: no)
     --key=FILENAME         SSL key in PEM format (default:
                            "/etc/pki/systemd/private/journal-remote.pem")
     --cert=FILENAME        SSL certificate in PEM format (default:
                            "/etc/pki/systemd/certs/journal-remote.pem")
     --trust=FILENAME|all   SSL CA certificate or disable checking (default:
                            "/etc/pki/systemd/ca/trusted.pem")
     --gnutls-log=CATEGORY...
                            Specify a list of gnutls logging categories
     --split-mode=none|host How many output files to create

Note: file descriptors from sd_listen_fds() will be consumed, too.

See the systemd-journal-remote.service(8) man page for details.
leap@leaptest:/usr/lib/systemd> 

leap@leaptest:/usr/lib/systemd> LD_DEBUG=libs ./systemd-journal-remote --help
     10679:     find library=libsystemd-shared-249.so [0]; searching
     10679:      search path=/usr/lib/systemd/tls/x86_64/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls:/usr/lib/systemd/x86_64/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd            (RUNPATH from file ./systemd-journal-remote)
     10679:       trying file=/usr/lib/systemd/tls/x86_64/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/tls/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/tls/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/tls/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/x86_64/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/x86_64/libsystemd-shared-249.so
     10679:       trying file=/usr/lib/systemd/libsystemd-shared-249.so
     10679:
     10679:     find library=libmicrohttpd.so.12 [0]; searching
     10679:      search path=/usr/lib/systemd           (RUNPATH from file ./systemd-journal-remote)
     10679:       trying file=/usr/lib/systemd/libmicrohttpd.so.12
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libmicrohttpd.so.12
     10679:
     10679:     find library=libgnutls.so.30 [0]; searching
     10679:      search path=/usr/lib/systemd           (RUNPATH from file ./systemd-journal-remote)
     10679:       trying file=/usr/lib/systemd/libgnutls.so.30
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libgnutls.so.30
     10679:
     10679:     find library=libpthread.so.0 [0]; searching
     10679:      search path=/usr/lib/systemd           (RUNPATH from file ./systemd-journal-remote)
     10679:       trying file=/usr/lib/systemd/libpthread.so.0
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libpthread.so.0
     10679:
     10679:     find library=libc.so.6 [0]; searching
     10679:      search path=/usr/lib/systemd           (RUNPATH from file ./systemd-journal-remote)
     10679:       trying file=/usr/lib/systemd/libc.so.6
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libc.so.6
     10679:
     10679:     find library=libacl.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libacl.so.1
     10679:
     10679:     find library=libblkid.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libblkid.so.1
     10679:
     10679:     find library=libcap.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libcap.so.2
     10679:
     10679:     find library=libcrypt.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libcrypt.so.1
     10679:
     10679:     find library=libdl.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libdl.so.2
     10679:
     10679:     find library=libgcrypt.so.20 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libgcrypt.so.20
     10679:
     10679:     find library=libip4tc.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libip4tc.so.2
     10679:
     10679:     find library=libkmod.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libkmod.so.2
     10679:
     10679:     find library=liblz4.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/liblz4.so.1
     10679:
     10679:     find library=libmount.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libmount.so.1
     10679:
     10679:     find library=libcrypto.so.1.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libcrypto.so.1.1
     10679:
     10679:     find library=libp11-kit.so.0 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libp11-kit.so.0
     10679:
     10679:     find library=libpam.so.0 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libpam.so.0
     10679:
     10679:     find library=librt.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/librt.so.1
     10679:
     10679:     find library=libseccomp.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libseccomp.so.2
     10679:
     10679:     find library=libselinux.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libselinux.so.1
     10679:
     10679:     find library=libzstd.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libzstd.so.1
     10679:
     10679:     find library=liblzma.so.5 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/liblzma.so.5
     10679:
     10679:     find library=libjitterentropy.so.3 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libjitterentropy.so.3
     10679:
     10679:     find library=libidn2.so.0 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libidn2.so.0
     10679:
     10679:     find library=libunistring.so.2 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libunistring.so.2
     10679:
     10679:     find library=libtasn1.so.6 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libtasn1.so.6
     10679:
     10679:     find library=libnettle.so.8 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libnettle.so.8
     10679:
     10679:     find library=libhogweed.so.6 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libhogweed.so.6
     10679:
     10679:     find library=libgmp.so.10 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libgmp.so.10
     10679:
     10679:     find library=libattr.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libattr.so.1
     10679:
     10679:     find library=libgpg-error.so.0 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libgpg-error.so.0
     10679:
     10679:     find library=libz.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/lib64/libz.so.1
     10679:
     10679:     find library=libffi.so.7 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libffi.so.7
     10679:
     10679:     find library=libaudit.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libaudit.so.1
     10679:
     10679:     find library=libpcre.so.1 [0]; searching
     10679:      search cache=/etc/ld.so.cache
     10679:       trying file=/usr/lib64/libpcre.so.1
     10679:
     10679:
     10679:     calling init: /lib64/libpthread.so.0
     10679:
     10679:
     10679:     calling init: /lib64/libc.so.6
     10679:
     10679:
     10679:     calling init: /usr/lib64/libpcre.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libaudit.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libffi.so.7
     10679:
     10679:
     10679:     calling init: /lib64/libz.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libgpg-error.so.0
     10679:
     10679:
     10679:     calling init: /lib64/libattr.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libgmp.so.10
     10679:
     10679:
     10679:     calling init: /usr/lib64/libnettle.so.8
     10679:
     10679:
     10679:     calling init: /usr/lib64/libhogweed.so.6
     10679:
     10679:
     10679:     calling init: /usr/lib64/libtasn1.so.6
     10679:
     10679:
     10679:     calling init: /usr/lib64/libunistring.so.2
     10679:
     10679:
     10679:     calling init: /usr/lib64/libidn2.so.0
     10679:
     10679:
     10679:     calling init: /usr/lib64/libjitterentropy.so.3
     10679:
     10679:
     10679:     calling init: /usr/lib64/liblzma.so.5
     10679:
     10679:
     10679:     calling init: /usr/lib64/libzstd.so.1
     10679:
     10679:
     10679:     calling init: /lib64/libdl.so.2
     10679:
     10679:
     10679:     calling init: /lib64/libselinux.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libseccomp.so.2
     10679:
     10679:
     10679:     calling init: /lib64/librt.so.1
     10679:
     10679:
     10679:     calling init: /lib64/libpam.so.0
     10679:
     10679:
     10679:     calling init: /usr/lib64/libp11-kit.so.0
     10679:
     10679:
     10679:     calling init: /usr/lib64/libcrypto.so.1.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libblkid.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libmount.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/liblz4.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libkmod.so.2
     10679:
     10679:
     10679:     calling init: /usr/lib64/libip4tc.so.2
     10679:
     10679:
     10679:     calling init: /usr/lib64/libgcrypt.so.20
     10679:
     10679:
     10679:     calling init: /usr/lib64/libcrypt.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libcap.so.2
     10679:
     10679:
     10679:     calling init: /lib64/libacl.so.1
     10679:
     10679:
     10679:     calling init: /usr/lib64/libgnutls.so.30
     10679:
     10679:
     10679:     calling init: /usr/lib64/libmicrohttpd.so.12
     10679:
     10679:
     10679:     calling init: /usr/lib/systemd/libsystemd-shared-249.so
     10679:
     10679:
     10679:     initialize program: ./systemd-journal-remote
     10679:
     10679:
     10679:     transferring control: ./systemd-journal-remote
     10679:
systemd-journal-remote [OPTIONS...] {FILE|-}...

Write external journal events to journal file(s).

  -h --help                 Show this help
     --version              Show package version
     --url=URL              Read events from systemd-journal-gatewayd at URL
     --getter=COMMAND       Read events from the output of COMMAND
     --listen-raw=ADDR      Listen for connections at ADDR
     --listen-http=ADDR     Listen for HTTP connections at ADDR
     --listen-https=ADDR    Listen for HTTPS connections at ADDR
  -o --output=FILE|DIR      Write output to FILE or DIR/external-*.journal
     --compress[=BOOL]      Use compression in the output journal (default: yes)
     --seal[=BOOL]          Use event sealing (default: no)
     --key=FILENAME         SSL key in PEM format (default:
                            "/etc/pki/systemd/private/journal-remote.pem")
     --cert=FILENAME        SSL certificate in PEM format (default:
                            "/etc/pki/systemd/certs/journal-remote.pem")
     --trust=FILENAME|all   SSL CA certificate or disable checking (default:
                            "/etc/pki/systemd/ca/trusted.pem")
     --gnutls-log=CATEGORY...
                            Specify a list of gnutls logging categories
     --split-mode=none|host How many output files to create

Note: file descriptors from sd_listen_fds() will be consumed, too.

See the systemd-journal-remote.service(8) man page for details.
     10679:
     10679:     calling fini: ./systemd-journal-remote [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib/systemd/libsystemd-shared-249.so [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libmicrohttpd.so.12 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libgnutls.so.30 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libacl.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libcap.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libcrypt.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libgcrypt.so.20 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libip4tc.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libkmod.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/liblz4.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libmount.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libblkid.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libcrypto.so.1.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libp11-kit.so.0 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libpam.so.0 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/librt.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libseccomp.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libselinux.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libdl.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libzstd.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/liblzma.so.5 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libjitterentropy.so.3 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libpthread.so.0 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libidn2.so.0 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libunistring.so.2 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libtasn1.so.6 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libhogweed.so.6 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libnettle.so.8 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libgmp.so.10 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libattr.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libgpg-error.so.0 [0]
     10679:
     10679:
     10679:     calling fini: /lib64/libz.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libffi.so.7 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libaudit.so.1 [0]
     10679:
     10679:
     10679:     calling fini: /usr/lib64/libpcre.so.1 [0]
     10679:
leap@leaptest:/usr/lib/systemd> 

Sorry, but it seems you understood nothing :unamused:

  1. I broke nothing. (As I haven’t installed anything but the manul copy of one file).
  2. The output of the first command is irrelevant to the real issue.
  3. The real issue is that the default shared object loader uses a search path with lots of non-existing directories.
  4. You can verify the issue ON YOUR SYSTEM where “nothing is broken” by running this command:
LD_DEBUG=libs /usr/lib/systemd/systemd-makefs 2>&1 | grep RUNPATH

Expected output on Leap 15.4:

 65706:      search path=/usr/lib/systemd/tls/x86_64/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls/x86_64:/usr/lib/systemd/tls:/usr/lib/systemd/x86_64/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd/x86_64:/usr/lib/systemd           (RUNPATH from file /usr/lib/systemd/systemd-makefs)
     65706:      search path=/usr/lib/systemd           (RUNPATH from file /usr/lib/systemd/systemd-makefs)

Whatever floats your boat…

The search path is valid.

But your system is broken, as can be seen on many of your outputs from your first post…

Thank you for your first real contribution here.

Quote:

The GLIBC loader takes RPATH from your executable, and appends PLATFORM strings (tls/x86_64:tls:x86_64 here) to it to construct the full search path.

Once the loader discovers that e.g. RPATH/tls directory does not exist, it removes that entry from the search path (there is no point in repeatedly searching non-existent directory). Eventually you end up with just RPATH itself being present.

So I’d understand this such that when I run the command the 2nd time, the non-existing entries should have been removed from the search path. This is not the case, I get exact the same output as before. The statement from the linked stackoverflow question seems to be false.

there is no point in repeatedly searching non-existent directory” Word!

Thanks for the flowers :wink:
You wasn’t able to find this yourself?

https://man7.org/linux/man-pages/man8/ld.so.8.html

The ld.so(8) man page does not provide any info on the matter.

You could shut down the forums because of inactivity if everybody always did intensive Web search with the correct search terms before posting here. If you want to blame me because I didn’t google “/tls/x86_64/x86_64” or similar, fine.

However, the provided answer in the stackoverflow question seems to be false which turns my question into a “valid” forums issue, I think. :wink:

Well, if you read it you will understand. It explains that $ORIGIN, $LIB, and $PLATFORM (tls, x86_64 and so on) get appended to the search path.

And LD_DEBUG=libs does nothing more than “Display library search paths”.

So this explains exactly the output of “LD_DEBUG=libs /usr/lib/systemd/systemd-makefs 2>&1 | grep RUNPATH”

1 Like

Excerpt from the ld.so(8) man page:

   $ORIGIN (or equivalently ${ORIGIN})
          This expands to the directory containing the program or
          shared object.  

   $LIB (or equivalently ${LIB})
          This expands to lib or lib64 depending on the architecture
          (e.g., on x86-64, it expands to lib64 and on x86-32, it
          expands to lib).

   $PLATFORM (or equivalently ${PLATFORM})
          This expands to a string corresponding to the processor
          type of the host system (e.g., "x86_64"). 

the directory”, “lib or lib64”, “x86_64” - singular, not plural.

Search path:

/usr/lib/systemd/tls/x86_64/x86_64
/usr/lib/systemd/tls/x86_64
/usr/lib/systemd/tls/x86_64
/usr/lib/systemd/tls
/usr/lib/systemd/x86_64/x86_64
/usr/lib/systemd/x86_64
/usr/lib/systemd/x86_64
/usr/lib/systemd

$ORIGIN='/usr/lib/systemd'
$LIB=*whatever*
$PLATFORM='x86_64'

So if rpath was given as $ORIGIN/$PLATFORM, we’d get /usr/lib/systemd/x86_64

But where do the other seven directories come from?

In particular, who would use crap like $PLATFORM/$PLATFORM like in /x86_64/x86_64 ?!

And where does ‘tls’ and/or not ‘tls’ come from?

It gets even more confusing:

/usr/bin/ld is a symlink to /etc/alternatives/ld which is a symlink to /usr/bin/ld.bfd.

So I looked up the man page for GNU ld.bfd and found that GNU ld support the -L searchdir and --library-path=searchdir options:

Add path searchdir to the list of paths that ld will search for archive libraries and ld control scripts. You may use this option any number of times. …

The default set of paths searched (without being specified with -L) depends on which emulation mode ld is using, and in some cases also on how it was configured.

The paths can also be specified in a link script with the “SEARCH_DIR” command. Directories specified this way are searched at the point in which the linker script appears in the command line.

On another page on linker scripts I found this info:

The linker always uses a linker script. If you do not supply one yourself, the linker will use a default script that is compiled into the linker executable. You can use the ` --verbose ’ command line option to display the default linker script.

$ ld --verbose
GNU ld (GNU Binutils; SUSE Linux Enterprise 15) 2.39.0.20220810-150100.7.40
...
using internal linker script:
==================================================
/* Script for -z combreloc */
/* Copyright (C) 2014-2022 Free Software Foundation, Inc.
   Copying and distribution of this script, with or without modification,
   are permitted in any medium without royalty provided the copyright
   notice and this notice are preserved.  */
OUTPUT_FORMAT("elf64-x86-64", "elf64-x86-64",
              "elf64-x86-64")
OUTPUT_ARCH(i386:x86-64)
ENTRY(_start)
SEARCH_DIR("/usr/x86_64-suse-linux/lib64"); SEARCH_DIR("/usr/lib64"); SEARCH_DIR("/usr/local/lib64"); SEARCH_DIR("/lib64"); SEARCH_DIR("/usr/x86_64-suse-linux/lib"); SEARCH_DIR("/usr/local/lib"); SEARCH_DIR("/lib"); SEARCH_DIR("/usr/lib");

using internal linker script” does not make sense as the output that we get with LD_DEBUG is quite different.

Using rpm -ql binutils I could see that the package comes with a directory full of (default) linker scripts in /usr/lib64/ldscripts. However, none of them contain “tls” as you can easily verify:

$ cd /usr/lib64/ldscripts
$ grep SEARCH_DIR *.* | grep tls

So the initial question is still not answered.

@duise:

Please note:

 # LANG=C zypper info systemd-journal-remote
systemd-journal-remote            systemd-journal-remote-debuginfo  
 # LANG=C zypper info systemd-journal-remote
Loading repository data...
Reading installed packages...


Information for package systemd-journal-remote:
-----------------------------------------------
Repository     : Update repository with updates from SUSE Linux Enterprise 15
Name           : systemd-journal-remote
Version        : 249.16-150400.8.25.7
Arch           : x86_64
Vendor         : SUSE LLC <https://www.suse.com/>
Installed Size : 177.0 KiB
Installed      : No
Status         : not installed
Source package : systemd-249.16-150400.8.25.7.src
Upstream URL   : http://www.freedesktop.org/wiki/Software/systemd
Summary        : Gateway for serving journal events over the network using HTTP
Description    : 
    This extends the journal functionality to keep a copy of logs on a
    remote server by providing programs to forward journal entries over
    the network, using encrypted HTTP, and to write journal files from
    serialized journal contents.

    This package contains systemd-journal-gatewayd,
    systemd-journal-remote, and systemd-journal-upload.

 # 
 # LANG=C zypper install systemd-journal-remote
Loading repository data...
Reading installed packages...
Resolving package dependencies...

The following 2 NEW packages are going to be installed:
  libmicrohttpd12 systemd-journal-remote

2 new packages to install.
Overall download size: 364.9 KiB. Already cached: 0 B. After the operation, additional 322.1 KiB will be used.
Continue? [y/n/v/...? shows all options] (y): 
Retrieving: libmicrohttpd12-0.9.57-150000.3.3.1.x86_64 (Update repository with updates from SUSE Linux Enterprise 15)
                                                                                                    (1/2),  73.9 KiB    
Retrieving: libmicrohttpd12-0.9.57-150000.3.3.1.x86_64.rpm ...........................................[done (2.5 KiB/s)]
Retrieving: systemd-journal-remote-249.16-150400.8.25.7.x86_64 (Update repository with updates from SUSE Linux
Enterprise 15)
                                                                                                    (2/2), 291.0 KiB    
Retrieving: systemd-journal-remote-249.16-150400.8.25.7.x86_64.rpm ...................................[done (1.1 KiB/s)]

Checking for file conflicts: .....................................................................................[done]
(1/2) Installing: libmicrohttpd12-0.9.57-150000.3.3.1.x86_64 .....................................................[done]
Creating group systemd-journal-remote with gid 445.
Creating user systemd-journal-remote (systemd Journal Remote) with uid 445 and gid 445.
(2/2) Installing: systemd-journal-remote-249.16-150400.8.25.7.x86_64 .............................................[done]
 # 
 # which -a systemd-journal-remote
which: no systemd-journal-remote in (/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin)
 #

And, with a “normal” user:

 > rpm -ql systemd-journal-remote 
/etc/systemd/journal-remote.conf
/etc/systemd/journal-upload.conf
/usr/lib/systemd/system/systemd-journal-gatewayd.service
/usr/lib/systemd/system/systemd-journal-gatewayd.socket
/usr/lib/systemd/system/systemd-journal-remote.service
/usr/lib/systemd/system/systemd-journal-remote.socket
/usr/lib/systemd/system/systemd-journal-upload.service
/usr/lib/systemd/systemd-journal-gatewayd
/usr/lib/systemd/systemd-journal-remote
/usr/lib/systemd/systemd-journal-upload
/usr/lib/sysusers.d/systemd-remote.conf
/usr/share/man/man5/journal-remote.conf.5.gz
/usr/share/man/man5/journal-remote.conf.d.5.gz
/usr/share/man/man5/journal-upload.conf.5.gz
/usr/share/man/man5/journal-upload.conf.d.5.gz
/usr/share/man/man8/systemd-journal-gatewayd.8.gz
/usr/share/man/man8/systemd-journal-gatewayd.service.8.gz
/usr/share/man/man8/systemd-journal-gatewayd.socket.8.gz
/usr/share/man/man8/systemd-journal-remote.8.gz
/usr/share/man/man8/systemd-journal-remote.service.8.gz
/usr/share/man/man8/systemd-journal-remote.socket.8.gz
/usr/share/man/man8/systemd-journal-upload.8.gz
/usr/share/man/man8/systemd-journal-upload.service.8.gz
/usr/share/systemd/gatewayd
/usr/share/systemd/gatewayd/browse.html
 > 
 > which -a systemd-journal-remote
which: no systemd-journal-remote in (/home/???/bin:/usr/local/bin:/usr/bin:/bin)
 > 
 > systemctl status systemd-journal-remote.service 
○ systemd-journal-remote.service - Journal Remote Sink Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-journal-remote.service; indirect; vendor preset: disabled)
     Active: inactive (dead)
TriggeredBy: ○ systemd-journal-remote.socket
       Docs: man:systemd-journal-remote(8)
             man:journal-remote.conf(5)
 > 
 > cat /usr/lib/systemd/system/systemd-journal-remote.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Journal Remote Sink Service
Documentation=man:systemd-journal-remote(8) man:journal-remote.conf(5)
Requires=systemd-journal-remote.socket

[Service]
ExecStart=/usr/lib/systemd/systemd-journal-remote --listen-https=-3 --output=/var/log/journal/remote/
LockPersonality=yes
LogsDirectory=journal/remote
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateDevices=yes
PrivateNetwork=yes
PrivateTmp=yes
ProtectProc=invisible
ProtectClock=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectSystem=strict
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
User=systemd-journal-remote
WatchdogSec=3min

# If there are many split up journal files we need a lot of fds to access them
# all in parallel.
LimitNOFILE=524288

[Install]
Also=systemd-journal-remote.socket
 > 
 > l -h /usr/lib/systemd/systemd-journal-remote
-rwxr-xr-x 1 root root 59K 28. Mär 15:29 /usr/lib/systemd/systemd-journal-remote*
 > 
 > file /usr/lib/systemd/systemd-journal-remote
/usr/lib/systemd/systemd-journal-remote: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=4f78ea653cd2d0afc99dd6569ab5b64f1f3101f1, for GNU/Linux 3.2.0, stripped
 > 

Sorry, but, which ever way anyone wants to cut this cake, on openSUSE Leap 15.4 the “systemd-journal-remote” functionality only ever runs as a systemd service with the executable located in ‘/usr/lib/systemd/’ – and, despite the man page, never as a binary executed from a CLI …

???

I can’t draw any connection between your comment and the question that I raised in the initial post.

Beside that, I don’t know which “cake” you’re talking of. Only because someone wraps an executable in a .service file to let it be executed by systemd at system startup does not mean that this executable may not be used as other CLI tools on the command line. There is no such thing as an explicit either-or-policy for executables. That is simply nonsense. This holds in particular true for such .service files of type Oneshot–the commands could be executed at any time on the command line. See klog.service (without `d´) for an example.

You can find such an use case as CLI tool for systemd-journal-remote here.

plain false!