Container to container dns lookup failed in podman 4.4.4 opensuse leap 15.4

English Virtualization
1

On running,

podman run -p 27017:27017 -e MONGO_INITDB_ROOT_USERNAME=admin -e MONGO_INITDB_ROOT_PASSWORD=password --name mongo-db --net mongo-network -d  mongo

and

podman run -d -p 8081:8081 -e ME_CONFIG_MONGODB_ADMINUSERNAME=admin -e ME_CONFIG_MONGO
DB_ADMINPASSWORD=password -e ME_CONFIG_MONGODB_SERVER=mongodb --net mongo-network --name mongo-express mongo-express

gives the error: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [Error: getaddrinfo ENOTFOUND mongodb

------------------------


(node:7) [MONGODB DRIVER] Warning: Current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
Could not connect to database using connectionString: mongodb://admin:password@mongodb:27017/"
�(node:7) UnhandledPromiseRejectionWarning: MongoNetworkError: failed to connect to server [mongodb:27017] on first connect [Error: getaddrinfo ENOTFOUND mongodb
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
  name: 'MongoNetworkError'
}]
    at Pool.<anonymous> (/node_modules/mongodb/lib/core/topologies/server.js:441:11)
    at Pool.emit (events.js:314:20)
    at /node_modules/mongodb/lib/core/connection/pool.js:564:14
    at /node_modules/mongodb/lib/core/connection/pool.js:1000:11
    at /node_modules/mongodb/lib/core/connection/connect.js:32:7
    at callback (/node_modules/mongodb/lib/core/connection/connect.js:300:5)
    at Socket.<anonymous> (/node_modules/mongodb/lib/core/connection/connect.js:330:7)
    at Object.onceWrapper (events.js:421:26)
    at Socket.emit (events.js:314:20)
    at emitErrorNT (internal/streams/destroy.js:92:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:60:3)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
�(node:7) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
�(node:7) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

On checking custom network mongo-network using podman network inspect mongo-network it displays dns_enabled is set to false.

     {
          "name": "mongo-network",
          "id": "9f4de276f68b6ae6b2df977ff9e8d824ce15a864e690ae11fbe45a07956ec274",
          "driver": "bridge",
          "network_interface": "cni-podman1",
          "created": "2023-06-04T10:52:16.680533377+05:30",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

On debian 11, these commands work perfectly. but on opensuse leap 15.4 container to container dns lookup failed.

2

So change it to true.

So on Debian 11 the mongo-network was created with dns_enabled set to true.

3

the container exits as soon as it is started. So can’t change anything. On reading podman documentation, podman network create <network name> by default sets dns_enabled to true, but in this case opensuse is setting it to false. See the linked documentation page:
https://docs.podman.io/en/latest/markdown/podman-network-create.1.html
even explicity setting --disable-dns = false has no effect. it still remained disabled. @arvidjaar

4

Debian 11 is using podman 3. openSUSE is using podman 4. It seems podman 4 changed how name resolution is implemented. There is similar discussion for Fedora:

If I read the above correctly, podman 4 introduced new network backend which implements name resolution internally.

If using cni backend podman sets dns_enabled if dnsname plugin is installed. It does not seem to be available for Leap by default.

So you could either change net backend or try installing dnsname plugin from Show devel:microos / cni-plugin-dnsname - openSUSE Build Service

5

Thanks for the reply. As per documentation, container to container is only supported by bridge driver, which is default option. So net backend may not solve the issue. I installed cni-pluin-dnsname from opensuse build service, it did change the dns_enable to true. But container to container dnslookup still fails with same error as described above.

6

You confuse backend and driver.

7

Thanks for your valuable time. Yes I confused network_backend with driver. Sorry for that. I changed network_backend from cni to netavark by copying cp /usr/share/containers/containers.conf to /etc/containers/ and appending network_backend = "netavark". After running podman system reset, it did change the backend to netavark. This is confirmed by running podman info --format {{.Host.NetworkBackend}}. After this I recreated the containers from scratch with same commands. This solved the problem.

1 Like
8

One more help needed. How to mark this topic as solved???. Its my first time on opensuse forum.

9

You already told us it is solved. That is it.

10

So that is probably the reason. There is no running container so resolution of container name fails. When I try to run docker.io/library/mongo:latest it fails with

WARNING: MongoDB 5.0+ requires a CPU with AVX support, and your current system does not appear to have that!
  see https://jira.mongodb.org/browse/SERVER-54407
  see also https://www.mongodb.com/community/forums/t/mongodb-5-0-cpu-intel-g4650-compatibility/116610/2
  see also https://github.com/docker-library/mongo/issues/485#issuecomment-891991814

/usr/local/bin/docker-entrypoint.sh: line 416:    24 Illegal instruction     (core dumped) "${mongodHackedArgs[@]}" --fork
11

No, this is not the reason. I have successfully run the container to container networking. There is no need for cni-plugin-dnsname, if you are not using cni as network_backend. See this link: https://www.redhat.com/sysadmin/podman-new-network-stack. For configuration use this link: Configuring Networking for Podman

I have installed netavark and aardvarak-dns from opensuse repos to use netavark as network_backend.

12

Please be aware that if you run sudo podman system reset it will DELETE all your existing containers in podman. So use this command cautiously. This commend is mentioned in second link in Oracle docs.

13

I was struggling same issue from last 36 hours, and I found your only solution which worked for me, Thanks a lot, no chatgpt,bard helped me, my google skills helped me find your solution,