Traefik unable to pick up routers/services, 404s when accessing API container

I’ve got a docker compose setup including traefik which is working fine on Ubuntu (in WSL2) and macOS, but I recently switched to openSUSE Tumbleweed and no matter what I try, I get traefik’s 404 not found response when trying to access my API. I’ve confirmed that I can get the correct response if I directly hit the API with docker exec -it backend sh -c “curl localhost/api/v2/utils/health”. I’ve also tried out the basic whoami example from the traefik site and I still get a 404. Below are my docker-compose.yml and docker-compose.override.yml files. Wondering if there’s anyone that can give me a hand. I’ve tried a ton of things I’ve seen online, including stopping the firewalld service to see if it’s related to that, adding the networks in docker-compose.yml to the backend (the API) container, among other things.

I’m pretty bad with networking stuff, but there were a couple things I thought were odd when I started digging. Just default Tumbleweed stuff, seemingly. There was no static hostname when looking at hostnamectl. Only a transient of localhost. /etc/hostname was set to localhost, though. But I set the hostname to localhost so that’s static now. There was also a fallback to 127.0.0.1 in /etc/hosts to localhost.localdomain which I removed. Not sure if it makes sense to do this or not.

I’ve also confirmed that the server in the backend container is running on 0.0.0.0:80 .I’m also in the docker group on my machine, and am running rootless.

At this point I’m sort of thinking it’s just some Docker/OS config that I don’t have set up correctly.

docker-compose.yml

version: "3.8"

services:
  proxy:
    image: traefik:v2.10
    networks:
      - ${TRAEFIK_PUBLIC_NETWORK?TRAEFIK_PUBLIC_NETWORK variable not set}
      - default
    env_file:
      - .env
    environment:
      - STACK_NAME=${STACK_NAME?STACK_NAME variable not set}
      - TRAEFIK_TAG=${TRAEFIK_TAG?TRAEFIK_TAG variable not set}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command:
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # Add a constraint to only use services with the label for this stack
      # from the env var TRAEFIK_TAG
      - --providers.docker.constraints=Label(`traefik.constraint-label-stack`, `${TRAEFIK_TAG?TRAEFIK_TAG variable not set}`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Enable Docker Swarm mode
      - --providers.docker.swarmmode
      # Enable the access log, with HTTP requests
      - --accesslog
      # Enable the Traefik log, for configurations and errors
      - --log
      # Enable the Dashboard and API
      - --api
  # removed deploy.labels here since it's not useful

  backend:
    image: "XXXXX/${DOCKER_IMAGE_BACKEND?DOCKER_IMAGE_BACKEND variable not set}:${TAG-latest}"
    env_file:
      - .env
    environment:
      # ...
      # bunch of envars
      # ...
    build:
      context: ./backend
      args:
        INSTALL_DEV: ${INSTALL_DEV-false}
    # removed deploy.labels here since it's not useful

volumes:
  app-db-data:

networks:
  traefik-public:
    # Allow setting it to false for testing
    external: ${TRAEFIK_PUBLIC_NETWORK_IS_EXTERNAL-true}

docker-compse.override.yml

version: "3.8"

services:
  proxy:
    ports:
      - "80:80"
      - "8090:8080"
    command:
      # Enable Docker in Traefik, so that it reads labels from Docker services
      - --providers.docker
      # Add a constraint to only use services with the label for this stack
      # from the env var TRAEFIK_TAG
      - --providers.docker.constraints=Label(`traefik.constraint-label-stack`, `${TRAEFIK_TAG?TRAEFIK_TAG not set}`)
      # Do not expose all Docker services, only the ones explicitly exposed
      - --providers.docker.exposedbydefault=false
      # Disable Docker Swarm mode for local development
      # - --providers.docker.swarmmode
      # Enable the access log, with HTTP requests
      - --accesslog
      # Enable the Traefik log, for configurations and errors
      - --log
      - --log.level=DEBUG
      # Enable the Dashboard and API
      - --api
      # Enable the Dashboard and API in insecure mode for local development
      - --api.insecure=true
    labels:
      - traefik.enable=true
      - traefik.http.routers.traefik-public-http.rule=Host(`${DOMAIN?DOMAIN not set}`)
      - traefik.http.services.traefik-public.loadbalancer.server.port=80

  pgadmin:
    ports:
      - "5050:5050"

  backend:
    ports:
      - "8888:8888"
    volumes:
      - ./backend/app:/app
    environment:
      - SERVER_HOST=http://${DOMAIN?DOMAIN not set}
    build:
      context: ./backend
      args:
        INSTALL_DEV: ${INSTALL_DEV-true}
    command: /start-reload.sh
    labels:
      - traefik.enable=true
      - traefik.constraint-label-stack=${TRAEFIK_TAG?TRAEFIK_TAG not set}
      - traefik.http.routers.backend-http.rule=PathPrefix(`/api`) || PathPrefix(`/docs`) || PathPrefix(`/redoc`)
      - traefik.http.services.backend.loadbalancer.server.port=80

networks:
  traefik-public:
    # For local dev, don't expect an external Traefik network
    external: false

I have recently run into something similar and I think it is a problem with the new network that is created with the compose file. For some reason, any new network other than docker0 cannot route outside properly. Just doing something as simple of this shows these results:

~
❯ docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
f38b91fe01f6   bridge    bridge    local
d9eb7abf5062   host      host      local
8ec2362a9863   none      null      local

~
❯ docker network create test-network
16a6b7f492c8c452ca25d9efb2a147212881898fd161a5bafa32354879201a31

~
❯ docker network ls
NETWORK ID     NAME           DRIVER    SCOPE
f38b91fe01f6   bridge         bridge    local
d9eb7abf5062   host           host      local
8ec2362a9863   none           null      local
16a6b7f492c8   test-network   bridge    local

~
❯ docker run -ti --rm alpine:latest ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=57 time=8.303 ms
64 bytes from 8.8.8.8: seq=1 ttl=57 time=8.369 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 8.303/8.336/8.369 ms

~ took 2s
❯ docker run -ti --rm --network=test-network alpine:latest ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss

~ took 5s
❯

I still haven’t figured out a way around this so if anyone else has some ideas, I am open to hearing them. :blush:

Hello, and welcome to the openSUSE forums.

I have no idea about the original problem, nor about yours. But it is in general not a good idea to hang a problem, even if similar, at the end of an existing thread. Exposure to potential helpers is always best with a new thread with good title.