Docker container external internet connectivity issues

dijksterhuis · May 19, 2017, 9:16pm

Hello all,

First post, but hopefully will get involved in other bits after this… Caveat - I’m not a networking boffin.l have experience with linux + docker, but not super hot.

OS: Tumbleweed 20170505

Docker version:

Client:
 Version:      1.13.0
 API version:  1.25
 Go version:   go1.7
 Git commit:   78d1802
 Built:        Mon Apr 17 09:04:48 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.0
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7
 Git commit:   78d1802
 Built:        Mon Apr 17 09:04:48 2017
 OS/Arch:      linux/amd64
 Experimental: false

Can pull and run docker images fine - e.g. hello-world, redis, ubuntu, alpine. However, containers can’t connect to the internet to, for example, perform apt-get or pip install. Happens in both docker builds or docker runs. Ran the following and no return packets:

dijksterhuis@pc-67-96:~> docker run --rm -it alpine '/bin/ash'                                
/ # ping google.com
PING google.com (216.58.212.110): 56 data bytes

Also ran following to check above wasn’t dodgy alpine container.


dijksterhuis@pc-67-96:~> docker run --name test -it ubuntu "/bin/bash"  
root@ea7f1bb9646c:/# apt-get update
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [2127 B]
Get:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [2127 B]
Get:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [2127 B]
Err:1 http://archive.ubuntu.com/ubuntu xenial InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Err:3 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Err:4 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [2127 B]
Err:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
Fetched 4254 B in 0s (71.9 kB/s)    
Reading package lists... Done
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-updates/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/xenial-backports/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Failed to fetch http://security.ubuntu.com/ubuntu/dists/xenial-security/InRelease  Clearsigned file isn't valid, got 'NOSPLIT' (does the network require authentication?)
E: Some index files failed to download. They have been ignored, or old ones used instead.

My ISP set up 1:2:1 NAT for me last night to allow me to SSH into my desktop. Thought it could be an issue so have asked them to disable and rebooted. No luck. I do know for a fact that they don’t like port forwarding (incidentally, they won’t allow SSH on port 22, SSH must be on a port higher than 1024 - could this be something to do with it?).

Attempted same tests on Macbook Pro (same network) and worked fine. So don’t believe it’s external network issue. Tried zypper rm docker ; zypper in docker. No luck. systemctl restart docker has no effect on issue.

Scoured some github issues and looked at tcpdump and iptables. For tcpdump, ran:


sudo tcpdump -i docker0 icmp -v
sudo  tcpdump -i enp2s0 icmp -v 
sudo tcpdump -i veth537b204 icmp -v

Only successful packet transfers I saw were to google. Nothing seems to returns successfully (port unreachable error?)

tcpdump docker0

18:52:23.188925 IP (tos 0x0, ttl 64, id 45280, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 53, length 64
18:52:23.189163 IP (tos 0xc0, ttl 63, id 40164, offset 0, flags [none], proto ICMP (1), length 112)
    gw.customer.marketgait.collegiate.lan > 172.17.0.2: ICMP google-public-dns-a.google.com protocol 1 port 27949 unreachable, length 92
        IP (tos 0x0, ttl 62, id 45280, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 53, length 64
18:52:24.189053 IP (tos 0x0, ttl 64, id 45346, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 54, length 64
18:52:24.189372 IP (tos 0xc0, ttl 63, id 40168, offset 0, flags [none], proto ICMP (1), length 112)
    gw.customer.marketgait.collegiate.lan > 172.17.0.2: ICMP google-public-dns-a.google.com protocol 1 port 40169 unreachable, length 92
        IP (tos 0x0, ttl 62, id 45346, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 54, length 64

tcmpdump enp2s0

18:56:14.214688 IP (tos 0x0, ttl 63, id 9117, offset 0, flags [DF], proto ICMP (1), length 84)
    pc-67-96.customer.marketgait.collegiate.lan > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 284, length 64
18:56:14.214929 IP (tos 0xc0, ttl 64, id 40640, offset 0, flags [none], proto ICMP (1), length 112)
    gw.customer.marketgait.collegiate.lan > pc-67-96.customer.marketgait.collegiate.lan: ICMP google-public-dns-a.google.com protocol 1 port 23308 unreachable, length 92
        IP (tos 0x0, ttl 62, id 9117, offset 0, flags [DF], proto ICMP (1), length 84)
    pc-67-96.customer.marketgait.collegiate.lan > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 284, length 64
18:56:15.214815 IP (tos 0x0, ttl 63, id 9190, offset 0, flags [DF], proto ICMP (1), length 84)
    pc-67-96.customer.marketgait.collegiate.lan > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 285, length 64
18:56:15.215117 IP (tos 0xc0, ttl 64, id 40641, offset 0, flags [none], proto ICMP (1), length 112)
    gw.customer.marketgait.collegiate.lan > pc-67-96.customer.marketgait.collegiate.lan: ICMP google-public-dns-a.google.com protocol 1 port 36296 unreachable, length 92
        IP (tos 0x0, ttl 62, id 9190, offset 0, flags [DF], proto ICMP (1), length 84)
    pc-67-96.customer.marketgait.collegiate.lan > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 285, length 64

e.g. veth537b204

18:57:07.221139 IP (tos 0x0, ttl 64, id 16578, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 337, length 64
18:57:07.221367 IP (tos 0xc0, ttl 63, id 40695, offset 0, flags [none], proto ICMP (1), length 112)
    gw.customer.marketgait.collegiate.lan > 172.17.0.2: ICMP google-public-dns-a.google.com protocol 1 port 41987 unreachable, length 92
        IP (tos 0x0, ttl 62, id 16578, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 337, length 64
18:57:08.221252 IP (tos 0x0, ttl 64, id 16721, offset 0, flags [DF], proto ICMP (1), length 84)
    172.17.0.2 > google-public-dns-a.google.com: ICMP echo request, id 2816, seq 338, length 64

iptables output (not sure what looking at here). Entries that see packet increases marked (!)


dijksterhuis@pc-67-96:~> clear ; sudo iptables -t filter -nvL
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination          
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0            
(!) 62242   70M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate ESTABLISHED
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED
   17  3258 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:5353 PKTTYPE = multicast
   11  4754 input_ext  all  --  *      *       0.0.0.0/0            0.0.0.0/0            
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 LOG flags 6 level 4 prefix "SFW2-IN-ILL-TARGET "
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination          
(!)  284 34620 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0            
(!)  141 22735 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            
(!)  141 22735 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
(!)  143 11885 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0            
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0            
    0     0 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 LOG flags 6 level 4 prefix "SFW2-FWD-ILL-ROUTING "

Chain OUTPUT (policy ACCEPT 31759 packets, 3621K bytes)
 pkts bytes target     prot opt in     out     source               destination          
    0     0 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0            

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination          

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination          
  284 34620 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Chain forward_ext (0 references)
 pkts bytes target     prot opt in     out     source               destination          

Chain input_ext (1 references)
 pkts bytes target     prot opt in     out     source               destination          
    6  4110 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            PKTTYPE = broadcast
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 4
    0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 8
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* sfw2.insert.pos */ PKTTYPE != unicast
    0     0 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 tcp flags:0x17/0x02 LOG flags 6 level 4 prefix "SFW2-INext-DROP-DEFLT "
    0     0 LOG        icmp --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 LOG flags 6 level 4 prefix "SFW2-INext-DROP-DEFLT "
    0     0 LOG        udp  --  *      *       0.0.0.0/0            0.0.0.0/0            limit: avg 3/min burst 5 ctstate NEW LOG flags 6 level 4 prefix "SFW2-INext-DROP-DEFLT "
    5   644 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Chain reject_func (0 references)
 pkts bytes target     prot opt in     out     source               destination          
    0     0 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with tcp-reset
    0     0 REJECT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
    0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            reject-with icmp-proto-unreachable

Any suggestions or help would be much appreciated! At my wits end. I’d rather not switch distros, Tumbleweed is really growing on me.

Thanks in advance for any help,

Dijksterhuis

dijksterhuis · May 19, 2017, 9:59pm

Solved (sort of) - discovered the issue here when I was posting the above on stackoverflow. Will be running with -net=host arg for now… not particularly sure about sockets. If someone has any ideas about that, would love some help/pointers…

http://stackoverflow.com/questions/33793470/docker-container-cannot-access-internet#

tsu2 · May 20, 2017, 4:01am

Take a look at my Docker “How To” that improved on content that was posted at docker.com. Although things have changed slightly, most of the content on my page is still relevant, like one way to configure networking. Note that there are <many> networking concepts if you want to master (or be versatile) configuring networking (The problem with your command is that you didn’t define a networking interface as you discovered). Don’t know what you mean by “sockets” – network sockets? something else?

https://github.com/putztzu/docker/blob/master/docs/installation/linux/SUSE.md

Note the concepts… What kind of networking interface to configure, and if you want to configure inbound connections you have to configure forwarding from the Host to the container (you don’t have to configure this if you only need to enable outbound connections).

If you want to run through some additional quick exercises, I adapted a number of older docker demos to openSUSE commands which can be found listed on this page (along with a bunch of other stuff I’ve written about), all useful stuff if you’re brand new to docker…

https://en.opensuse.org/User:Tsu2

TSU

dijksterhuis · May 22, 2017, 4:09pm

@tsu2

Yep, saw your page but only glossed over it in early research - didn’t think it would apply or something. Silly me! Will check out the networking stuff you talk about when I’ve got some spare time. the --net=host arg hack should work for now, but not ideal.

Plenty of experience with docker, just on my Macbook. Thanks for the links though. Bookmarked them.

tsu2 · May 22, 2017, 4:18pm

Yes,
Actually setting “-net=host” is an early configuration not suggested often in current documentation, I offered that as a “quick configuration” because it’s fast, easy to configure and “just works.”

Links to “more normal” networking when the container has its own networking interface are in the links I provided in that same section.

TSU