HTTP SYN packets being ignored

English Network/Internet
1

I am trying to get a squid accelerator (reverse proxy) server running. Everything works as expected until a WAN client tries to connect, at which point the SYN packets on port 80 get ignored. They are visible with both tcpdump and Wireshark. Before blaming squid as the problem, I installed Apache and disabled squid, but Apache doesn’t seem to receive the packets, either.

Where oh where could they be going?!!

Environment:

openSuSE 13.1, updates to 1200UCT 31 July applied from the on-line repos. It’s actually a VirtualBox VM, which I don’t think matters as the packets show up on it.

Network topology:

ISP (Comcast :frowning: ) modem/firewall/router/switch ->10.n.n.n subnet (DMZ) (configured to forward HTTP and HTTPS to 10.n.n.110)
Second firewall/router/switch ->192.168.n.n subnet (INTERNAL)
Ethernet interface for squid server (10.n.n.110)

From the INTERNAL zone, the squid server is correctly accessible. But from a wireless connection to a Comcast public hotspot, all that arrives are the SYN packets with a zero length, and they receive no response.

I have tried dynamically disabling tcp_timestamp in the squid server. There was no visible change in behavior.

Note that the firewall on the squid server is disabled. Yes, yes, it is a risk, but this is a test machine and the backend servers are off-line etc etc. I don’t think the firewall is the problem, but let’s keep it out of the mix.

This was working a few weeks ago, and then Comcast pushed out an upgrade to the modem. Perhaps that was really a downgrade? [One of the other effects is that when I connect to an xfinitywifi hotspot, I get a 192.168.1.n IP whereas prior to that it was a public IP.]

The whole thing will be pretty useless if an external client can’t connect. :frowning:

Cheers.

Tim

2

Question;
Is there a specific reason you’re doing this on Squid as opposed to say Varnish - I’m assuming you wish to setup a load balancing/caching frontend server for web servers in the back?

3

Test every interface with tcpdump.

  • Does the client send the “SYN”?
  • Does the proxy get the “SYN” on the right interface (eth0 or eth1)?
  • Does the proxy transfer the request to the right interface? (spoofing seperate on every interface!)

If you wouldn’t get the transfer, look after the log files in /var/log/squid/… .
If you can’t find any informations about that there, look after the routing between both subnets.

4

Miuku Question;
Is there a specific reason you’re doing this on Squid as opposed to say Varnish - I’m assuming you wish to setup a load balancing/caching frontend server for web servers in the back?

The goal is to allow external testers and reviewers access to a prototyping and testing environment, including the bug tracking software, as well as a couple of private web sites. For convenience, these are running in the Internal zone. It would certainly be an option to move them into the DMZ, but not preferable.

Using a bastion host / reverse proxy allows the internal servers to move around as necessary by setting the name associations via the Squid configuration.

AdaLovelace Test every interface with tcpdump.

  • Does the client send the “SYN”?
  • Does the proxy get the “SYN” on the right interface (eth0 or eth1)?
  • Does the proxy transfer the request to the right interface? (spoofing seperate on every interface!)

If you wouldn’t get the transfer, look after the log files in /var/log/squid/… .
If you can’t find any informations about that there, look after the routing between both subnets.

I have been running tcpdump almost continuously, which is how I see the traffic from the internal and external clients. I use the following command:

tcpdump -i 2 ‘tcp port 80’

– 2 is the correct interface for the DMZ, as verified by ifconfig
– squid is configured to listen on port 80. The first two lines of squid.conf:

>http_port 80 accel defaultsite=webhost.mydomaindottld
>#
>forwarded_for on # so that the true requestor shows up in the server logs

Also, note that to rule out Squid, I started Apache in its place, listening on port 80, and the same behavior occurs, that is, the SYN packets from the external address are dropped in the bit bucket.

– I do not know for sure that the external client is sending the SYN packets. That is a valid question. It is a Win7Pro laptop connected to a Comcast public hotspot (Comcast’s latest cable modems include the “feature” that they are a public hotspot, but that’s a whole separate discussion and not openSuSE specific). I have not (yet) installed network diagnostic software on the Win7 client, so traceroute is not available. The arrival of the external SYN packets corresponds to when I ask Firefox to connect to the site, and the source IP as reported by tcpdump maps to a Comcast address originating in Philadelphia, which is appropriate; I am about 15 miles west of the city. Without some diagnostics (not even traceroute is there!) it’s hard to be sure, but a ping from the external client does get a response from the proper public IP address.

– Once the packets arrive on the proxy host, I don’t see where subnet routing means anything. For diagnostics I have completely disabled the firewall on the proxy host; iptables -L lists no rules at all. The firewall is one of the first places I look for these sorts of problems (my experience on 13.1 is that the first update after a fresh build resets all the interfaces to the External zone), but it is turned off!!

What had me puzzled initially was that the Squid logs (access, and cache with debug set to 6) showed no evidence of the SYN connection requests. Requests from the internal clients were properly logged, including denials due to ACL rules. So as to eliminate the Squid ACL rules as the problem, I tried running Apache on the same host (with Squid stopped, obviously). Apache behaves the same as Squid: it responds properly to any requests coming from the Internal zone (these do arrive on the DMZ interface) but the SYN packets from the external world are ignored. The fact that two different servers have the same behavior makes me conclude that it’s not the servers. If I could point to Squid, I would have posted in the squid-cache forums rather than here in openSuSE, but I don’t think it’s fair to ask them to diagnose problems not related to Squid.

I have been mucking with IP and Linux for twenty years beginning by using the Slackware distro to build custom kernels for firewalls and bastion hosts, then taking up with SuSE ~2001 because it would install cleanly on the Compaq M700 laptops we were using at work. But I am not an IP expert. I have turned over every rock I can think of, and the answer is probably very obvious to somebody who works with the IP stack on a regular basis. I can certainly post the output of ifconfig and tcpdump just to be sure that I am reading it correctly.

This was working six or eight weeks ago. External users were able to connect through the reverse proxy to bugzilla, login, create a sample bug and everybody received the notification e-mail. It was all good. Then, because of firewall problems, Comcast did an update to the modem. Over those last six weeks I have acquired some decent firewalls and routers (SonicWall and Cisco), so I am sore tempted to remove the Comcast modem from the configuration. I will first install IP diagnostic tools on the Win7 external client.

In summary, I may be crazy, but this is really making me nuts! :stuck_out_tongue:

5

Your topology is confusing. Is it as I think you are describing

Internet <> Comcast router <DMZ>> openSUSE VM <> Internal network

or something else like

Internet <> Comcast router <DMZ>> openSUSE VM
<Internal Zone>> LAN

or something else?

And, I’m unclear how “WAN clients” are connecting, are they outside the Comcast router so are no different than Internet clients or are they connecting through either the Comcast router or something else?

Generally speaking, Proxies are configured either as
Reverse - All clients connect through the public interface, usually a designated physical interface (but not necessarily). The webservers sit behind the Proxy
Pass-through - The Proxy services LAN clients, serving content so clients don’t have to download content from across the Internet.

When you consider the above two configurations, it seems that maybe your configuration (ie direction facing) is reversed

I’m not sure by your unclear topology, but you should try connecting to the public IP address of your reverse proxy (should not matter whether your clients are LAN or external).

TSU

6

TSU

Your topology is confusing. Is it as I think you are describing

Sorry, I’ve been a bit inconsistent in my terminology. I should stick to firewall terminology of External/DMZ/Internal zones, where External is anything on the Comcast side of the gateway, that is, the public Internet.

[ol]
[li]External -> [/li][li]Comcast gateway which provides an interface on the 10. (default) subnet, or the “DMZ” (This unit is a modem, firewall, router, switch, wifi access point and dhcp server for the 10. subnet. It is configured to forward HTTP/S to the reverse proxy) [/li][li]Within the 10. subnet there is[/li][LIST=1]
[li]another firewall/gateway to the Internal zone, DMZ static IP 10.n.n.101 [/li][li]a host for the SuSE VM reverse proxy server, DMZ static IP 110 [/li][/ol]

[li]In the Internal zone are the assorted servers and development environments, all in a single 192. subnet: DNS, DHCP, PXEBOOT, NTP and others. [/li][/LIST]

When testing the connection (rather, lack thereof) from an External client, that client is connected only to a Comcast xfinity wifi hotspot, which resolves to the External zone (the CAT5 cable is unplugged). This host has no knowledge of anything inside of the Comcast gateway noted in (1) above. Prior to attempting any connection on this host, I verify its general connectivity by accessing cnn.com and bbc.com. It may not be the most desirable configuration for testing, but it’s what I’ve got. Short of having a second ISP connection, there are challenges associated with debugging external connections.

TSU

I’m not sure by your unclear topology, but you should try connecting to the public IP address of your reverse proxy (should not matter whether your clients are LAN or external).

Which IP address are we referring to? (1) the IP of the reverse proxy’s DMZ interface or (2) the External IP of the Comcast gateway?

In the case of (1), connecting from the Internal zone, everything works. Exactly as the documentation describes. I can show you the tcpdump logs from the reverse proxy server.

In the case of (2), connection from the Internal zone, the SYN packets are sent by Firefox, correctly addressed to the External IP of the Comcast gateway, but vanish before they arrive on the DMZ interface of the reverse proxy server. They don’t show up on the DMZ interface of the VirtualBox host either. But, keep in mind that this is a Comcast-provided gateway with their “custom” firmware. They have made who knows what changes, starting with disabling dynamic DNS (the option remains in the config menu but brings up an error page).

This does not change the fact that, when connecting from an External client, the SYN packets are captured by tcpdump and Wireshark running on the reverse proxy host.

AdaLovelace

  • Does the client send the “SYN”?

Yes, it does. I installed Wireshark on the client and it sends the same number of SYN packets as arrive on the reverse proxy host. On one monitor I watch them be transmitted, on another monitor I watch them arrive.

7

If I understand your description now, your Comcast router/firewall is configured as a “3 legged firewall,” ie it likely has 3 physical interfaces assigned to the 3 firewall zones.

Given that, then your “case (2)” should not be surprising. Packets from your internal zone destined to the DMZ will not be routed through the external interface, they are routed directly from your internal to your DMZ zones. You may be misled that apparently your DMZ hosts are configured as bridged addresses (They are using the same NetworkID as the external zone instead of behind a NAT or PAT) but the Comcast router is likely routing to the interface at a lower layer, probably the MAC address.

This also is consistent with a better security design… So, for instance your Squid reverse proxy is in the DMZ but typically the web server farm is not in the same zone (particularly with bridged addresses as you are configured), that would expose your web farm to threats from the Internet. You didn’t describe explicitly but I assume your web farm is in your Internal zone and then you can open a highly restricted pin hole connection between your Squid proxy (in the DMZ) and your web farm (in your Internal or other protected zone). And, you should expect this pin-hole connection should not be exposed by routing through your Public (Internet) interface.

I don’t know if that also answers your original post which is still somewhat confusing. Your OP suggested (to me) that you were asking a different question about connecting from external (public) hosts. Are you saying that they are unable to connect to your Squid at all?

HTH,
TSU

8

All discussion is helpful. I am not an IP networking expert, particularly at the lower layers. I sorta know what they are but that’s the end of it. N-tier client-server databases were my claim to fame, such as using Oracle’s Tuxedo. (It was AT&T’s then …)

Actually my configuration is a dual firewall, not three-legged. The Comcast gateway is designed for residential use and provides modem, firewall, router, dhcp server, WiFi AP and a four port switch in a single unit (it’s a customized Arris T862G, standard Comcast issue) and I believe it has NAT functionality. It is serving as the perimeter firewall, and I have been referring to its subnet as the DMZ. The Squid host is cabled to one switch port and the core firewall is cabled to another switch port. Both have static IP’s. This is the same configuration I used 20 yrs ago when managing an ISP consultancy, but there were no firewall appliances available so I used custom kernels and the TIS Firewall Tool Kit. While appliances are nice, they have a couple drawbacks: (1) you are at the mercy of the vendor’s firmware (and his updates breaking things) and (2) sometimes it’s difficult to get diagnostics like tcpdump.

Your analysis of the goal is spot-on: The web farm is behind the core firewall and the Squid reverse proxy will get just a pinhole. But the reverse proxy has to work first. :wink:

As for the original problem statement:

HTTP connection requests from the public Internet, as evidenced by SYN packets, arrive on the reverse proxy server’s NIC and then proceed to vanish. Connections from the Internal zone behave as expected. For testing/debugging this problem, the firewall is disabled on the reverse proxy server.

  • Since I have DNS servers running internally, there is an A record for the IP of the Squid proxy’s DMZ NIC. Clients from the Internal Zone are able to connect and get the expected responses, either from Squid or from the parent server. This has been invaluable in testing both the Squid ACLs and the Apache vhost definitions, where the syntax of the requested url matters.
  • The public DNS zones are maintained by a DDNS hosting service and updated via ddclient. With a client in the External zone, nslookup returns the public IP of the Comcast gateway and ping reaches it. But a browser session fails to connect. I’ve only used Firefox so far; perhaps I need to try Opera and (gulp!) IE as well. What seems to happen is that the browser sends a series of SYN packets, I see them leave the test client and I see them arrive in the tcpdump trace on the Squid host, and that’s the end of it. Nothing is returned to the requesting client, and nothing appears in any logs.

Initially the reverse proxy’s firewall was set to DMZ, accepting HTTP and HTTPS. I relaxed that to Internal and then simply disabled it so as to be sure that it wasn’t trapping these packets. Yet they continue to vanish. I’ve done everything I can think of to find the trap, so I seek suggestions as to where to look next.

Any and all discussion is appreciated.

Tim