[iptables] Marking packets on PREOROUTING and Redirect with 2 internet isp

Hello,

I think this is my first post (however I use OSuse since 10.2)

Now I got a task:

1.I have 2 ISPs
a. An ISP with static IP, let’s call it Vomistar, gw 146.47.200.73 IP 146.47.200.74 and 146.47.200.72 is the network of this isp, I guess
b. An ISP with dinamyc IP, let’s call it VTRobo
2.I have an Out-of-the-box OPENSUSE 11.3 as router with 3 network cards.
a. eth0 is VTRobo in External zone
b. eth2 is Vomistar in External zone
c. eth3 is my LAN (192.168.2.0/24) in Internal zone
3. IP Masquerading is active, however I have squid working, but stopped.

My Task goal is redirect from LAN to Internet some sites though VTRobo or Vomistar when the destinaton is a known site in a list.

for example:
If some guy inside LAN checks google.com site, this conection must reach the internet, for default, throught VTRobo conection.
However if this guy check the enterprise web site, the conection must go throught Vomistar, becouse I now the IP of our website and I can recongize the destination.

And… how can I do that? The answer is easy, but the implemetation is hard as cancel SNL from air.

When OSuse gets incoming connection from LAN thought eth3, using iptables I can mark the tcp packet if the machine detect certain criteria on it. Before the outside of the tcp packet and the machine detect this mark, then redirect it to Vomistar. If it is there no mark, go to default ISP, VTRobo.

In comands (VTrobo is giving internet for deafault to the machine):


(As root)
# echo "200 vomistar" >> /etc/iproute2/rt_tables #This add a table
# ip route flush table vomistar
# ip route show table main | grep -Ev ^default \
>   | while read ROUTE ; do
>     ip route add table vomistar $ROUTE
> done #I clone main in vomistar table
# ip route add table vomistar default via 146.47.200.73 # so, I gave the gw number to this conection to make it work
# iptables -i eth3 -t mangle -A PREROUTING -p tcp --dst 69.163.223.20  -s 192.168.2.0/24 -j MARK --set-mark 2 #In prerouting in mangle table, I mark with "2" every thing using tcp protocol that coming from the LAN connected to eth3, when they decide to load a site like http://www.smellwing.com becouse it has 69.163.223.20 ip
# iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 146.47.200.74  #This make present the vomistar conection in the nat table
# ip rule add fwmark 2 table vomistar #this is the rule to get every tcp transmision marked with "2" go to omistar table
# ip route flush cache

I get all this from 10.4. Multiple Connections to the Internet and a few other extra thing from differents sites.

So… this didn’t work. I think the process is right, but nothing happens. When I go to SmellWing Cómics, Webcómics Chilenos para todos I see it from VTRobo conection and no Vomistar.

I Try to use tcpdump And Wireshark to check if the tcp o http packets are marked “2” when I load this site… but I don’t know where to see for that mark. Also if I replace “–dst 69.163.223.20” with “–dport 80” (so every web transmition must go to vomistar) to test the sttings, didn’t work either.

My 5 questions:

1.What am I doing wrong?
2 How can I correct it?
3. Where must I see to find the mark “2” on tcpdump?
4. How can I trace all the travel of te packet?
5. can I simplify this with anoter tool?

Thanks for help me!!

pd.:I guess you note my bad redaction and spelling, it is becouse english isn’t my strong language, sorry :frowning:

If some guy inside LAN checks google.com site, this conection must reach the internet, for default, throught VTRobo conection.
However if this guy check the enterprise web site, the conection must go throught Vomistar, becouse I now the IP of our website and I can recongize the destination.

Setting up routing properly would do the job just right IMHO. NAT is only needed because I guess You don’t have a public IP address for every host on your local network. The type of NAT You want to use is, again I guess, masquerade so You change the source IP of packets coming from your LAN to the public IP of one of your external interfaces. This is a special example of SNAT, which does it’s magic **after **routing and so the interface through which the packets will go out to reach the internet is not determined by NAT at all, so You need to be sure You have your routing configured correctly first.

You need to determine which external interface is your default gw, then configure specific routes for the known sites to go through the interface You want. The routes for your local servers should already be in the routing table so that should not be a problem. There are lots of tools to configure the routing table. One of them is “route” the other one is “ip” and others. Choose whatever you’re most comfortable with, check what is your routing table now and configure it to do what You want. Keep in mind, that the routes with a longer mask take precedence over the routes with a shorter mask (because they are more specific) and that is why the default route is sometimes called the last resource route, because the netmask is as short as possible so it’s only used if there is no other more specific route in the routing table.

Hope the explanation is more or less clear. Good luck.

Best regards,
Greg

Thank you @glistwan, your text was very clear and let mi re-think all my configurations.

I setup wrong my router configuration. It let SuseFirewall2 tu masquerade all the things, so I cannot change NAT.

Then I must built the routing by hand.

a. eth0 is VTRobo in External zone. Is a dlink (router) with gw 192.168.1.1 and is in the network 192.168.1.0
b. eth3 is my LAN (192.168.2.0/24) in Internal zone
c. the OSuse router has the ip 192.168.1.100/24 on eth0 and 192.168.2.7/24 en eth3
d. 192.168.2.19 is a machine connected to a switch
e. the switch is connected to the OSuse and assing the ip 192.168.2.19 to the other machine via dhcp4 server.


# iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.1.100
# ip route add 192.168.2.0/24 dev eth0 table main
# ip route 127.0.0.0/8 dev lo table main

I have access to internet on the OSuse router. Of course, masquereading (with ip fowarding)it will give acces to the net to my 192.168.2.0 entery network. But I don’t want masquerading, becouse it will make conflict with my original goal.


# ip route show table main
192.168.1.0 dev eth0 scope link src 192.168.1.100
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.7
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100
169.254.0.0/16 dev lo scope link
127.0.0.0/8 dev lo scope link
defult via 192.168.1.1 dev et

A machine, 192.168.2.19, gets a deny conection when I try to go to a website. So I guess, since I can ping 192.168.2.7 succesfully, the OSuse router don’t know what to do with the packets coming from 192.168.2.19.

What piece of code am I missing to route succefully and get Internet in 192.168.2.19?
To get my original goal acomplished I must avoid masquerading but, must I avoid IP fowarding?

IP forwarding is mandatory if You want your openSUSE to do routing. Usually routing is done based on the destination IP address (not source) so openSUSE does know what to do with packets destined to internet from 192.168.2.19. It will forward them to 192.168.1.1. Maybe the dlink doesn’t know that 192.168.2.0/24 has to be reached through 192.168.1.100 ?

Only You can know this. Run wireshark on both openSUSE eth0 and eth3 interfaces, run a ping to internet from 192.168.2.19 and see what is happening. My guess is that the ICMP requests will go to 192.168.1.1, then to internet, then they will get back to dlink’s external IP and then the dlink will not know what to do with them.

Best regards,
Greg

@glistwan

Thank you very much. Your help is great.

I check every thing and leran a lot of thing.

Even with masquerading, I can mark the packages in PREROUTING. And the marked ones are routing to Vomistar, like my post #1 wants.

I have 3 routing tables.

main, VTRobo and Vomistar.


146.54.200.72/29 dev eth2  proto kernel  scope link  src 146.54.200.74 
192.168.2.0/24 dev eth3  proto kernel  scope link  src 192.168.2.7 
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.103 
169.254.0.0/16 dev eth0  scope link 
127.0.0.0/8 dev lo  scope link 

Yep, eth0 is not longer dhcp.

So, the three tables are equal.


# ip route add table VTRobo default via 192.168.1.1
# ip route add table Vomistar default via 146.54.200.74
# ip route add table main default via 192.168.1.1

Because main table had his own default gw, if I load Internet from VTRobo, it works. However Vomistar do nothing.

It seems when I try to use Vomistar table, they use the gw from main table. If I delelete the default gw from main and do:


# ip route add table main default via 146.54.200.74

Vomistar works like a charm. VTRobo loose conection.

Of course, if I let mai without default gw, I loose all the Internet from all ISPs

Why OpenSuse use my default gw form main even if a im routing the traffic to another table (like my post #1)?

When I check from Wireshark,

With 192.168.1.1 as default gw in main,
My laptop (192.168.2.14) I check a site. So 192.168.1.103 (VTRobo) talks to google and google talks with 192.168.1.103.
But if I check a site that I mark with the PREROUTING rules and go to vomistar table, My latop talks directly with the site! Of course, when it’s back form the Internet to the firewall, the firewall don’t know what to do with 192.168.2.14… exatly how you think in your post above.

So, What am I missing is a rule that tells what default gw must to be used when the marked package go to Vomistar table…

I don’t know. I’ve never set up iptables to work the way You want on openSUSE.

I need some clarification though. From what I understood You wanted to just access specific sites using Vomistar. You don’t need a default gw for that. Default gw matches any destination and is the least specific route. If You want to reach for example forums.opensuse.org through Vomistar You need to add a specific route to that destination :

ip route add 130.57.4.15/32 via 146.54.200.74

With an entry like this and a default route through 192.168.1.1 all the other internet traffic will go to 192.168.1.1.

Best regards,
Greg

Ok I think that You need to read this link and see what rules You have got configured.
IPROUTE2 Utility Suite Documentation
Maybe the different routing tables are not checked at all or not in the order You want because of the lack of those rules ?

ip rule show

Should be helpful to determine this.

Best regards,
Greg

On Wed, 26 Oct 2011 22:06:02 GMT, smellwing
<smellwing@no-mx.forums.opensuse.org> wrote:

>
>@glistwan
>
>Thank you very much. Your help is great.
>
>I check every thing and leran a lot of thing.
>
>Even with masquerading, I can mark the packages in PREROUTING. And the
>marked ones are routing to Vomistar, like my post #1 wants.
>
>I have 3 routing tables.
>
>main, VTRobo and Vomistar.
>
>
>Code:
>--------------------
>
> 146.54.200.72/29 dev eth2 proto kernel scope link src 146.54.200.74
> 192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.7
> 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.103

I think you may want to try 192.168.1.0/22 and 192.168.2.0/22 for the
routing. This should allow the 192.168.1.X and the 192.168.2.X
subnetworks to see each other.

> 169.254.0.0/16 dev eth0 scope link
> 127.0.0.0/8 dev lo scope link
>
>--------------------
>
>
>Yep, eth0 is not longer dhcp.
>
>So, the three tables are equal.
>
>Code:
>--------------------
>
> # ip route add table VTRobo default via 192.168.1.1
> # ip route add table Vomistar default via 146.54.200.74
> # ip route add table main default via 192.168.1.1
>
>--------------------
>
>
>Because main table had his own default gw, if I load Internet from
>VTRobo, it works. However Vomistar do nothing.
>
>It seems when I try to use Vomistar table, they use the gw from main
>table. If I delelete the default gw from main and do:
>
>
>Code:
>--------------------
>
> # ip route add table main default via 146.54.200.74
>
>--------------------
>
>
>Vomistar works like a charm. VTRobo loose conection.
>
>Of course, if I let mai without default gw, I loose all the Internet
>from all ISPs
>
>Why OpenSuse use my default gw form main even if a im routing the
>traffic to another table (like my post #1)?

Thak you both @glistwan and @47josephkk

Because your post and my own research I learn a lot about ip, iproute and GNU/Linux networking. I am fell bad when I realize the knowledge I adquire was not enough.

So I give up and I use Shorewall, waht from the star I try to avoid. But this time is diferent: I know what I am doing!

So I build up everithing in 4 o 5 hours with the help of this Shorewall official paper:

Shorewall and Multiple Internet Connections

I have problems with Shorewall 4.0.5 in the past and now I installed Shorewall 4.4.24. So, I don’t know how it will work.

Thank again for all your help.

Thanks for the feedback and use whatever works :slight_smile:

Best regards,
Greg

On Wed, 02 Nov 2011 12:46:02 GMT, smellwing
<smellwing@no-mx.forums.opensuse.org> wrote:

>
>Thak you both @glistwan and @47josephkk
>
>Because your post and my own research I learn a lot about ip, iproute
>and GNU/Linux networking. I am fell bad when I realize the knowledge I
>adquire was not enough.
>
>So I give up and I use Shorewall, waht from the star I try to avoid.
>But this time is diferent: I know what I am doing!
>
>So I build up everithing in 4 o 5 hours with the help of this Shorewall
>official paper:
>
>‘Shorewall and Multiple Internet Connections’
>(http://www.shorewall.net/MultiISP.html)
>
>I have problems with Shorewall 4.0.5 in the past and now I installed
>Shorewall 4.4.24. So, I don’t know how it will work.
>
>Thank again for all your help.

I am pleased that anything i offered helped you.

?-)