Network throughput capped to 30MBps?

Hi all!

I have an internet router / firewall that has been recently upgraded to openSuse 11.4. The new config’s network throughput looks to be capped. (We have been using similar configs for about 8-10 years with different versions of Suse. No problems so far.) The main problem is that network throughput somehow seems to be capped at 30 MBps. Everything else (firewall, DHCP, DNS service, openVPN) works well. Transfer speed between LAN interfaces will not go above 30 MBps regardless of protocol or source and target machines, just if like some shaper or traffic limiter would be installed.

Here is some details:
there are 5 physical network interfaces, 4x 1000 Mbit, and one 100Mbit. Three of the 1000Mbit interfaces server LAN, the fourth one is connected to Cable (WAN). The 100Mbit interface is connected to a DSL line. Default routing interface is the Cable (the other one is being used to connect some specific networks through VPN). 1000MBit interfaces are D-Link cards (DGE-528T)

I have installed openSuse 11.4 with pretty simply with basic packages only, console version, no KDE or any other GUI. Services that I run: DHCP, named, openVPN, and a custom firewall script (SuseFirewall is inactive). I have NOT installed nor set up any shaping, like wondershaper. Also searched the sysconfig settings for some relevant settings. No success.

After making some tests, I found out that this strange speed limit works one way only, from the router to the LAN. The other way seems to be OK, so for example uploading to the router from the LAN works pretty well. From the router’s point of view: download is OK, upload is limited.

Any idea?

rege wrote:

> After making some tests, I found out that this strange speed limit
> works one way only, from the router to the LAN. The other way seems to
> be OK, so for example uploading to the router from the LAN works pretty
> well. From the router’s point of view: download is OK, upload is
> limited.
>
> Any idea?
>
>

Hi there,

There is nothing kernel wise or Suse-specific wise which would (normally)
cause this.

Honestly, this really sounds like a rate-limit on the switch. Can you triple
check the config on the switch?

It might be good to absolutly isolate the issue: OS or network. One way
would be to set up a crossover connection to a netbook, etc. and run a speed
test device to device, rule out the switch alltogether. (Or even just from
one interface to another.) This way you know for sure if you are testing the
NIC and tcp/ip stack throughput only, and not a hidden switchport config.


Cheers,
login_name

30mbytes/s is generally typical for file transfers when my network is Fast Ethernet (rated 100mbits/s), and that’s the case no matter the OS.

Always is the rate for example when I do backup/restore across the network.

Depending on the size of file transferred and network topology, if you <really> think you’re not getting throughput anywhere close to what you believe you should be getting, you might consider doing some or all the things I list in my artilcle published here

Optimize and Fix your Network Connection - Su Networking Technical

Tony

@login_name: Agree, there is no reason to apply something like that in default. Switches are simple D-Link latest GBit ones, connected at 1000MBps (it’s a Small Office environment), there is not much to configure, or screw up on them. Isolation is a good idea, let me come back after making some more tests.

@Tony: thanks for the link, FYI 30 is meant in Mbit/s. I have mixed up the casing, sorry! :slight_smile: (AFAIR Fast Ethernet max. speed is around 11-12 MByte/s)

Thanks for the comments so far, as soon as I have made some more tests, I reply the results.

On 2011-10-17 11:06, rege wrote:
>
> @login_name: Agree, there is no reason to apply something like that in
> default. Switches are simple D-Link latest GBit ones, connected at
> 1000MBps (it’s a Small Office environment), there is not much to
> configure, or screw up on them. Isolation is a good idea, let me come
> back after making some more tests.

Cables.

If the cables are bad, the connection should reset to a lower speed (100 or
10). Or, if it doesn’t, then the error rate and retries go up.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

There is theoretical, and there is practical. Practical <never> approaches the full rated speed of the medium for many reasons even under favorable conditions… ie. normal line losses, bi-directional nature of TCP/IP, more.

Unless you really want to pursue the math and do some comprehensive network packet analysis, IMO the most practical approach is to just “know” what you should expect… And I’m saying that 30Mbytes over Fast Ethernet is fairly normal in my experience.

Tony

I have a different take on it than Tony, which is all good. But from my experience, in data center environments, I would never tolerate 30Mbps from a 100Mbit connection. 80Mbit would be more to my liking, and there is no reason it can not be achieved, unless there is a bad NIC, switchport, cable or router issue. Of course it also depends on the exact network topology involved - how many switches / routers are between the hosts etc. Even cheap consumer switches should do better than 30Mbit a sec.

I would suggest: Look at your interface statistics to see if there are a lot of errors / collisions (ethtool -S eth0, ifconfig et0, check logs of any relevant messages (dropped packets, renegotiation of links, etc.). Do a direct connect test to the host in question (cross-over), or just on the same switch and subnet to eliminate going through the router - from two devices on the same switch you should absolutely see near line speed.

I find nttcp and iperf to be excellent tools for testing throughput issues.

Cheers,
Lews Therin

Not 30mbs over Fast Ethernet,
30 megabytes per second file transfer.

In other words, 8x faster.

There is bits as the standard unit measurement when measuring bandwidth, then there can be bytes or megabytes which is the standard unit of measurement of a file size. I quickly recognized the “30 megabytes of file transfer” by noting how long it takes to transfer a system image for backup or recovery across a network when the image is typically approx 8gigbytes in size.

Tony

On Wed, 19 Oct 2011 09:06:02 GMT, LewsTherinTelemon
<LewsTherinTelemon@no-mx.forums.opensuse.org> wrote:

>
>I have a different take on it than Tony, which is all good. But from my
>experience, in data center environments, I would never tolerate 30Mbps
>from a 100Mbit connection. 80Mbit would be more to my liking, and there
>is no reason it can not be achieved, unless there is a bad NIC,
>switchport, cable or router issue. Of course it also depends on the
>exact network topology involved - how many switches / routers are
>between the hosts etc. Even cheap consumer switches should do better
>than 30Mbit a sec.
>
>I would suggest: Look at your interface statistics to see if there are
>a lot of errors / collisions (ethtool -S eth0, ifconfig et0, check logs
>of any relevant messages (dropped packets, renegotiation of links,
>etc.). Do a direct connect test to the host in question (cross-over), or
>just on the same switch and subnet to eliminate going through the router
>- from two devices on the same switch you should absolutely see near
>line speed.
>
>I find nttcp and iperf to be excellent tools for testing throughput
>issues.
>
>Cheers,
>Lews Therin

Let us not confuse line rates with throughputs. Network topology and
number of nodes gets into the mix really fast. One 100tx box to another
with nothing else should do 80 Mbits or slightly better. If 20 boxes are
all there and all transmissions are faithfully echoed to all boxes just
one fully driven file transfer will not get much more than 40 Mbits. Just
too much hash flying around. It is a result of using CSMA/CD (carrier
sense multiple access - [with] collision detection).

?-)

Not if You’re using a switch. Then the collisions are avoided and a full throughput can be used for all hosts.

Best regards,
Greg

On Thu, 20 Oct 2011 11:56:02 GMT, glistwan
<glistwan@no-mx.forums.opensuse.org> wrote:

>
>josephkk;2395366 Wrote:
>> On Wed, 19 Oct 2011 09:06:02 GMT, LewsTherinTelemon
>> <LewsTherinTelemon@no-mx.forums.opensuse.org> wrote:
>>
>> >
>> >I have a different take on it than Tony, which is all good. But from
>> my
>> >experience, in data center environments, I would never tolerate 30Mbps
>> >from a 100Mbit connection. 80Mbit would be more to my liking, and
>> there
>> >is no reason it can not be achieved, unless there is a bad NIC,
>> >switchport, cable or router issue. Of course it also depends on the
>> >exact network topology involved - how many switches / routers are
>> >between the hosts etc. Even cheap consumer switches should do better
>> >than 30Mbit a sec.
>> >
>> >I would suggest: Look at your interface statistics to see if there are
>> >a lot of errors / collisions (ethtool -S eth0, ifconfig et0, check
>> logs
>> >of any relevant messages (dropped packets, renegotiation of links,
>> >etc.). Do a direct connect test to the host in question (cross-over),
>> or
>> >just on the same switch and subnet to eliminate going through the
>> router
>> >- from two devices on the same switch you should absolutely see near
>> >line speed.
>> >
>> >I find nttcp and iperf to be excellent tools for testing throughput
>> >issues.
>> >
>> >Cheers,
>> >Lews Therin
>>
>> Let us not confuse line rates with throughputs. Network topology and
>> number of nodes gets into the mix really fast. One 100tx box to
>> another
>> with nothing else should do 80 Mbits or slightly better. If 20 boxes
>> are
>> all there and all transmissions are faithfully echoed to all boxes just
>> one fully driven file transfer will not get much more than 40 Mbits.
>> Just
>> too much hash flying around. It is a result of using CSMA/CD (carrier
>> sense multiple access - [with] collision detection).
>>
>> ?-)
>Not if You’re using a switch. Then the collisions are avoided and a
>full throughput can be used for all hosts.
>
>Best regards,
>Greg

It helps a little in the average case in that average network latency goes
down, but he switch does not reduce the total traffic. And many
inexpensive “switches” and “routers” are “store and forward hubs” and all
the nodes see all the traffic.

?-)

I agree although the inexpensive switches nowadays are getting better and better :slight_smile:

Best regards,
Greg

Hi guys!

This problem is solved now. (Sort of…)

A short summary of what I did, maybe it will help someone else.

I have made tests on 3 interfaces out of the 5 we had. All of them showed the same symptoms (download around 30MByte/s, upload 4MBit/s). NICs are brand new, but after your first comments, i was quite confident, that this problem is hardware-related.

First I have tried to exchange the NICs with each other, that did not help, but when I replaced one of them with an older Intel one, the “speedcap” has changed. ** System-wide(!).** I realized, that the whole issue is probably something bus-related.

There is something, I forgot to mention: the MoBo is an Asrock one, AM2 architecture, 1x AGP and 5xPCI slots, I was using 4 of the PCIs + the built in NIC. That was probably too much. Removing 2 NICs has helped, the “speedcap” significantly raised, or maybe even has gone. At this point I stopped making tests, since I realized that this config will not work.

Finally, I was not able to find out the real cause, since i did not find any rules or hints in the documentation for populating the PCI slots. BIOS does not have any PCI related setting either, but my guess is on IRQs, or maybe some motherboard problem. Anyway, it did note have anything to do with Suse.

So I ended up at purchasing a completely mobo and CPU. The same system runs now on an AMD A4 and works like a charm. Thanks for all the help!

This sounds identical to my issue. I tried swapping cables, etc, but found the same router works fine with XP and even works well with Mint, but not Suse 11.xx (I think I have 11.3 now)
These guys seem to be talking about teh same thing too.
How to run an Asus N13 USB adaptor at full speed