Belkin Gigabit USB

[pythagorean]

This is a genuine happy story involving Linux/Fedora: I bought a Belkin Gigabit USB ethernet adapter and found that it simply works with Fedora out of the box and does not work at all with Ubuntu or Windows Vista Ulimate. That was rather uplifting.

I started experimenting with my network after reading networking threads on this forum. I realized that I had always given a lot of thought to cpu speed but I had never given much thought to my network performance which was based on a 10/100 fast ethernet/wireless G router and cat 5e cable. On this network I only ever got 11 megabytes per second transfer speeds between my computers. SO, I got a gigabit ethernet/wireless N router... only to discover that my laptop ethernet cards are 10/100 fast ethernet.

My question is this: now that I have the whole network set up as gigabit ethernet, I have found that a 600 MB avi file transfer between computers will still only get up to 12 megabytes per second max but if I transfer a 4 GB vdi between computers then the transfer will get up to 40 Megabytes per second... does a larger file size somehow tell the network to do the transfer faster than a smaller file? I mean: does the network know that a very large file is being transfered and therefore give more bandwidth to the transfer?

The comments that follow this Tom's Hardware article seem to suggest that this is the case:

http://www.tomshardware.com/reviews/gigabit-ethernet-bandwidth,2321.html

[jpollard]

A lot of the difference depends on the memory available. The initial delay
(loading buffers, filling readahead buffers, scheduling I/O) for small files
is significant; but for large files, this is amortized over a longer period.

Next, (on the destination side), it takes time to allocate space in the
filesystem. For small files this is a significant amount of time. For large
files, the allocation may take the same time... but doesn't occur very
often. It may happen once/twice for small files, but for large ones the
I/O overlap covers a lot of the time.

As far as I know, there is nothing special about a large file vs small file
in network transfer. What has a bigger effect is the method of transfer.
One thing FTP had going for it was that the size of data transfer could
be sent at the beginning - allowing the reciever to preallocate a large
chunk of filesystem space once, instead of incrementally. This is up to
the applications performing the transfer, not the network.

A lot of delay will occur in the disk storage. RAID (with parallel data
transfers) will always be faster than single disks.

This is a bit rambling, but I've seen contortions done to raid configurations
trying to make 5 pound sausages hold 15 pounds of ground meat. The
end result - yes you can make configurations that will transfer data near
the maximum theoretical rate... but there is a truly horrible penalty for
anything else.

[Darton]

@jpollard: Agreed. One quick comment on the difference in file transfer rates:

The vdi may have a sparser data density than the avi (i.e. it is more compressible), which could affect the speed. A better comparison would be two files of the same type with different sizes.

[pythagorean]

Thanks for the posts. I am new to networking really. Hitherto I had just bought a router, plugged in the ethernet cables and used the network without giving it much thought. Its only now that I have taken an interest in the network that I have started to wonder about how it behaves.

I am going to try transfers with different sizes of AVI files tonight to see if the network reports different speeds.

---------- Post added at 10:38 PM ---------- Previous post was at 11:16 AM ----------

The results of my experimentation with my D Link DIR-655 router and Belkin USB gigabit adapter:

The biggest files I have are virtual disk images that Virtualbox creates when you make a virtual machine. My Ubuntu Lucid vdi is slightly larger than 4 GB.

Transferring it over the network to another desktop with a built in gigabit ethernet card achieved 41 GB per second maximum.

Transferring it to a laptop that just has a 10/100 ethernet card achieved 6.5 GB per second max.

Transferring to a the same laptop as above using wireless N with the router set at 20/40 MHz achieved a 3 GB per sec max whereas the same computer with the router set at 20 MHz per sec only got up to 1.3 GB per second and with router set to "g only" the transfer achieved 900 KB per sec.

I am wondering if these results are good or bad... ?


What do other people get?

[DBelton]

those rates don't sound even close to being correct.

on a 100mb lan, you will only get a max of around 12.5MB/sec
on a gigabit lan, you will only get a max of around 125MB/sec and you may not even get that depending on your hard drive speed.

using wireless N, you won't get more than about 600mbit/sec and that is only if you are using 4 streams @ 40mhz. that would be a maximum throughput of about 75MB/sec. You may not even get close to that since you are using 2 different brand router/network card.

[pythagorean]

125 MB/s is a THEORETICAL maximum. I have never seen anyone report higher than 65 MB per second in my researching on the internet about how the network should perform, and even 65 MB/s is going to result from RAM to RAM transfer. I don't think the hard drive itself ever gets over 90 MB/s even though the SATA connection is supposedly able to deliver 3GB/s.

It turns out that different types of files transfer at different rates on my network. I just moved the 700 MB Fedora 14 ISO from one computer to another using gigabit ethernet (with the Belkin gigabit ethernet USB) and I got 8.4 MB/s tranfer whereas my 14 GB music folder only ever got up to 5.37 MB/s on the same network with the same devices.

I am interested in reading a discussion about these topics. I am not achieving Gigabit speeds with the Belkin USB Gigabit ethernet device but it is faster than what I was getting with the laptops 10/100 built-in ethernet.

[jpollard]

I have seen 80MB/sec on a gigabit network... though that was a research
ATM network.

It takes custom configuration of both source and destination. The source
to make sure the data is always available, and the destination to make
sure there is always buffer space for the data.

Getting the data into memory at the source always took a "precaching"
approach (we had 100GB (out of 256GB) of memory to work with, so
loading relatively large files into memory was possible, and another
100GB on the data sink (another big system).

Filesystem loads/disk delays... not a problem - everything was raid 5
(64k per block I think it was) on both ends.

This was back when GBit networks were new.

One of the things used to get high throughput was to use jumbo packets
and large amounts of data in a streaming mode. This reduces the round
trip overhead, plus the source and destination systems were directly
connected (crossover cable), so no router delays (causing fragmentation
delays) were involved.

There are very few home level routers that can truly handle Gbit networks
at wire speed. Most have some buffering involved, or cause
routing delays for other reasons.

Home routers are cheaply made trying to reach a low cost. The
high performance routers are NOT cheap. At work, the cheapest router
we had (at full speed) was about $15K US.

[pythagorean]

Very interesting points jpollard.

I actually decided the Belkin USB Gigabit adapter wasn't working properly, because 5 MB/s transfer speed is ridiculous. I am transfering my 54 GB Video folder from a Windows 7 machine (gigabit ethernet card) to my Ubuntu 10.04 laptop, which only has a 10/100 ethernet card, and I am getting 11.2 MB/s. The same transfer with the Belkin only got up to 5 MB/s or so.

I was using this little ubuntu laptop as a sort of NAS device but now I think the data transfer speeds are too low for it to serve this purpose in the long run.

[jpollard]

That 11.2 MB/sec transfer should translate into 89.6 Mbit rate, which
isn't too bad for a 100 Mbit interface.

The way I translate speeds-
1) identify the slowest network interface in the line between source and
destination
2) speed/8 - raw transfer in bytes, subtract 5-10% for extra overhead.

100 Mbit interface would be 12.5 Mbyte (100/8). multiply by .9 to account
for overhead -> 11.25.

This works best for ethernet. Other hardware level protocols have different
amounts of overhead. One advantage ethernet has had is that the protocol
is very flexible about the overhead. there are lots of ways to implement it:

1)buffering/forwarding packets
2)buffering only the header, process routing needs, output new header..followed
by the rest of the data at wire speed (this trusts the hardware input queue for
the buffering required)
3)process routing needs at wire speed..(the input queue is modified on the fly)

1 is the cheapest router. Works, is very flexible, cheap, and can be implemented
in almost anything with two or more generic network interfaces. Any checksums
are computed by the software.
2 requires some hardware assist - not as cheap, and not as flexible, but almost
as fast as 3. checksum alterations may be done by software assisted by hardware.
3 is the expensive one, and is not flexible. Any changes usually require a minimum
of microcode changes, and sometimes ASIC changes, or both. Checksums are
handled by hardware.

Fragmentation/defragmentation in 2/3 is usually done by falling back to #1. But the higher priced
Cisco routers can (I think) do it on the fly (I'm thinking of that Tbit jumbo job here, but have
no experience with it).