Five times faster TCP speed on high latency high bandwith connections

All right, it took me quite some time to figure this out so I will just give you some background to see if you are in the same situation, then the solution.

My home connection in Singapore is a 100/10 Mbit/sec cable line, speedtest.net measures an average 10ms ping, 103.8Mbit/s (12 682 KiB/s) download and 5.9 Mbit/s (714 KiB/s) upload speed, which is kind of a decent connection (there is 500Mbps fiber available to those who really need it. :)

Ther server and network I would like to access is in Hungary, on a 120/20 Mbps cable, so obviously that 20Mbit/sec uplink should be limiting my download speeds at around 2441 KiB/s=20*1000*1000/1024/8.

I did test the intercontinental connection speed with speedtest.net:

•  Magyar Telecom, Vodafone and Telenor servers
• SG daytime and SG late evening HU daytime as well

The daunting average numbers (average of 3 or 5 for each server-daytime combination): 411ms ping (RTT), 1114 KiB/s download, 168 KiB/s upload. Also there is a huge variance even in the short term, and a large drop when both time zones are in daytime.

I have set up OpenVPN but could not get anything more than 175 KiB/s (neither using SMB nor pure FTP transfer).

Next tried SCP and SFTP without VPN: the best I could get was a much favorable 700 KiB/s. (Not CPU bound, even the far side Linux on and Atom CPU was only used 10%) That however is a protocol most movie players will not stream over. :)

Then I figure I will need to go rouge (I mean raw) and have set up iperf (and the necessary port forwarding on both sides). On the first run with default parameters I got back to results around 175 KiB/s!

After tuning TOS, increasing send buffer and TCP window size I was able to get it up to nearly 512 KiB/s.Clearly, something was still limiting it.
Windows accepted any window size I threw at it, but Linux maxed out at 320KiB for some reason. ("Should be enough for everyone!" some Linux believers might scream.)

But in fact calculating the Bandwidth Delay Product (see iperf page) for the six intercontinental per server averages I get, e.g. to Vodafone server in the evening 1235 KiB/s * 0.516sec (RTT) = 637 KiB of data can be in flight!

Then I just had to look up these network tuning parameters:
net.core.netdev_max_backlog = 5000
net.core.wmem_max = 12582912
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 10240 87380 12582912
net.ipv4.tcp_wmem = 10240 87380 12582912
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
Which you should put into /etc/sysctl.conf, then run 'sudo sysctl -p', and set TCP window size to 640KB (that should be enough for everyone!), voila:
[  3] 10.0-20.0 sec  9.75 MBytes   998 KBytes/sec
[  3] 20.0-30.0 sec  9.75 MBytes   998 KBytes/sec
[  3] 30.0-40.0 sec  9.62 MBytes   986 KBytes/sec
[  3] 40.0-50.0 sec  9.44 MBytes   966 KBytes/sec
[  3] 50.0-60.0 sec  9.88 MBytes  1011 KBytes/sec
[  3] 60.0-70.0 sec  9.56 MBytes   979 KBytes/sec
[  3] 70.0-80.0 sec  9.38 MBytes   960 KBytes/sec
[  3] 80.0-90.0 sec  9.38 MBytes   960 KBytes/sec

I admit I did not do a lot of further testing and tuning, since I got pretty close to the average value that was achievable according to the intercontinental speedtest measurements. :)

ToDo: 8 and 16 parallel TCP streams can rob more bandwidth and perform at a total of 1.2-1.3 MiB/s, so there is some more space for tuning.
But that may also just be fooling the rate limiting algorithms for a bit longer in the several routers between the endpoints, also streaming a single movie over multiple TCP streams is not feasible, so I think I am pretty much done for now. :)

EDIT: Another article about Windows 7 TCP tuning suggests:
netsh int tcp set global congestionprovider=ctcp
For better broadband utilization. :)