Quick Links

Technology

fasp vs. FTP

fasp vs. FTP on TCP Tuned for High Bandwidth-High Delay Networks

The throughput of FTP on links that have both high bandwidth and high latency can be improved by "tuning" the operating system parameters used by the TCP networking stack on the file transfer endpoints. The tuning usually involves increasing memory buffer sizes such that TCP can use the large data windows necessary to fill a high bandwidth - high delay link. TCP uses the concept of "windows" to throttle the amount of data the sender injects into the network, depending on the receiver's available capacity and the other link traffic. A window is the amount of new, unacknowledged data that TCP allows to be in flight between sender and receiver. In order to keep a link filled to capacity, TCP must use a window size equal to the bandwidth times the round-trip latency of the link, sometimes referred to as the "Bandwidth Delay Product".

The Bandwidth Delay Product of a very high bandwidth local link or a high bandwidth WAN link can be very large and exceed the default TCP buffer sizes used by the operating system on the file transfer endpoints. For example, a local gigabit ethernet link with 10 ms latency has a Bandwidth Delay Product of 1.2 megabytes: 1,000,000,000 bits/s * 0.010 seconds = 1.2 megabytes. A 45 Mbps cross-country link with 90 ms latency has a Bandwidth Delay Product of 494 kilobytes. In contrast, the default TCP buffer size on most operating systems is much less, usually 64 kilobytes. By raising the default buffer size to the Bandwidth Delay Product, in theory TCP will fill the link to capacity, and an ideal FTP file transfer will run at link speed. However, in practice, packet loss on the link will cause TCP to reduce its instantaneous window size dramatically, and reduce throughput below link capacity. The result is that tuning TCP improves FTP throughputs on local links where loss is low but has almost no impact on FTP throughputs over wide area links having some packet loss.

The underlying problem is that standard TCP is overly conservative in the face of normal packet loss for high bandwidth or high delay links. When TCP experiences any packet loss, independent of the cause, TCP assumes that network congestion is pending and applies a congestion avoidance algorithm. TCP reduces the window size exponentially and then slowly reopens the window to increase rate. On high-bandwidth or high-delay links, where relatively more packets are traversing more network nodes, some loss due to random error is normal, yet TCP assumes congestion and throttles the rate down. The higher the Bandwidth Delay Product, the smaller the packet loss tolerated before TCP throughput falls well below link capacity.

The graph above compares FASP, FTP over standard TCP, and FTP over TCP tuned for the Bandwidth Delay Product with all optimizations enabled, such as selective acknowledgments. When packet loss is 0.1% (one packet loss per 1000), tuning TCP improves FTP throughput (21 to 38 Mbps), but FTP is still 70 times slower than FASP (238 Mbps). On WAN links where packet loss is higher, tuning TCP has almost no impact on FTP throughput. In a coast-to-coast transfer (90ms/1% loss), Tuned FTP and standard FTP both run at about 1.5 Mbps. In this case the loss dominates, sending TCP into congestion avoidance and dramatically reducing throughput.

 

Next Page