[105157] in North American Network Operators' Group
Re: Best utilizing fat long pipes and large file transfer
daemon@ATHENA.MIT.EDU (Sean Knox)
Fri Jun 13 13:44:03 2008
Date: Fri, 13 Jun 2008 10:43:23 -0700
From: Sean Knox <sean@craigslist.org>
To: nanog@merit.edu
In-Reply-To: <4851A53B.4010400@craigslist.org>
Errors-To: nanog-bounces@nanog.org
Many thanks for great replies on and off-list.
The suggestions basically ranged from these options:
1. tune TCP on all hosts you wish to transfer between
2. create tuned TCP proxies and transfer through those hosts
3. setup a socat (netcat++) proxy and send through this host
4. use an alternative to plain netcat/scp for large file transfers
My needs are pretty simple: occasionally I need to push large database
files (300Gb+) around linux hosts. #4 seems like the best option for me.
People suggested a slew of methods to do this: RBUDP, gridftp, bbcp, and
many others, with programs either sending with reliable UDP or breaking
large transfers into multiple streams. Because it was easy to use right
away, I tried RBUDP and was able to copy a tarball at about 700Mb/s over
a 20ms delay link, and when factoring in destination disk write speed,
isn't too bad a starting point. GridFTP and bbcp look very useful too;
I'll be exploring them as well. The presentation links Kevin O. sent
were very interesting.
I've looked at HPN-SSH before but haven't played with it much. I'll
definitely try it out based on the feedback from this thread.
Thanks again.
sk
Sean Knox wrote:
> Hi,
>
> I'm looking for input on the best practices for sending large files over
> a long fat pipe between facilities (gigabit private circuit, ~20ms RTT).
> I'd like to avoid modifying TCP windows and options on end hosts where
> possible (I have a lot of them). I've seen products that work as
> "transfer stations" using "reliable UDP" to get around the windowing
> problem.
>
> I'm thinking of setting up servers with optimized TCP settings to push
> big files around data centers but I'm curious to know how others deal
> with LFN+large transfers.
>
> thanks,
> Sean
