Hi All, I have a question with regards to file transfer speeds. What is the protocol used when uploading/downloading files through OOD? We’re transferring large data sets in the range of 400GB plus and we’re seeing 12hrs or more to transfer from HPC to workstation. If we setup an SCP to move the files we can get them downloaded in roughly an hour. Is this typical? Thanks in advance, Kyle
I’ll let others speak to the specific protocol, but want to chime in that at OSC, we limit our OOD file transfers to about a max of 9GB in size. I think this is largely based upon the available disk space in the underlying VM. We also have dedicated data transfer nodes for scp file copies. In testing we’ve found that file transfers via OOD at OSC is typically about 2x as fast as via command line scp.
It’s always http(s). And that slow of a speed for that large of a file/data set is probably typical.
SCP just has fewer layers than we do and likely has optimizations in it for this (clearly).
Thanks Jeff, appreciate the info.
A few relevant details: the files app as of OnDemand 1.8 uploads the file to /tmp on the OnDemand host, and then after the upload completes, moves the file to the destination. So there are several things that can impact the speed of the “file upload”.