When downloading a folder larger than 10737418240 bytes, the following error message appears: “The directory is too large to download as a zip. The directory should be less than 10737418240 bytes.” Is it possible to increase this limit? I already changed the download limit in nginx_stage.yml.
I’ll let one of my colleagues chime in as to whether it’s technically possible, but from a practical standpoint, there are numerous reasons you really shouldn’t increase that. The limit is ~10GB.
There are a LOT of various elements in the workflow/pipeline between the physical files on the host storage device and the resulting files on the client’s storage device that are involved in transfers, not the least of which are various caches / memory buffers. If you allow really large files to be transferred and multiple clients end up doing that simultaneously, you can quickly and inadvertently overwhelm those elements, causing crashes and unexpected results.
Our general recommendation is to utilize OOD only to transfer files up to a few GB in size. If you want to transfer more data, you should consider using something like Globus (which will be somewhat integrated into the file browser in release 3.1, coming soon), rclone (which is already available) or breaking the data up into more manageable chunks.
I’d second what @alanc said if you are regularly downloading files of large sizes. Better to have the right tool for the job, and though OOD can handle some of this, it would be better to use tools designed to move large datasets if needed.
Technically, that sounds like the default value being hit, are you sure you set the value as instructed here?
https://osc.github.io/ood-documentation/latest/customizations.html?highlight=upload#set-download-limits
This wouldn’t involve you touching anything in nginx_stage per-se, but only adding an .env file to then set the environment variable which NGINX will use.
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.