Hello everyone, I’m building a hybrid system using OOD + Slurm that will spin up EC2 instances when jobs are submitted. Normally, on-premises servers share data via NFS, but using NFS over a Site-to-Site VPN is extremely slow. I use a lot of libraries like anaconda3, etc., so using NFS seems infeasible. Is there any way to work around this so that EC2 instances can sync data with the on-premises environment?
Or is this architecture actually viable at all? Or do I need to migrate everything fully to the cloud?
This is more a Slurm topic than an OOD topic, but it’s definitely viable. There are at least a few conference talks floating around from groups that have done this.
The one I have in my browser history is the HPC Cloud Bursting talk by Dacian Reece-Stremtan at GWU.
Yea I’m not sure there’s a 3rd option here, or maybe I’m just not imaginative enough. You need all these libraries there, so they either already have to exist there or they exist over the network from your data center.
AWS gave a great presentation at the March 2023 Tips and Tricks call that might be helpful as well, which can be viewed here: March 2 2023 - Google Drive
We have a great relationship with them and would be happy to put you in touch with somebody on their technical team.