Hello! I’m currently researching to onboard CryoSPARC to an OOD application. To get started, I’m wondering if anyone has a working CryoSPARC application on their site and is willing to share? If not, any advice on navigating this would also be appreciated.
Hi!
I know for sure someone has tried. I’d try searching this forum for CryoSPARC to see what folks have come up with and if they have indeed succeeded in this.
1 Like
Several people have indeed developed CryoSPARC workflows. The one I know the most about is from Ecosystem for Research Networking. We have a whole case study on them at Ecosystem for Research Networking develops platform for remote scientific instrument control | Open OnDemand and they’ve published several papers:
1 Like
I was the engineer on the ERN project. Unrelated to the ERN setup, I have an app that utilizes the virtual desktop (via noVNC) base app and per user installed cryosparc installs. The app starts up a virtual desktop and then starts cryosparc, it sleeps for a few while cryosparc gets up and running and then launches a firefox window that connects to the localhost port that cryosparc starts on.
This allows you to spin up cryosparc on compute nodes like you would any other ood app. There are some complexities that I have solved for like some basic math based on the users UID to choose random ports so installs do not clash and the port that cryosparc listens on is predictable.
I can probably put together a sanitized tar ball of the app and the per user installation script.
Unfortunately due to (imo) a design flaw with the Cryosparc web framework, we can’t just reverse proxy via OOD like Jupyter and RStudio do, hence the virt desktop need.
1 Like
Hi Morgan, thanks for the response! I did originally think that we would be able to launch Cryosparc just like Jupyter and RStudio before reading the case study. It would be extremely helpful if you could provide the app and script you mentioned!
I just bundled the installer script and the ood app and stuck it here: GitHub - mjbludwig/cryosparc_ood: Cryosparc installer and OOD app for launching on compute nodes via OOD novnc XFCE virtual desktop
I didn’t go as far as making a README. Things it expects are:
- Cryosparc License
- Cryosparc source tar balls for both master and worker
- XFCE virtual desktop requirements on compute nodes
- install cryosparc in /home/$USER and that an NFS /scratch/$USER exists (although you can change those locations as needed)
- An environment module for Cuda and Firefox
- Slurm as the scheduler
Files you will likely need to tweak and/or should be read through and understand:
- install_cryosparc.sh
- cryosparc/form.yml
- cryosparc/submit.yml.erb
- cryosparc/template/clean.sh
- cryosparc/template/script.sh.erb
cryosparc/template/script.sh.erb specifically does most of the heavy lifting. I sanitized this so again, read through these files and understand what they do because there are some places you will need to edit.
For context, the reason we can’t just proxy it is because Cryosparc always assumes its running at the root of its FQDN so when URI’s are changed via a proxy, it breaks the routing for its Javascript bits and other assets. At one point Jeff and I had looked into using Apache’s mod_substitute which allows you to rewrite HTML bits on the fly, i.e. tweak the pathing on the way out and the way in to match what Cryosparc is expecting but the version of that module for Apache 2.4 is missing some features that would be needed. I think the version in 2.5 might have them but until the industry moves to 2.5, we aren’t going to look into it.
Another thing to keep in mind is that Cryosparc is pretty clumsy on shutting down and if you don’t use its shutdown command and try to just kill the procs (like what happens when slurm jobs hit their walltime), it leaves around lock files which will then prevent Cryosparc from opening up again on the node. Why they don’t just do that when the process’s get a SIGTERM is beyond me but given other issues I have found with it, not surprising… To mitigate this, I have a section at the end of the main script that queries squeue every so often and if it sees the walltime is getting close, it initiates the shutdown procedure.
For good measure, I also have the clean.sh script which OOD looks for in any apps template dir that gets triggered when the user shuts down an app session in the interactive apps section.
Also note that I start firefox in kiosk mode so it kind of obscures the fact you are in a virt desktop but you can just remove that flag if you want the full desktop functionality.
I think this is enough to get you going but let me know if you have any questions. Hopefully this thread will what folks in the future find when they come looking. Not the first time I have shared this setup!
1 Like
Thanks a lot! We’re making some good progress with running the CryoSPARC master and worker on the same compute node. That will really help us out.
What are some challenges with using OOD to run the master and worker on separate nodes? Our old hacked-together bash scripts ran the master and worker on separate nodes, and there was some fragile code to discover the hostnames and available ports to communicate between master and worker.
OOD makes it easy to discover the host to run the master; and a port to connect from OOD to the master (with the $host and $port variables). So I guess my question is: does OOD provide an easy way to discover other nodes’ hostnames in the job and other available ports to communicate betwen nodes?
No, but most schedulers will tell you the node list of the job. In Slurm it’s SLURM_JOB_NODELIST.
As to ports, that’s a bit trickier since they’re not real until they are. I’d suggest using hard coded values to get going. Beyond that - who knows, you’d likely have to do something pretty complicated like all scripts write to (append to) a single file during the job, then the master node waits until it sees N lines in that file (one for every node)?
1 Like
Thanks. We’ve got some ideas along those lines, too. Just wanted to make sure we weren’t re-inventing the wheel before we went any further
Cryosparc does hard code the ports per install (although I think there is a cryosparcm command now to change which port it is). The problem is that Cryosparc was designed to talk to the scheduler itself so by nature it is out of scope of how OOD works with the scheduler. I get why they tied the scheduler bits to close into how Cryosparc launches its workers but it does make Cryosparc kind of in flexible.
You might be able to go with a mixed solution though where OOD launches the master onto a compute node and then you configure lanes in Cryosparc to talk directly to Slurm where it can then submit jobs for its workers. At that point though, it would almost be worth just spinning up a Cryosparc intended head-node where users can spawn their master processes and have them talk directly to Slurm, i.e. totally work outside of OOD.