Hello! I’m currently researching to onboard CryoSPARC to an OOD application. To get started, I’m wondering if anyone has a working CryoSPARC application on their site and is willing to share? If not, any advice on navigating this would also be appreciated.
Hi!
I know for sure someone has tried. I’d try searching this forum for CryoSPARC to see what folks have come up with and if they have indeed succeeded in this.
1 Like
Several people have indeed developed CryoSPARC workflows. The one I know the most about is from Ecosystem for Research Networking. We have a whole case study on them at Ecosystem for Research Networking develops platform for remote scientific instrument control | Open OnDemand and they’ve published several papers:
1 Like
I was the engineer on the ERN project. Unrelated to the ERN setup, I have an app that utilizes the virtual desktop (via noVNC) base app and per user installed cryosparc installs. The app starts up a virtual desktop and then starts cryosparc, it sleeps for a few while cryosparc gets up and running and then launches a firefox window that connects to the localhost port that cryosparc starts on.
This allows you to spin up cryosparc on compute nodes like you would any other ood app. There are some complexities that I have solved for like some basic math based on the users UID to choose random ports so installs do not clash and the port that cryosparc listens on is predictable.
I can probably put together a sanitized tar ball of the app and the per user installation script.
Unfortunately due to (imo) a design flaw with the Cryosparc web framework, we can’t just reverse proxy via OOD like Jupyter and RStudio do, hence the virt desktop need.
1 Like
Hi Morgan, thanks for the response! I did originally think that we would be able to launch Cryosparc just like Jupyter and RStudio before reading the case study. It would be extremely helpful if you could provide the app and script you mentioned!
I just bundled the installer script and the ood app and stuck it here: GitHub - mjbludwig/cryosparc_ood: Cryosparc installer and OOD app for launching on compute nodes via OOD novnc XFCE virtual desktop
I didn’t go as far as making a README. Things it expects are:
- Cryosparc License
- Cryosparc source tar balls for both master and worker
- XFCE virtual desktop requirements on compute nodes
- install cryosparc in /home/$USER and that an NFS /scratch/$USER exists (although you can change those locations as needed)
- An environment module for Cuda and Firefox
- Slurm as the scheduler
Files you will likely need to tweak and/or should be read through and understand:
- install_cryosparc.sh
- cryosparc/form.yml
- cryosparc/submit.yml.erb
- cryosparc/template/clean.sh
- cryosparc/template/script.sh.erb
cryosparc/template/script.sh.erb specifically does most of the heavy lifting. I sanitized this so again, read through these files and understand what they do because there are some places you will need to edit.
For context, the reason we can’t just proxy it is because Cryosparc always assumes its running at the root of its FQDN so when URI’s are changed via a proxy, it breaks the routing for its Javascript bits and other assets. At one point Jeff and I had looked into using Apache’s mod_substitute which allows you to rewrite HTML bits on the fly, i.e. tweak the pathing on the way out and the way in to match what Cryosparc is expecting but the version of that module for Apache 2.4 is missing some features that would be needed. I think the version in 2.5 might have them but until the industry moves to 2.5, we aren’t going to look into it.
Another thing to keep in mind is that Cryosparc is pretty clumsy on shutting down and if you don’t use its shutdown command and try to just kill the procs (like what happens when slurm jobs hit their walltime), it leaves around lock files which will then prevent Cryosparc from opening up again on the node. Why they don’t just do that when the process’s get a SIGTERM is beyond me but given other issues I have found with it, not surprising… To mitigate this, I have a section at the end of the main script that queries squeue every so often and if it sees the walltime is getting close, it initiates the shutdown procedure.
For good measure, I also have the clean.sh script which OOD looks for in any apps template dir that gets triggered when the user shuts down an app session in the interactive apps section.
Also note that I start firefox in kiosk mode so it kind of obscures the fact you are in a virt desktop but you can just remove that flag if you want the full desktop functionality.
I think this is enough to get you going but let me know if you have any questions. Hopefully this thread will what folks in the future find when they come looking. Not the first time I have shared this setup!