Hi all, I am very new to OOD, so please bear with me if the question below has a trivial answer.
Is there a way to provide customized Python environment as the kernel of the Jupyter notebook inside the OnDemand interface? I’m thinking specifically for a containerized Python–it could also include the Jupyter server process itself. Case in point: Some users want to use containerized software provided by NVIDIA (NGC) on our machine. So at the very least the Python needs to be run inside the container. In my mind there are two possible scenarios:
Have everything (notebook server, ipython, and kernel) running inside the container
Or have notebook server running outside the container, but then talk to ipython & kernel inside the container.
The second option seems to be the minimum so that all the capabilities of the software inside the container can be utilized.
The short answer is, yes this is absolutely possible. OOD just launches jobs and tries to proxy web request. It doesn’t know or care how that webserver got started.
The first option could be OK, but you may get into an uber-container situation where folks just keep needing some package installed in it.
The second option sounds nicer, because it’s more flexible. It’s either going to fairly easy to manage (the launcher scripts are just like ours, only they run singularity exec ... instead of module loading and calling ipython) or a nightmare where the python kernel needs to talk to the host /proc filesystem or needs some special capability or it’s just being contained in some way that makes it very hard to communicate with the host processes (and/or filesystem).
This approach is a lot like modules, only they’re containers instead. The tricky bit is getting them all to play nice with each other, because after all, they’re contained process’ that aren’t supposed limit themselves.
We’re using Singularity quite successfully on our HPC. One solution would be to have Python or the Python modules installed outside of the container and simply binding them into the container and adjusting the PYTHONPATH appropriately.