Jupyter notebook interactive app via conda environments


We have been getting OnDemand rolled out and find it to be a great software so far and are now starting to work on integrating Jupyter notebook via an interactive app but have some questions on the best way to implement this. Currently our users use Jupyter notebook by installing the package inside their own conda environments and launching from there.
However this does not seem to work too well with the way Jupyter notebook is launched via OnDemand since the launch command does not activate the conda environment where it is installed. This means the launch fails due to not finding the command.

My question is whether launching an interactive app from a conda environment is possible or if OnDemand expects an installed version loaded via module. If it is possible with conda environments, is there documentation that can be referenced to work this out?

Thank you all for your help!

Launching from a conda environment is definitely possible, just probably not advisable. OnDemand doens’t expect anything. It comes down to the what’s in the template/script.sh.erb. You can do all sorts of user environement setup in that shell script, because it is going to be running in your scheduler as that user so it has access that user’s home directory.

Have you seen the jupyter we run at OSC? So we load the module python/3.6-conda5.2 and that gives us /usr/local/python/3.6-conda5.2/bin/jupyter where someone has presumably pip installed it as root (I’m not 100% sure how our module installations happen)?

I’d suggest doing the same/similar. It gives the best ‘it just works’ experience to your users. Of course you can always configure the script to boot $HOME/.local/bin/jupyter, but again, I think it “just working” without users having to install things and do prep work is the best experience.

Now you may notice a lot of searching conda environments in how we load kernels, and I can (sorta) answer questions on all that - but essentially that provides functionality to load different python/IJUlia kernels if the user has installed them.

Hi Jeff, I took your advice and did a pip installation with one of our python modules.
This then allowed the app to start as expected and I was able to open up notebook. I will continue testing but wanted to report back and thank you for your help. :smile:

Hi @jonathanc,

Check this jupyter app out: https://gitlab.com/mjbludwig/jupyter_experimental

I have done a bunch of customization and javascript to load various anaconda installs (available as environment modules/lmod) then allows users to specify the path to a specific conda env if needed.

Its not the cleanest so let me know if you have questions on it!

We do Jupyter from Conda and it works great. We package the Python environment into an Lmod module, and stay away from Virtual Environments for several reasons listed below.

One of the caveats is that the virtual environments do have problems, since you need to register each virtual environment into the Jupyter separately (see the last paragraph of https://www-test.chpc.utah.edu/documentation/software/jupyterhub.php).

In case of OOD, we’d have to have 2 Lmod modules, one to load the base Anaconda and other to load the virtual environment. I consider that to be a hassle (the VE initialization does plug into the module syntax nicely), our users would need to get educated to write the VE module, or they would have to activate the VE in the OOD webform. Plus, I have seen problems with that before which details I don’t remember, so, my recommendation is to have multiple Anaconda (miniconda) installations, one for each VE you’d want to run, each loadable by separate Lmod module, https://www.chpc.utah.edu/documentation/software/python-anaconda.php#mi. The miniconda install is simple as is a modification of its Lmod module.

Then, in OOD, we have a text box that allows users to put in any environment commands they want (credit to Yale for giving us that suggestion), https://www.chpc.utah.edu/documentation/software/ondemand.php#oodjupyter.

Our OOD app is at https://github.com/CHPC-UofU/bc_osc_jupyter

1 Like

I will take a look at these two other repos and try and adapt it to our environment. Thank you all!

Although it is not directly related to Jupiter notebook, may I know the way to launch Rstudio with conda environment? I will need to use specific R packages that were installed in the conda environment (thus have better version control). I am not sure how to install specific packages without conda. Thanks!