Jupyterlab and conda (conda init fail's)

Get Help
1

Hello, i couldn’t find any related topic or possible solution to my problem and maybe others also running into this issue.

I am trying to give our user’s the oppurtunity to start from inside their own individual conda environments.
But this always fails due to the reason that the conda environment can’t be activated as there appears:

Script starting...
Waiting for Jupyter Notebook server to open port 34311...
TIMING - Starting wait at: Mon Nov 20 17:41:36 CET 2023
++ echo pytorch
pytorch
++ set -x
++ conda --version
conda 4.10.3
++ conda activate pytorch

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

My template conda.sh.erb for the conda enviroments is this one:

#!/usr/bin/env base
set -x 
echo "$CONDA_NAME"
<%- if context.sub_type == "conda_env" -%>
set -x
conda --version
#conda init bash
conda activate $CONDA_NAME
<%- end -%>

jupyter () {
	"$CONDA_PATH"/jupyter lab --config="${CONFIG_FILE}"
}
export -f jupyter
#conda init bash
#conda activate $CONDA_NAME

relevant parts of form.yml.erb

conda_path:
    label: Conda Enviroment
    help: |
      Select a Conda Enviroment : path to the **/bin folder that includes Jupyterlab**. This is required when using a Conda Submission Environment! Use **pip install jupyterlab** inside your conda enviroment.
    cacheable: false
    data-filepicker: true
    data-show-hidden: true
    data-target-file-type: dirs
    data-target-file-pattern: 'bin'
    readonly: true
  
  conda_name:
    widget: "text_field"
    label: "Name of your conda Env"
    value: "envXY"
    help: |
      - Name of your custom conda enviroment **required !**

So the path is used to invoke the jupyter-lab from the /bin path of the conda environment and the conda name is used to activate the conda environment.

I have also a classical Jupyterhub running in combination with the batchspawner which is using sudo -E to run the srun command to run the jupyterlab from inside the users conda environment in this setup the activation of the shell is working.

Thanks in advance

Best Regards
Kreefd

2

Instead of having the app boot up the conda environment, which will be different for all users and potentially even not relevant for some - I would suggest instead that you direct your users to turn their conda environments into Jupyter kernels.

That way it’s the best of both worlds - conda users get to see & use their environments while the rest of the users aren’t pushed into conda environments. What’s more is that this opens up the possibility to use other python packaging systems like venv too.

Here’s the OSC documentation for the same.

We provide some helper scripts, which looking at them have lots of toggles/helper stuff (like creating a kernel from a venv as well!) but I think this article or just googling jupyter create kernel from conda env will turn up what you’re looking for.

3

And that’s not all! Additionally, when pushing users to create and manage their own kernels, you get

  • fewer options in the form
  • users don’t have to restart the job to change conda environments (kernels) - they can just choose another kernel
  • Jupyter has all kinds of support for other languages. Notably, Julia, through you see I have a Ruby kernel that’s my own (not system installed).

Here I’ve highlighted my own conda kernel and my Ruby kernel. These are both my own, they’re not system installed/available for other users. (I think Julia 1.8.5 is also my own - i.e., not system installed, but didn’t check),

image
image1791×853 66.3 KB

4

So you mean every user can have their individual kernels while the jupyter or jupyterlab is started from an shared instance/path ?

To be concret how do you start the jupyterlab for the users ? Through modules from an shared path ?

Thanks in advance

5

Yes that’s right. Here’s our production application below. jupyter itself is from a module but all the users’ kernels are in their $HOME (as seen below). This is a standard path that juypter will automatically recognize.

image
image858×116 11.7 KB