Hello, i couldn’t find any related topic or possible solution to my problem and maybe others also running into this issue.
I am trying to give our user’s the oppurtunity to start from inside their own individual conda environments.
But this always fails due to the reason that the conda environment can’t be activated as there appears:
Script starting...
Waiting for Jupyter Notebook server to open port 34311...
TIMING - Starting wait at: Mon Nov 20 17:41:36 CET 2023
++ echo pytorch
pytorch
++ set -x
++ conda --version
conda 4.10.3
++ conda activate pytorch
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run
$ conda init <SHELL_NAME>
Currently supported shells are:
- bash
- fish
- tcsh
- xonsh
- zsh
- powershell
See 'conda init --help' for more information and options.
IMPORTANT: You may need to close and restart your shell after running 'conda init'.
My template conda.sh.erb for the conda enviroments is this one:
#!/usr/bin/env base
set -x
echo "$CONDA_NAME"
<%- if context.sub_type == "conda_env" -%>
set -x
conda --version
#conda init bash
conda activate $CONDA_NAME
<%- end -%>
jupyter () {
"$CONDA_PATH"/jupyter lab --config="${CONFIG_FILE}"
}
export -f jupyter
#conda init bash
#conda activate $CONDA_NAME
relevant parts of form.yml.erb
conda_path:
label: Conda Enviroment
help: |
Select a Conda Enviroment : path to the **/bin folder that includes Jupyterlab**. This is required when using a Conda Submission Environment! Use **pip install jupyterlab** inside your conda enviroment.
cacheable: false
data-filepicker: true
data-show-hidden: true
data-target-file-type: dirs
data-target-file-pattern: 'bin'
readonly: true
conda_name:
widget: "text_field"
label: "Name of your conda Env"
value: "envXY"
help: |
- Name of your custom conda enviroment **required !**
So the path is used to invoke the jupyter-lab from the /bin path of the conda environment and the conda name is used to activate the conda environment.
I have also a classical Jupyterhub running in combination with the batchspawner which is using sudo -E to run the srun command to run the jupyterlab from inside the users conda environment in this setup the activation of the shell is working.
Instead of having the app boot up the conda environment, which will be different for all users and potentially even not relevant for some - I would suggest instead that you direct your users to turn their conda environments into Jupyter kernels.
That way it’s the best of both worlds - conda users get to see & use their environments while the rest of the users aren’t pushed into conda environments. What’s more is that this opens up the possibility to use other python packaging systems like venv too.
Here’s the OSC documentation for the same.
We provide some helper scripts, which looking at them have lots of toggles/helper stuff (like creating a kernel from a venv as well!) but I think this article or just googling jupyter create kernel from conda env will turn up what you’re looking for.
And that’s not all! Additionally, when pushing users to create and manage their own kernels, you get
fewer options in the form
users don’t have to restart the job to change conda environments (kernels) - they can just choose another kernel
Jupyter has all kinds of support for other languages. Notably, Julia, through you see I have a Ruby kernel that’s my own (not system installed).
Here I’ve highlighted my own conda kernel and my Ruby kernel. These are both my own, they’re not system installed/available for other users. (I think Julia 1.8.5 is also my own - i.e., not system installed, but didn’t check),
Yes that’s right. Here’s our production application below. jupyter itself is from a module but all the users’ kernels are in their $HOME (as seen below). This is a standard path that juypter will automatically recognize.
Hey back again, i found a probably very simple way to solve the conda init problem:
conda.sh.erb
#!/usr/bin/env base
set -x
echo "$CONDA_NAME"
<%- if context.sub_type == "conda_env" -%>
set -x
conda --version
#conda init bash
eval "$(conda shell.bash hook)"
conda activate $CONDA_NAME
<%- end -%>
jupyter () {
"$CONDA_PATH"/jupyter lab --config="${CONFIG_FILE}"
}
This is the line which now let’s recognize the right conda shell.
eval “$(conda shell.bash hook)”
So instead of starting from a centralized path now every jupyter notebook/lab gets started from inside of the users conda environment the only thing needed is that they install jupyterlab inside thei environment.
Maybe that could be interesting for other sites, too.
