Okay, so I’m not sure how on topic this is for OOD, but I have a Jupyter Lab instance working in OOD 3.0 with various additional kernels like R, SageMath and Matlab. I’m trying to add Julia as an available kernel. It seems Julia doesn’t like to install packages centrally, so I installed Julia centrally, but IJulia is in my .julia folder. It does show up as an available kernel in Jupyter Lab, but if I try to use it, it fails to load.
When you launch your Jupyter App, are you loading the same Julia module somewhere in the template/script.erb.sh file?
The process you followed is exactly what I tell my users to do. I have a section in my app’s form where users can fill in the name of a module they want to add during the runtime that is then injected into a module add xxx in the script.erb.sh file before Jupyter is launched.
That’s weird. I guess it was interpreting something in my cut-and-paste of the script.sh.erb. Let’s try it again by posting directly in the forum instead of replying to the email.
#!/usr/bin/env bash
# Benchmark info
echo "TIMING - Starting main script at: $(date)"
# Set working directory to home directory
cd "${HOME}"
#
# Start Jupyter Notebook Server
#
export NB=notebooks
if [ ! -e ~/$NB ]; then
echo "Making ~/$NB folder ..."
mkdir ~/$NB
else
if [ -f ~/$NB ]; then
echo "Renaming existing ~/$NB file to ~/$NB.file."
mv ~/$NB ~/$NB.file
echo "Making ~/$NB folder ..."
mkdir ~/$NB
else
echo "The ~/$NB folder already exists, proceeding..."
fi
fi
# Clean the environment
module purge
#module load DefaultModules default-environment
module try-add autotools
module try-add prun
module try-add gnu9
module try-add openmpi4
module try-add tcnjhpc
export JUPYTER="$(which jupyter)"
module add julia/1.9.3
# Load the require modules
#module load python/3.10.11 R/4.1.3 matlab/R2023a
module load <%= context.tcnjhpc_version %>
<%- if context.tcnjhpc_use_gpu == "yes" -%>
module load cuda
<%- end -%>
<% unless context.tcnjhpc_modules.blank? %>
module add <%= context.tcnjhpc_modules %>
<% end %>
# List loaded modules
module list
# Benchmark info
echo "TIMING - Starting jupyter at: $(date)"
# Launch the Jupyter Notebook Server
set -x
export MLM_LICENSE_FILE="/opt/tcnjhpc/matlab_R2022a/licenses/network.lic"
#jupyter lab --config="${CONFIG_FILE}" <%= context.tcnjhpc_extra_jupyter_args %>
jupyter <%= context.tcnjhpc_classic_jupyter == "1" ? "notebook" : "lab" %> --config="${CONFIG_FILE}" <%= context.tcnjhpc_extra_jupyter_args %>
Seems I found an issue when having the Matlab module loaded for the iMatlab kernel and IJulia. I’ll have to look to see what the issue is. Maybe updating imatlab will fix thing.
I think I was able to replicate. You should check your output.log for errors - when I tried I found this error
FileNotFoundError: [Errno 2] No such file or directory: 'julia': 'julia'
I can reach out to my scientific applications group for specifics but I think I get the gist and will relay the same to you.
Basically - your kernel cannot find the julia binary because it’s an entirely different process and in fact you (the jupyter’s process tree) may have never loaded the julia module yet.
To replicate this issue - I went on our online documentation to see how we do this (because I was sure that we do).
In the section Install Julia kernel for Jupyter - we provide a wrapper/helper script in the julia module’s PATH called create_julia_kernel. This is that script:
#!/usr/bin/env julia
using Pkg
if ! haskey(Pkg.installed(), "IJulia")
println("Installing IJulia")
Pkg.add("IJulia")
end
println("IJulia found: ", Pkg.installed()["IJulia"])
using IJulia
installkernel("Julia", julia = `osc_kernel_wrapper.sh julia`, env=Dict("MODULES"=>string("xalt/latest julia/", VERSION)))
Note that it creates the kernel with some specific environment file and uses a wrapper shell script.
That wrapper shell script this: (It’s in the Jupyter module’s PATH)
#!/usr/bin/env bash
# Load the required environment
module load ${MODULES}
module list
# Launch the original command
set -x
exec "${@}"
So - you use an environment variable to load the appropriate modules, then issue julia commands through the wrapper (IDK why we list or don’t do >/dev/null 2>&1 because I would assume that throws stuff off, but I guess not).
It was the matlabengine+imatlab kernel that was messing with the IJulia kernel. Our latest Matlab (R2023a) wasn’t working correctly as kernel under Python 3.10.11. Once I removed matlabengine & imatlab, the IJulia kernel loaded and worked fine. I’ll have to wait for our central IT to get our license server updated to support Matlab R2023b before trying the matlabengine & imatlab install again.
@ssivy glad to hear you found and fixed the issue.
That being said If you are interested in an alternative our site moved Julia to it’s own interactive app to better handle the issues it has with global package installs. What we found is that once you add global packages to a Julia installation it can break users from being able to install additional Julia packages in their home directory. That, or when users have packages in their home directory the global install packages are ignored and break Jupyter. To fix this we install the Julia packages in the users home or project directory when the OOD app is launched. Here is an example of our helper script for launcher jupyter-lab template/bin/jupyter-lab · main · NMSU_HPC / ood_bc_julia · GitLab.
You can find our Julia app here (NMSU_HPC / ood_bc_julia · GitLab) including an example singularity/apptainer build file. It allows the use of modules (assuming the module contains a .sif image denoted by $SIF env) or user provided containers. At our site we ship Julia in a container that also has jupyter installed for simplicity and list those containers on our module system. The app supports multiple frontends such as Pluto.jl, code-server, Jupyter Notebook, and Jupyter Lab.
Great! Thanks. Yes, I noticed the issue of global vs. user packages. I was expecting it to work like Python, but it doesn’t. I’ll take a look at your helper script.
hmm, testing this app, I guess exposing displaying pluto session secret in process dump (ps is unavoidable, right…?
ps aux | grep jose
...
jose 2238806 6.9 0.0 10451224 661636 ? Sl 18:10 0:28 julia -e using Pluto; Pluto.run(Pluto.ServerSession(secret="notSoSecret", options=Pluto.Configuration.from_flat_kwargs(host="0.0.0.0", port=25785)))
...
the quick fix for us was to modify the launcher to take the password from environment variable…:
julia -e "using Pluto; Pluto.run(Pluto.ServerSession(secret=get(ENV,\"PASSWORD\",'0'), options=Pluto.Configuration.from_flat_kwargs(host=\"0.0.0.0\", port=$PORT)))"
Actually, I did not fully understand the need of passing password(and port) as positional arguments from template/script.sh to template/bin/pluto as it seems to be redundant to the environment variable PASSWORD which is set as well…
I like this solution. I’ll update our stuff based on this.
The reason we use arguments is we use these bin files in a few places other than OOD where it is desirable to provide the password as an argument. That being said we can upgrade these to default to priority of Argument → Env Var → Generate New. There is a logic issue with this currently. I’ll update and report back.
I’ve updated the Pluto.jl portion of my (NMSU’s) Julia OOD app to use environment variables instead of arguments (NMSU_HPC / ood_bc_julia · GitLab). Thanks @jose-d for the recommendation.
This will both simplify things and protect the password from showing in /proc via ‘ps’ calls.