Julia in JupyterLab on OOD

Okay, so I’m not sure how on topic this is for OOD, but I have a Jupyter Lab instance working in OOD 3.0 with various additional kernels like R, SageMath and Matlab. I’m trying to add Julia as an available kernel. It seems Julia doesn’t like to install packages centrally, so I installed Julia centrally, but IJulia is in my .julia folder. It does show up as an available kernel in Jupyter Lab, but if I try to use it, it fails to load.

Here is how I did the install.

[ssivy@node062 ~]$ module add julia/1.9.3
[ssivy@node062 ~]$ module add python/3.10.11
[ssivy@node062 ~]$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.3 (2023-08-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using Pkg

julia> Pkg.add("IJulia")
  Installing known registries into `~/.julia`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
   Installed JLLWrappers ───── v1.5.0
   Installed Conda ─────────── v1.9.1
   Installed IJulia ────────── v1.24.2
   Installed Parsers ───────── v2.7.2
   Installed JSON ──────────── v0.21.4
   Installed ZMQ ───────────── v1.2.2
   Installed PrecompileTools ─ v1.2.0
   Installed Preferences ───── v1.4.1
   Installed VersionParsing ── v1.3.0
   Installed SoftGlobalScope ─ v1.1.0
   Installed libsodium_jll ─── v1.0.20+0
   Installed ZeroMQ_jll ────── v4.3.4+0
   Installed MbedTLS ───────── v1.1.7
  Downloaded artifact: ZeroMQ
  Downloaded artifact: libsodium
    Updating `~/.julia/environments/v1.9/Project.toml`
  [7073ff75] + IJulia v1.24.2
    Updating `~/.julia/environments/v1.9/Manifest.toml`
  [8f4d0f93] + Conda v1.9.1
  [7073ff75] + IJulia v1.24.2
  [692b3bcd] + JLLWrappers v1.5.0
  [682c06a0] + JSON v0.21.4
  [739be429] + MbedTLS v1.1.7
  [69de0a69] + Parsers v2.7.2
  [aea7be01] + PrecompileTools v1.2.0
  [21216c6a] + Preferences v1.4.1
  [b85f4697] + SoftGlobalScope v1.1.0
  [81def892] + VersionParsing v1.3.0
  [c2297ded] + ZMQ v1.2.2
  [8f1865be] + ZeroMQ_jll v4.3.4+0
  [a9144af2] + libsodium_jll v1.0.20+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.9.2
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.2+0
  [14a3606d] + MozillaCACerts_jll v2022.10.11
  [83775a58] + Zlib_jll v1.2.13+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0
    Building Conda ─→ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/8c86e48c0db1564a1d49548d3515ced5d604c408/build.log`
    Building IJulia → `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/47ac8cc196b81001a711f4b2c12c97372338f00c/build.log`
Precompiling project...
  13 dependencies successfully precompiled in 43 seconds. 3 already precompiled.

julia> exit()
[ssivy@node062 ~]$ cat .local/share/jupyter/kernels/julia-1.9/kernel.json 
{
  "display_name": "Julia 1.9.3",
  "argv": [
    "/opt/tcnjhpc/julia/1.9.3/bin/julia",
    "-i",
    "--color=yes",
    "--project=@.",
    "/home/hpc/ssivy/.julia/packages/IJulia/Vo51o/src/kernel.jl",
    "{connection_file}"
  ],
  "language": "julia",
  "env": {},
  "interrupt_mode": "signal"
}
[ssivy@node062 ~]$

When I try to use the kernel, I get this in the OnDemand Session ‘output.log’ file.

[I 2023-11-06 11:33:27.111 ServerApp] Creating new notebook in 
[I 2023-11-06 11:33:27.379 ServerApp] Kernel started: a2f7d3d2-604d-44bf-ad52-bf0d089fd85d
[I 2023-11-06 11:33:30.366 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
[I 2023-11-06 11:33:33.383 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (2/5), new random ports
[I 2023-11-06 11:33:36.399 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (3/5), new random ports
[I 2023-11-06 11:33:39.414 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (4/5), new random ports
[I 2023-11-06 11:33:42.429 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
[W 2023-11-06 11:33:45.445 ServerApp] AsyncIOLoopKernelRestarter: restart failed
[W 2023-11-06 11:33:45.445 ServerApp] Kernel a2f7d3d2-604d-44bf-ad52-bf0d089fd85d died, removing from map.

I see others have this working, but I’m not sure what step I’ve missed. It looks like IJulia is trying to start another Jupyter session.

When you launch your Jupyter App, are you loading the same Julia module somewhere in the template/script.erb.sh file?

The process you followed is exactly what I tell my users to do. I have a section in my app’s form where users can fill in the name of a module they want to add during the runtime that is then injected into a module add xxx in the script.erb.sh file before Jupyter is launched.

Yes. I load the module before the Python module.

Here is an excerpt from my script.erb

#!/usr/bin/env bash

Benchmark info

echo “TIMING - Starting main script at: $(date)”

Set working directory to home directory

cd “${HOME}”

I am guessing that post is missing the rest of the excerpt but regardless, sounds like your app is setup correctly.

I would add --debug to your Jupyter Lab launch command and see if you can get more info out of the Jupyter logs. I am thinking your app is fine.

That’s weird. I guess it was interpreting something in my cut-and-paste of the script.sh.erb. Let’s try it again by posting directly in the forum instead of replying to the email.

#!/usr/bin/env bash

# Benchmark info
echo "TIMING - Starting main script at: $(date)"

# Set working directory to home directory
cd "${HOME}"

#
# Start Jupyter Notebook Server
#

export NB=notebooks
if [ ! -e ~/$NB ]; then
  echo "Making ~/$NB folder ..."
  mkdir ~/$NB
else
  if [ -f ~/$NB ]; then
    echo "Renaming existing ~/$NB file to ~/$NB.file."
    mv ~/$NB ~/$NB.file
    echo "Making ~/$NB folder ..."
    mkdir ~/$NB
  else
    echo "The ~/$NB folder already exists, proceeding..."
  fi
fi

# Clean the environment
module purge
#module load DefaultModules default-environment
module try-add autotools
module try-add prun
module try-add gnu9
module try-add openmpi4
module try-add tcnjhpc

export JUPYTER="$(which jupyter)"
module add julia/1.9.3

# Load the require modules
#module load python/3.10.11 R/4.1.3 matlab/R2023a
module load <%= context.tcnjhpc_version %>

<%- if context.tcnjhpc_use_gpu == "yes" -%>
module load cuda
<%- end -%>

<% unless context.tcnjhpc_modules.blank? %>
module add <%= context.tcnjhpc_modules %>
<% end %>

# List loaded modules
module list

# Benchmark info
echo "TIMING - Starting jupyter at: $(date)"

# Launch the Jupyter Notebook Server
set -x
export MLM_LICENSE_FILE="/opt/tcnjhpc/matlab_R2022a/licenses/network.lic"
#jupyter lab --config="${CONFIG_FILE}" <%= context.tcnjhpc_extra_jupyter_args %>
jupyter <%= context.tcnjhpc_classic_jupyter == "1" ? "notebook" : "lab" %> --config="${CONFIG_FILE}" <%= context.tcnjhpc_extra_jupyter_args %>

Seems I found an issue when having the Matlab module loaded for the iMatlab kernel and IJulia. I’ll have to look to see what the issue is. Maybe updating imatlab will fix thing.

I think I was able to replicate. You should check your output.log for errors - when I tried I found this error

FileNotFoundError: [Errno 2] No such file or directory: 'julia': 'julia'

I can reach out to my scientific applications group for specifics but I think I get the gist and will relay the same to you.

Basically - your kernel cannot find the julia binary because it’s an entirely different process and in fact you (the jupyter’s process tree) may have never loaded the julia module yet.

To replicate this issue - I went on our online documentation to see how we do this (because I was sure that we do).

In the section Install Julia kernel for Jupyter - we provide a wrapper/helper script in the julia module’s PATH called create_julia_kernel. This is that script:

#!/usr/bin/env julia

using Pkg
if ! haskey(Pkg.installed(), "IJulia")
  println("Installing IJulia")
  Pkg.add("IJulia")
end

println("IJulia found: ", Pkg.installed()["IJulia"])
using IJulia
installkernel("Julia", julia = `osc_kernel_wrapper.sh julia`, env=Dict("MODULES"=>string("xalt/latest julia/", VERSION)))

Note that it creates the kernel with some specific environment file and uses a wrapper shell script.

That wrapper shell script this: (It’s in the Jupyter module’s PATH)

#!/usr/bin/env bash

# Load the required environment
module load ${MODULES}
module list

# Launch the original command
set -x
exec "${@}"

So - you use an environment variable to load the appropriate modules, then issue julia commands through the wrapper (IDK why we list or don’t do >/dev/null 2>&1 because I would assume that throws stuff off, but I guess not).

It was the matlabengine+imatlab kernel that was messing with the IJulia kernel. Our latest Matlab (R2023a) wasn’t working correctly as kernel under Python 3.10.11. Once I removed matlabengine & imatlab, the IJulia kernel loaded and worked fine. I’ll have to wait for our central IT to get our license server updated to support Matlab R2023b before trying the matlabengine & imatlab install again.

@ssivy glad to hear you found and fixed the issue.

That being said If you are interested in an alternative our site moved Julia to it’s own interactive app to better handle the issues it has with global package installs. What we found is that once you add global packages to a Julia installation it can break users from being able to install additional Julia packages in their home directory. That, or when users have packages in their home directory the global install packages are ignored and break Jupyter. To fix this we install the Julia packages in the users home or project directory when the OOD app is launched. Here is an example of our helper script for launcher jupyter-lab template/bin/jupyter-lab · main · NMSU_HPC / ood_bc_julia · GitLab.

You can find our Julia app here (NMSU_HPC / ood_bc_julia · GitLab) including an example singularity/apptainer build file. It allows the use of modules (assuming the module contains a .sif image denoted by $SIF env) or user provided containers. At our site we ship Julia in a container that also has jupyter installed for simplicity and list those containers on our module system. The app supports multiple frontends such as Pluto.jl, code-server, Jupyter Notebook, and Jupyter Lab.

Great! Thanks. Yes, I noticed the issue of global vs. user packages. I was expecting it to work like Python, but it doesn’t. I’ll take a look at your helper script.

hmm, testing this app, I guess exposing displaying pluto session secret in process dump (ps is unavoidable, right…?

ps aux | grep jose
...
jose     2238806  6.9  0.0 10451224 661636 ?     Sl   18:10   0:28 julia -e using Pluto; Pluto.run(Pluto.ServerSession(secret="notSoSecret", options=Pluto.Configuration.from_flat_kwargs(host="0.0.0.0", port=25785)))
...

The pluto bits could be placed in a julia script and called that way. Would get around that I think. I’ll do some testing and report back.

1 Like

the quick fix for us was to modify the launcher to take the password from environment variable…:

julia -e "using Pluto; Pluto.run(Pluto.ServerSession(secret=get(ENV,\"PASSWORD\",'0'), options=Pluto.Configuration.from_flat_kwargs(host=\"0.0.0.0\", port=$PORT)))"

Actually, I did not fully understand the need of passing password(and port) as positional arguments from template/script.sh to template/bin/pluto as it seems to be redundant to the environment variable PASSWORD which is set as well…

I like this solution. I’ll update our stuff based on this.

The reason we use arguments is we use these bin files in a few places other than OOD where it is desirable to provide the password as an argument. That being said we can upgrade these to default to priority of Argument → Env Var → Generate New. There is a logic issue with this currently. I’ll update and report back.

1 Like

I’ve updated the Pluto.jl portion of my (NMSU’s) Julia OOD app to use environment variables instead of arguments (NMSU_HPC / ood_bc_julia · GitLab). Thanks @jose-d for the recommendation.

This will both simplify things and protect the password from showing in /proc via ‘ps’ calls.

1 Like

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.