Modules in Jupyter script.sh.erb file not found

Get Help
1

I am using parts of the bc_osc_jupyter setup at https://github.com/OSC/bc_osc_jupyter to more effectively deal with multiple kernels. Tried to use modules to set up two of the kernels but getting the module not found error (see launch_wrapper.log file below). Would prefer to figure out how to make the modules work instead of using workaround.

script.sh.erb file:
<%-
require ‘pathname’
wrapper = session.staged_root.join(“launch_wrapper.sh”)
wrapper_log = session.staged_root.join(“launch_wrapper.log”)
kernels = {
python36: {
display_name: “Python 3.6”,
language: “python”,
argv: [
wrapper,
“python3”,
“-m”,
“ipykernel_launcher”,
“-f”,
“{connection_file}”
],
env: {
MODULES: “intel python3”
}
},
python27: {
display_name: “Python 2.7”,
language: “python”,
argv: [
wrapper,
“python2”,
“-m”,
“ipykernel”,
“-f”,
“{connection_file}”
],
env: {
MODULES: “intel python2”
}
},
scala211: {
display_name: “scala-2.11”,
language: “scala”,
extensions: [“snb”],
argv: [
wrapper,
“scala”,
“–quiet”,
“/root/jove-scala-cli-0.1.1-1-SNAPSHOT/bin/jove-scala”,
“{connection_file}”
],
env: {
MODULES: “”
}
},
}.tap do |hsh|
end
-%>

echo “Starting main script…”
echo “TTT - $(date)”
module purge

echo “Creating launcher wrapper script…”
(
umask 077
sed ‘s/^ {2}//’ > “<%= wrapper %>” << EOL
#!/usr/bin/env bash
exec &>>"<%= wrapper_log %>"
module purge
module load ${MODULES}
module list
set -x
exec “${@}”
EOL
)
chmod 700 “<%= wrapper %>”
echo “TTT - $(date)”

echo “Creating custom Jupyter kernels…”
export JUPYTER_PATH="{PWD}/share/jupyter" <%- kernels.each do |name, kernel| -%> <%- path = "{JUPYTER_PATH}/kernels/sys_#{name}" -%>
mkdir -p “<%= path %>”
cp “{PWD}/assets/<%= name.to_s.gsub(/\d/, "") %>"/* "<%= path %>/." ( umask 077 cat > "<%= path %>/kernel.json" << EOL <%= JSON.pretty_generate kernel %> EOL ) <%- end -%> echo "TTT - (date)”

module load <%= context.version %>
module list
echo “TTT - $(date)”

set -x
jupyter kernelspec list
{ set +x; } 2>/dev/null
echo “TTT - $(date)”

set -x
jupyter <%= context.jupyterlab_switch == “1” ? “lab” : “notebook” %> --config="${CONFIG_FILE}"

output.log file
Script starting…
Waiting for Jupyter Notebook server to open port 62332…
TIMING - Starting wait at: Mon Apr 6 13:56:26 CDT 2020
Starting main script…
TTT - Mon Apr 6 13:56:26 CDT 2020
Creating launcher wrapper script…
TTT - Mon Apr 6 13:56:26 CDT 2020
Creating custom Jupyter kernels…
TTT - Mon Apr 6 13:56:26 CDT 2020
TTT - Mon Apr 6 13:56:26 CDT 2020
TTT - Mon Apr 6 13:56:26 CDT 2020
Available kernels:
scala-2.11 /home/thomasbr/.ipython/kernels/scala-2.11
sys_python27 /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/share/jupyter/kernels/sys_python27
sys_python36 /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/share/jupyter/kernels/sys_python36
sys_scala211 /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/share/jupyter/kernels/sys_scala211
python2 /usr/share/jupyter/kernels/python2
TTT - Mon Apr 6 13:56:27 CDT 2020
Discovered Jupyter Notebook server listening on port 62332!
TIMING - Wait ended at: Mon Apr 6 13:56:28 CDT 2020
Generating connection YAML file…

launch_wrapper.log file

/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
/home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found

  • exec python2 -m ipykernel -f /home/thomasbr/.local/share/jupyter/runtime/kernel-94318ec2-ed1d-467a-813f-9cd8df09a1e1.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python3 -m ipykernel_launcher -f /home/thomasbr/.local/share/jupyter/runtime/kernel-a51ab511-b30d-416e-afdd-2ad03969e510.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 10: exec: python3: not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python3 -m ipykernel_launcher -f /home/thomasbr/.local/share/jupyter/runtime/kernel-a51ab511-b30d-416e-afdd-2ad03969e510.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 10: exec: python3: not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python3 -m ipykernel_launcher -f /home/thomasbr/.local/share/jupyter/runtime/kernel-a51ab511-b30d-416e-afdd-2ad03969e510.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 10: exec: python3: not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python3 -m ipykernel_launcher -f /home/thomasbr/.local/share/jupyter/runtime/kernel-a51ab511-b30d-416e-afdd-2ad03969e510.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 10: exec: python3: not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python3 -m ipykernel_launcher -f /home/thomasbr/.local/share/jupyter/runtime/kernel-a51ab511-b30d-416e-afdd-2ad03969e510.json
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 10: exec: python3: not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 5: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 6: module: command not found
    /home/thomasbr/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/9893d217-b1d6-45c0-a89a-fd849a352446/launch_wrapper.sh: line 7: module: command not found
  • exec python2 -m ipykernel -f /home/thomasbr/.local/share/jupyter/runtime/kernel-71c8dd3f-6e99-4bf7-b8ca-3a314bb0a224.json
2

Yea so, you could cut OOD out of the equation to troubleshoot because this seems like a ‘job environment setup’ problem, so you may be able to just ssh into a compute node and troubleshoot without scheduling a job (or a simple hello_module.sh) script so it gets scheduled quickly.

In any case, I’m not super familiar with lmod. We use lmod and I tracked module down to an function in /usr/local/lmod/lmod/init/sh which is called in an /etc/profile.d/lmod.sh init file. So that’s why it exists for us, it’s a part of the init process.

You had to load ~/.bashrc maybe you need to force load /etc/profile as well?

As an aside, but possibly related because you’re not loading your environment correctly, I just found this to add as a header to scripts header (we previously fought this issue with script_wrapper in another topic.

It could be that because it’s on the 2nd line, it doesn’t actually being interpreted? Setting the header will make it the very first line and thus a proper header for a script.

batch_connect:
  vnc:
    header: "#!/bin/bash"
  basic:
    header: "#!/bin/bash"
3

Put “source /etc/profile.d/lmod.sh” in the /etc/ood/config/clusters.d/ivy.yml file and now it is finding the modules.

v2:
metadata:
title: “Ivy Cluster”
login:
host: “ivy.hpcc.ttu.edu”
job:
adapter: “sge”
cluster: “ivy”
bin: “/export/uge/bin/lx-amd64”
conf: “/tmp/test_sge.conf”
sge_root: “/export/uge”
libdrmaa_path: “/export/uge/lib/lx-amd64/libdrmaa.so”
batch_connect:
basic:
header: “#!/bin/bash”
script_wrapper: |
source /etc/profile.d/lmod.sh
set +o posix
module purge
%s
vnc:
header: “#!/bin/bash”
script_wrapper: |
source /etc/profile.d/lmod.sh
set +o posix
module purge
export PATH="/opt/TurboVNC/bin:$PATH"
export WEBSOCKIFY_CMD="/opt/websockify/run"
%s

It is now not able to find the ipykernel_launcher file.

Should we use ipykernel_launcher or ipykernel for the python 2 and 3 kernels?

4

python2 just reached it’s end of life, so I would suggest python3 just on that basis.

But at this point, we’re a bit out of my expertise. We have python (and the python kernels) behind modules so I can’t help directly with what do.