Has anyone worked on running the Open OnDemand services inside a Singularity container?
I am attempting to do this, and getting stuck where the HTTP server cant seem to connect to the user pun socket. Has anyone seen this, or know how to get it to connect to the unix sockets.
This is the error message:
Connection refused: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/ondemand-nginx//passenger.sock (*) failed
Hi and welcome!
Not that I’m aware of, though we use containers a lot on our work machines (not on our HPC systems). Though they’re through either podman or docker.
I would guess that the PUN never started. What’s the output in the /var/log/ondemand-nginx/ files?
the log shows:
[ N 2021-07-27 09:28:06.3240 131513/T1 age/Wat/WatchdogMain.cpp:1373 ]: Starting Passenger watchdog...
[ N 2021-07-27 09:28:06.3733 131516/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core...
[ N 2021-07-27 09:28:06.3735 131516/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2021-07-27 09:28:06.5248 131516/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 131516
It almost seems like the system cant properly connect to unix sockets between processes.
Yea maybe an SELinux policy that doesn’t allow apache to connect to it?
No selinux on the system.
turns out it was just a stale socket file, and removing
/var/run/ondemand-nginx//passenger.sock
got things going again.
Should have spotted that for ya.
Yea I would not mount var. I don’t use singularity alot, but I think there are default mounts? That socket especially should be ephemeral so, I’m make sure /var/run/ondemand-nginx/$USER isn’t mounted in the container.
Are you trying to run this at your site or do you use Singularity on your lap/desktop? I’d be interested in knowing the experience you’d want. We’ve started to get the tooling better for running containers, though singularity is missing.
I am working on a proof of concept of OnDemand for us on our large clusters. I was able to setup a basic Install on one of our test clusters. I built a desktop environment that starts up a singularity container on a compute node, and that worked well. I also setup a jupyter notebook/lab system in singularity container.
In looking at how to reliably deploy the OnDemand Frontend services, we wanted to see if it would run in a singularity container, such that after a reinstall of the controlling nodes it could just be started up quickly by just executing the container. We are starting it as the root user, and ran into some issues early on with the container not having the whole user database available, but worked past that by binding the nss paths into place.(I think this is what left the strange socket around). Once we cleaned that up the system works just as it did when it was installed on the main system.
Can you please share results of your PoC with OOD in Singularity? Because right now I am working on same issue.
So if I understood correctly you built singularity image as root, so you will have root in container.
Maybe you tried --fakeroot to build and run as non-root user?