This is an interesting idea. There are several blockers to consider.
-
If we could start the PUNs on the login nodes, the PUNs still need to access config in /etc/ood/config, and the deployed app code at /var/www/ood/apps/sys, both of which currently are only deployed to the web hosts. At some point it would appear that the bulk of OnDemand would be installed on the login node and Apache what is starting “remotely” on a web host.
-
The benefit we gain of having Apache and the PUN on the same host is they can communicate with each other through Unix domain sockets and that secures the communication between Apache and the PUNs because the PUNs are not listening on ports. If there was an ood-agent that started PUNs, we would need to support multual TLS authentication between the PUN and Apache. (more on this at the bottom of the email)
Your goals are:
- avoid needing to setup the OnDemand host as a submit host to the batch scheduler
- avoid needing to mount the file systems on the OnDemand host
Is there anything else I missed?
If these are it, another approach might be providing an abstraction in the form of two per user services, a job management service, and a file management service, that the OnDemand apps use. The apps would use these services for all their file management and job management.
We do sort’ve support this now with the job management. See Submit to scheduler on seperate node from OndDemand node for a description of this. We don’t have a per-user service that runs but you can use the commands via ssh to another node.
On mutual TLS authentication:
We do have a similar problem with Apache and the interactive apps. We want to add mutual TLS authentication between Apache and the interactive apps running on compute nodes (like RStudio and Jupyter) that would be supported the same way regardless of what app you are running. To do this we are thinking of some kind of service mesh, where a sidecar proxy runs alongside Jupyter or RStudio and Apache proxies to that sidecar. Or perhaps we would have two solutions, one for web servers and one for VNC servers where a side car already exists - websockify - that we could figure out how to add mutual TLS authentication to.
A good solution for this service mesh, one that keeps installation and maintenance of OnDemand as simple as it is today, could possibly be leveraged for mutual TLS between the Apache on the web host and any per user server or service running on the login nodes.
Some projects of interest are https://www.consul.io/ and https://www.cncf.io/ projects. But this investigation is slow going, mostly because many of the service mesh solutions out there are too complex for OnDemand, and those that might be candidates have so many configuration options it would take significant investment to avoid https://owasp.org/www-project-top-ten/OWASP_Top_Ten_2017/Top_10-2017_A6-Security_Misconfiguration