Following up the brief discussion from yesterday during the Tips and Tricks meeting.
Currently I run OpenOnDemand on one of my login node, installed as RPM with dnf and started/stopped with systemctl. It has its configuration settings sprinkled in various /etc and /var places.
That works just fine and has its advantages, but creates a challenge in that if I should upgrade it the old instance would be overwritten and if anything goes wrong both the old and the new instance would not be available. Not good.
Various web-typical solutions to this problem (containers, VMs etc) are definitely possible, however they come with their own set of problems, including but not limited to additional skill sets needed, complications with the networking and Slurm settings (for the external VMs) etc.
In my HPC-focused mindset, it would be fantastic if I could install Open OnDemand with spack, which would happen in a unique path (and never overwriting a previous install) while also keeping its configuration files in such installation path, so there would be nothing to find elsewhere. Besides perhaps some systemctl files, assuming one would still want to start/stop the service with that. Such systemctl-related files should include at least release number and perhaps spack hash in order to be unique. Relevant ports should be different between installs and easily swappable, so I could install a new version in production, while the old is still working, and after I am satisfied open it up to the first early/friendly user before making the swap to the whole community (and potentially leaving the old version still available, at least for a while).
I think at least a few HPC-focused people would love something like that, but the questions are:
is this even possible? I mean, with reasonable efforts, in software everything is possible, but with what effort?
are there enough of such people to justify the development of such a “feature”? Maybe it’s just a spack recipe and little more, but still…
is it worth it or the status quo is better? If the latter, how do people deal with the situation in which using external VM is out of the question? Does anybody use multiple containers in the login node to serve various versions and lubricate the transition from one to the other to mitigate the risk of breakage during an upgrade?
I’ve always had the impression that EasyBuild, Spack, and other similar tools are intended for software that runs in userspace.
Since OOD requires root access for well-known reasons (PUNs), wouldn’t Docker or Podman more suitable for this use case? Especially since a Dockerfile describing the OOD installation would only require a few dozen lines…
I must admit I wasn’t present at the mentioned call, so I’d be happy to hear more about your motivation to use alternative ways to install OOD.
Typically Spack is used for software that runs in userspace but it is not “intended” for that only. You can install anything in any way (and in fact we do).
Regarding Docker, the dockerfile itself may be short, but (AFAIK and that may be wrong) its execution and the resulting image would be large, no? Not to mention all the necessary infrastructure, firewall and other configuration settings, system people education, etc which makes sense if one is using lots of software that requires it. But if OnDemand is the only one (as it is at our site) we are trying to see if we can do it by leveraging existing infrastructure, configurations, know-how, etc. We didn’t go much deeper than that at the call, I basically said what I wrote above and a few people with their video on were nodding, so I assume that’s a common issue (since most people have more of an HPC skillset/expertise, matching more spack and friends, rather than a web on, matching docker etc)
Trying to build a spack recipe for OOD would really only handle the actual installation of specific versions and does not handle any of the configuration and non software dependencies that OOD has. What would you do about ood configs being different? What about different ports? What about ssl certs? Those are all outside of the scope of spack.
My OOD image is maybe a few GB at most and adds virtually no overhead on the system vs running baremetal. It would require no other infrastructure than just yum install podman, The firewall takes no configuration as you would run the container using the host’s network. I.e. no different than configuring the firewall for a baremetal install of ood. Since its a container, it is sandboxed so it makes handling varying configs trivial. It solves for ood versions, unique configs and even running multiple instances on the same host. It also makes standing up varying versions of OOD trivial as all of the software and dependencies is sandboxed in the containert build script and runtime environment.
If you really don’t want to spin up a dev vm or other baremetal server for testing, you could actually run each ood container (maybe one prod and one dev) on unique ports and use a system level apache or nginx instance on the host to translate specific CNAME’s to the internal ports so ondemand-prod.mysite.edu could get routed to the normal port 80 and 443 where as ondemand-dev.mysite.edu could get routed to 8080 or 8443 for instance. I actually do this for other containerized websites I host for some clients that are all on the same machine.
Again, I will be giving a talk at GOOD covering my OOD build via a Dockerfile and run via podman in a systemd unit file. Using my container build and run system as a base, I would argue it would take less than a few days to be rather comfortable with using the setup and ultimately, containers are pretty ubiquitous in HPC these days so I would be confident that it would be knowledge you will ultimately end up needing in some capacity at your institution anyways.
I know Spack would not solve the problem of OOD configs being different, that was in fact the main point of my question! I see containers as a (dirty) solution to this sort of problems with software that does not play nicely with itself (or rather, concurrent versions of itself, but I digress…)
It’s not that I don’t want to spin up dev VM or other servers, it’s that I don’t have access to those resources, and even if I did jump through many hoops to get access to them, they would sit in a totally different network and be completely isolated from the cluster, so could not really test job submissions and the likes, unless I completely duplicated in the test environment my full cluster, which I don’t have the manpower, hardware resources and software licenses to do (we use RHEL).
Regarding the firewall, and the needed infrastructure for containers, are you saying that just installing podman is really all that is needed? No other external things to install or download or access? Our network setup is quite tight and even a simple “oh just do wget that” is an almost impossible request. So if only podman is what’s needed, that makes it much more palatable.
Unfortunately I won’t be able to attend GOOD, but I am looking forward to seeing the recording of your presentation (if it gets recorded) or at least reading your slide.
its execution and the resulting image would be large, no?
Storage for Docker is not shared across multiple servers, and since it’s more than well within the capacity of commodity NVMe drives, I consider it to be a non-issue. This is particularly true when compared to the massive sizes of datasets used in cosmology and particle physics research at our facilities. …
But yeah, the important factor is, as I now see,
using external VM is out of the question
Based on my personal experience, having some small, flexible virtualization infrastructure is an essential complement to any HPC system… …no need of fancy enterprise™ Hypervisor. Three nodes from previous generation cluster, proxmox/qemu/etc. and that’s makes the life much easier…
@davide-q That is all very helpful context. Especially ““oh just do wget that” is an almost impossible request.” I totally feel your pain there!
I should ask if you are an selinux enabled or disabled shop. If so, there are just a few rules you do need to open for selinux to allow containers to connect to host network and pass http traffic (if I remember correctly).
As far as needed resources goes I think the list of requirements for my setup is pretty much just:
public ip and resolving public dns
ssl certs
podman installed
slurm configs on host
access to main slurm rpm (usually just pull from a local repo but could be just the rpm file too)
selinux off (but can be done with it on too)
firewall allowing port 80 and 443 traffic in
Effectively, if you have a login node, running the setup on that would fill many of the dependencies. OOD effectively acts like a head-node at the end of the day.
Three nodes from previous generation cluster, proxmox/qemu/etc. and that’s makes the life much easier…
Yes, that’s what I am planning to do for the next cluster. But for the current, I inherited what I have and I don’t have that. In fact even the disk space is somewhat limited, and whatever I use for system software and containers and the likes, I take away from user disk quota, so I prefer to be as frugal as possible, which explain my preference for Spack (which re-utilizes dependencies) vs containers (which do not)
I should ask if you are an selinux enabled or disabled shop.
Luckily disabled.
public ip and resolving public dns
I don’t have that, we are a completely private network with very little options to go to the outside (e.g. dnf from official RHEL repos) and everything else needs to be manually transferred
access to main slurm rpm
I manually download the source and build the RPMs myself
Everything else is similar to what I have already or easily doable.
You actually don’t really need a public IP, but if you want ssl you would have to do a self signed cert which browsers will throw an error on.
Indeed containers do not re-utilize deps but I just went and checked a fresh build of the image and the total space the ood image is taking up, along with the base images it pulls down is less than 3GB. Add maybe a few more GB for apps and that really is about it. Are you that short on space?
You actually don’t really need a public IP, but if you want ssl you would have to do a self signed cert which browsers will throw an error on.
Yeah, that’s what I am doing now sans-container
the total space the ood image is taking up, along with the base images it pulls down is less than 3GB. Add maybe a few more GB for apps and that really is about it. Are you that short on space?
Yes and no. At the moment I have several TB available, so even half a dozen of OOD images would be fine. But if I were to do containers more aggressively for other things (as you hinted in an earlier message) then yes, that would become a problem.
So looking forward to your talk/slides and giving this approach a spin!
