VirtualGL in OOD apps

This is a question for the OOD team.

I have noticed that some OOD apps that OSC publishes, which can use OpenGL graphics (Ansys, Matlab, VMD, Paraview, …) use VirtualGL. VGL needs X server running on the compute node, which in turn will sit on the GPU and potentially eat away resources from the CUDA computational jobs that may run on that GPU.

So, my question is, how does OSC technically allow VirtualGL? Do you run X Server on all your compute nodes? Or, start it with a job start using a flag (which I don’t see in the OOD apps so probably not). Or have cheap GPUs in each node used for GL that are independent from the computational GPUs? Or, something else?

I’d appreciate some details that would allow us to consider such deployment over here.

Naturally, if other HPC centers have their own solution for compute node GL rendering it’d be great to hear them.

What we do over here is to have a set of standalone (interactive) nodes that run X and VGL on mid-range Nvidia GTX cards, but, we don’t have any X or VGL on the cluster compute nodes. Most of our computes only have onboard video cards and our GPU nodes are heavily utilized with computation, but, we’d like to see if there is a room for using the GPU nodes for GL with OOD apps like Ansys or Paraview.


Hi, everybody

We are using such configuration : a graphic node supporting the turbovnc sessions and all the compute nodes running X11 with Nvidia driver. All the opengl codes called inside the script. sh. erb are prefixed by vglrun.

When I’ll be back in my office, I can add our config into this topic, if you’re interested

Jean Marie

Thanks @jms27000! @mcuma sorry I didn’t see this earlier! Yes I believe we have X11 libraries installed on all our compute nodes so that someone may run an interactive session on them, whether they have GPUs or not (and most do not, I can’ say the exact %, but let’s say only 1/4 do as an approximate guess).

So all compute nodes have VirtualGL libraries, and can run X11 sessions, but not all compute nodes have GPUs.

How we actually do segregation or limits, I’m not sure. If two users get scheduled on the same compute node, one requesting the GPU specifically, and the other by accident, what’s to stop the second user from using the GPU? cgroup configurations? Seems like something the scheduler should work out to limit the second user. I can’t say for sure.

@tdockendorf may have more for you.

OK, thanks. Looks like you all run X server then when the nodes are running. Since our admins are not fans of running X server on the computes, I think we’ll stick to our current setup with dedicated interactive nodes with X server and VirtualGL and revisit if users start demanding it on the computes (so far they don’t).

Our way of doing this is awful with Torque/Moab but for SLURM we have set a GRES that is no_consume named vis and if a job requests the vis GRES the SLURM TaskProlog will launch X on the GPU the job has been allocated. Because TaskProlog runs as the user and from within the job’s cgroup the X session is limited to the GPU requested and the CPU and memory resources available to the job. Once the job ends and the cgroup is deleted, the X session is killed as it was part of the cgroup. The SLURM cgroups limit access to GPUs as well as memory and CPU.

This is the code we use:

if [[ "$SLURM_LOCALID" == "0" && "$SLURM_JOB_GRES" == *"vis"* ]]; then
  if [ -n "$CUDA_VISIBLE_DEVICES" ]; then
    FIRSTGPU=$(echo $CUDA_VISIBLE_DEVICES | tr ',' "\n" | head -1)
    setsid /usr/bin/X :${FIRSTGPU} -noreset >& /dev/null &
    sleep 2
    if [ -n "$DISPLAY" ]; then
      echo "export OLDDISPLAY=$DISPLAY"
    echo "export DISPLAY=:$FIRSTGPU"

We have some extra logic in both SPANK plugin, CLI filters and Job Submit filters that inject SLURM_JOB_GRES into job’s environment so we can access that in Prolog and TaskProlog. I can provide links to those if there is interest.

1 Like

@tdockendorf Yes, please! We run a general-purpose Slurm cluster that has a mix of CPU and GPU nodes. I have a couple of end-user groups interested in running applications in OOD that needs accelerated OpenGL (specifically, ChimeraX and Napari), and have been racking my brain on how to do it properly.

We are very interested in whatever code/scripts that already exist, as well as any working xorg.conf and other VirtualGL configuration tricks. We successfully tested OOD with VirtualGL and TurboVNC on a dev node a while back, but were worried about having to run a wide-open X server on every GPU node in production, which may have security implications. But, from your description of using cgroups to limit access within a Slurm job, this will alleviate that.

Thanks very much in advance!

For the TaskProlog we rely on CLI filter Lua plugin. So we have something like this defined in /etc/slurm/cli_filter.lua enabled with CliFilterPlugins=lua:

function slurm_cli_pre_submit(options, cli_type)
  -- Set SLURM_JOB_GRES when a GRES is defined
  if options["gres"] ~= nil then
    local new_gres = gres_defaults(options["gres"])
    options["gres"] = new_gres
    posix.setenv("SLURM_JOB_GRES", options["gres"])
  return slurm.SUCCESS

The way we get the GRES information into Prolog is this SPANK plugin: Add gres_env cli_filter and spank plugin · treydock/slurm@d30f0a3 · GitHub

The GRES information in TaskProlog is present because we inject it into user’s environment with the CLI filter plugin. We also inject the environment variables using job_submit plugin here: Add job_submit/gres_env plugin · treydock/slurm@379553e · GitHub

Let me know if anything is unclear or if more information is needed.

Thank you!

Can you link to a working Xorg.conf please? Thanks again.

I lost my notes on how I generated mine but I think it was something like this if you have NVIDIA GPUs:

nvidia-xconfig -o /tmp/Xorg.conf -a

This is diff of generated config vs what we deploy, so our deployed config will add Virtual lines and FontPath and a few other things, might be something you can get with nvidia-xconfig but not sure. The diff is from a node with 2 GPUs. Let me know if need my full xorg.conf or if you are able to generate one yourself.

 Section "ServerLayout"
     Identifier     "Layout0"
@@ -10,6 +10,7 @@
 Section "Files"
+    FontPath        "/usr/share/fonts/default/Type1"
 Section "InputDevice"
@@ -32,6 +33,8 @@
     Identifier     "Monitor0"
     VendorName     "Unknown"
     ModelName      "Unknown"
+    HorizSync       28.0 - 33.0
+    VertRefresh     43.0 - 72.0
     Option         "DPMS"
@@ -39,6 +42,8 @@
     Identifier     "Monitor1"
     VendorName     "Unknown"
     ModelName      "Unknown"
+    HorizSync       28.0 - 33.0
+    VertRefresh     43.0 - 72.0
     Option         "DPMS"
@@ -64,6 +69,7 @@
     Monitor        "Monitor0"
     DefaultDepth    24
     SubSection     "Display"
+        Virtual     1280 1024
         Depth       24
@@ -74,7 +80,7 @@
     Monitor        "Monitor1"
     DefaultDepth    24
     SubSection     "Display"
+        Virtual     1280 1024
         Depth       24

Tip for anyone finding this thread: Using the latest VirtualGL (which is still sort-of in beta) 2.6.90 (3.0 beta1)
you can use the EGL backend.

My slurm taskprolog sets

if 'CUDA_VISIBLE_DEVICES' in os.environ:
    # Most reliable way to find the dri device:
    gpu_bus_id = commands.getoutput('nvidia-smi --query-gpu=gpu_bus_id --format=csv,noheader').split('\n')[0]
    # Need to strip off first 4 zeros
    dri_device = commands.getoutput('ls -d /sys/bus/pci/devices/{0}/drm/card*'.format(gpu_bus_id[4:])).split('/')[-1]
    print('export VGL_DISPLAY=/dev/dri/{0}'.format(dri_device))

In addition, you need to set the render device permissions. Since the nvidia device should be locked down to the right GPU anyway, I think it’s safe to set

# cat /etc/udev/rules.d/99-virtualgl-dri.rules 
KERNEL=="renderD*", MODE="0666", OWNER="root", GROUP="root"

No need to set up an X server. VirtualGL will run on the correct GPU. Works for all GLX apps I’ve tested (matlab, mathematica, glxgears, etc)