Apptainer Desktop Container Example

Hey there

I’m working on providing a XFCE desktop for users. From previous threads here in the forum I believe there are working setups around. Unfortunately, I wasn’t able to find a full example somewhere and wasn’t able to piece a working one together.

My goal is to provide a XFCE desktop based on Rocky Linux 9 (which our HPC currently runs on). I’m currently using a version of the example code from the OOD documentation (Batch Connect VNC Container Options — Open OnDemand 4.0.0 documentation), adapted for Rocky 9, to create the container:

Bootstrap: docker
From: rockylinux/rockylinux:9-minimal

%environment
  PATH=/opt/TurboVNC/bin:$PATH
  LANGUAGE="en_US.UTF-8"
  LC_ALL="en_US.UTF-8"
  LANG="en_US.UTF-8"

%post
    microdnf -y install dnf
    dnf install -y dnf-plugins-core
    dnf config-manager --set-enabled crb
    dnf -y update && dnf -y upgrade
    dnf install -y epel-release
    dnf install -y xfdesktop xfwm4 xfce4-session xfce4-settings xfce4-terminal
    dnf install -y xkbcomp
    dnf install -y python3-pip xorg-x11-xauth
    pip3 install ts
    dnf install -y https://yum.osc.edu/ondemand/latest/compute/el9Server/x86_64/python3-websockify-0.11.0-1.el9.noarch.rpm
    dnf install -y https://yum.osc.edu/ondemand/latest/compute/el9Server/x86_64/turbovnc-3.1.1-1.el9.x86_64.rpm 
    dnf clean all
    chown root:root /opt/TurboVNC/etc/turbovncserver-security.conf
    rm -rf /var/cache/dnf/*
    rm -f /var/log/*.log

My submit.yml.erb uses the ‘vnc_container’ template from batch_connect:
---
batch_connect:
template: "vnc_container"
websockify_cmd: "/usr/bin/websockify"
container_path: "/path/to/xfce-desktop.sif"
container_bindpath: ""
container_module: ""
container_command: "apptainer"
script:
native:
(SLURM parameters to run the job on our HPC)

Unfortunately, I haven’t found any matching script.sh.erb file to properly run the container. One thread suggested to just copy the xfce.sh script from the batch_connect template (template/desktops/xfce.sh) and run it inside the container. But that didn’t work for me.

I previously created a native desktop (where the xfce desktop environment is running locally on the compute node of the cluster) with some customization and tried to reuse the commands by running them inside the container. The resulting script.sh.erb:

#!/usr/bin/env bash

# Clean the environment
module purge

# Set working directory to home directory
cd "${HOME}"

# xfce configuration maintenance
## Clean up previous monitors
if [[ -f "${HOME}/.config/monitors.xml" ]]; then
  mv "${HOME}/.config/monitors.xml" "${HOME}/.config/monitors.xml.bak"
fi

## Copy over default panel if doesn't exist, otherwise it will prompt the user
PANEL_CONFIG="${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml"
if [[ ! -e "${PANEL_CONFIG}" ]]; then
  mkdir -p "$(dirname "${PANEL_CONFIG}")"
  cp "/etc/xdg/xfce4/panel/default.xml" "${PANEL_CONFIG}"
fi

## Set Xfce4 Terminal as login shell (sets proper TERM)
TERM_CONFIG="${HOME}/.config/xfce4/terminal/terminalrc"
if [[ ! -e "${TERM_CONFIG}" ]]; then
  mkdir -p "$(dirname "${TERM_CONFIG}")"
  sed 's/^ \{4\}//' > "${TERM_CONFIG}" << EOL
    [Configuration]
    CommandLoginShell=TRUE
EOL
else
  sed -i \
    '/^CommandLoginShell=/{h;s/=.*/=TRUE/};${x;/^$/{s//CommandLoginShell=TRUE/;H};x}' \
    "${TERM_CONFIG}"
fi

#
# Launch Xfce Window Manager and Panel
#
echo "Setting environment variables..."

# configure DISPLAY for container
export APPTAINERENV_DISPLAY=$DISPLAY
export SEND_256_COLORS_TO_REMOTE=1
export APPTAINERENV_SEND_256_COLORS_TO_REMOTE="$SEND_256_COLORS_TO_REMOTE"

# Set the user folder
export XDG_CONFIG_HOME="<%= session.staged_root.join("config") %>"
export APPTAINERENV_XDG_CONFIG_HOME="$XDG_CONFIG_HOME"

export XDG_DATA_HOME="<%= session.staged_root.join("share") %>"
export APPTAINERENV_XDG_DATA_HOME="$XDG_DATA_HOME"

export APPTAINERENV_PATH=$PATH
export APPTAINERENV_LD_LIBRARY_PATH=$LD_LIBRARY_PATH

echo "Starting desktop instance"

module restore
set -x

# Launch dbus-launch with full path to prevent conda PATH issues (https://github.com/OSC/ondemand/issues/700)
export $(apptainer exec instance://$INSTANCE_NAME /usr/bin/dbus-launch 2>/dev/null)
export APPTAINERENV_DBUS_SESSION_BUS_ADDRESS=$DBUS_SESSION_BUS_ADDRESS
export APPTAINERENV_DBUS_SESSION_BUS_PID=$DBUS_SESSION_BUS_PID

echo "Starting xfce desktop..."

# Launche xfce window manager
apptainer exec instance://$INSTANCE_NAME xfwm4 --compositor=off --sm-client-disable
#
apptainer exec instance://$INSTANCE_NAME xsetroot -solid "#D3D3D3"
#
apptainer exec instance://$INSTANCE_NAME xfsettingsd --sm-client-disable --daemon
# disable ssh autostart
apptainer exec instance://$INSTANCE_NAME xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
# disable gpg autostart
apptainer exec instance://$INSTANCE_NAME xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s false
# launch the desktop panel (used by HPC to monitor if desktop is still running)
apptainer exec instance://$INSTANCE_NAME xfce4-panel --sm-client-disable

But this did not work as expected. The session (and desktop) is starting, but noVNC can’t connect to it.

Since I’m not sure if the root cause of the this is related to an error in my configuration or if there is some setting missing on our HPC, I’d appreciate if someone could share a working example for running a desktop container so I can test if this is working here.

Please let me know if you want to inspect additional files (or if you maybe already have spotted the error?). I’d really appreciate any support or examples.

We don’t use this at OSC, but I can take a crack at replicating and see what I can see.

The issue is that apptainer expects the script to be inside the container

I use ‘apptainer_bind’ to bind mount it under /tmp, rather than embed it into the container

submit.yml.erb

  run_script: |-

    apptainer exec instance://$INSTANCE_NAME <%= xfce_script %>

I have it working. Here are my files

here is my form.yml

---
title: "XFCE Desktop (Apptainer Container)"
description: "Full XFCE desktop in Apptainer container"
cluster: "fortyfive"

attributes:
  profile:
    label: Operating System / Partition
    widget: select
    required: true
    value: "rocky9-cpu"
    options:
      - ["rocky8-cpu (CPU-only)", "rocky8-cpu"]
      - ["rocky8-gpu (Nvidia passthrough)", "rocky8-gpu"]
      - ["rocky9-cpu (CPU-only)", "rocky9-cpu"]
      - ["rocky9-gpu (Nvidia passthrough)", "rocky9-gpu"]
      - ["ubuntu22 (CPU-only)", "ubuntu22-cpu"]
      - ["ubuntu22 (GPU)", "ubuntu22-gpu"]
      - ["ubuntu24 (CPU-only)", "ubuntu24-cpu"]
      - ["ubuntu24 (GPU)", "ubuntu24-gpu"]
    help: "Select OS and GPU/CPU type."

  bc_num_hours:
    label: Number of hours
    widget: number_field
    value: 4
    min: 1
    max: 48
    step: 1

  bc_num_cpus:
    label: Number of CPUs
    widget: number_field
    value: 2
    min: 1
    max: 16
    step: 1

  memory_gb:
    label: Memory (GB)
    widget: number_field
    value: 8
    min: 4
    max: 128
    step: 1

  email_timer:
    label: Receive email warnings at 50%, 90%, and job end
    widget: check_box
    value: false

form:
  - profile
  - bc_num_hours
  - bc_num_cpus
  - memory_gb
  - email_timer
  - bc_email_on_started
  - bc_vnc_resolution

Here is my submit.yml.erb

<% 
  # Define all profiles here - much cleaner than pipe splitting
  profiles = {
    "rocky8-cpu"  => { container: "#{ENV['HOME']}/ondemand/dev/Rocky8_desktop/rocky8-desktop.sif",
                       xfce_script: "/tmp/xfce-rocky8.sh",
                       os_name: "rocky8",
                       slurm_partition: "eight",
                       use_gpu: false },

    "rocky8-gpu"  => { container: "#{ENV['HOME']}/ondemand/dev/Rocky8_desktop/rocky8-desktop.sif",
                       xfce_script: "/tmp/xfce-rocky8.sh",
                       os_name: "rocky8",
                       slurm_partition: "gpu-8",
                       use_gpu: true },

    "rocky9-cpu"  => { container: "#{ENV['HOME']}/ondemand/dev/Rocky9_desktop/rocky9-desktop.sif",
                       xfce_script: "/tmp/xfce-rocky9.sh",
                       os_name: "rocky9",
                       slurm_partition: "nine",
                       use_gpu: false },

    "rocky9-gpu"  => { container: "#{ENV['HOME']}/ondemand/dev/Rocky9_desktop/rocky9-desktop.sif",
                       xfce_script: "/tmp/xfce-rocky9.sh",
                       os_name: "rocky9",
                       slurm_partition: "gpu-9",
                       use_gpu: true },

    "ubuntu22-cpu" => { container: "#{ENV['HOME']}/ondemand/dev/ubuntu22-desktop.sif",
                        xfce_script: "/tmp/xfce-ubuntu22.sh",
                        os_name: "ubuntu22",
                        slurm_partition: "nine",
                        use_gpu: false },

    "ubuntu22-gpu" => { container: "#{ENV['HOME']}/ondemand/dev/ubuntu22-desktop.sif",
                        xfce_script: "/tmp/xfce-ubuntu22.sh",
                        os_name: "ubuntu22",
                        slurm_partition: "gpu-9",
                        use_gpu: true },

    "ubuntu24-cpu" => { container: "#{ENV['HOME']}/ondemand/dev/ubuntu24-desktop.sif",
                        xfce_script: "/tmp/xfce-ubuntu24.sh",
                        os_name: "ubuntu24",
                        slurm_partition: "nine",
                        use_gpu: false },

    "ubuntu24-gpu" => { container: "#{ENV['HOME']}/ondemand/dev/ubuntu24-desktop.sif",
                        xfce_script: "/tmp/xfce-ubuntu24.sh",
                        os_name: "ubuntu24",
                        slurm_partition: "gpu-9",
                        use_gpu: true },
  }

# apply values from predefined profile
  p = profiles[profile]
  container       = p[:container]
  xfce_script     = p[:xfce_script]
  os_name         = p[:os_name]
  slurm_partition = p[:slurm_partition]
  use_gpu         = p[:use_gpu]
%>


batch_connect:
  template: "vnc_container"
  websockify_cmd: "/usr/bin/websockify"
  container_path: "<%= container %>"
  container_command: "apptainer"
  container_module: ""
  container_bindpath: "/mnt/zeta,/opt/modules,/var,/run,<%= staged_root %>/tmp:/tmp"

  <% if use_gpu %>
  container_start_args: ["--nv"]
  <% else %>
  container_start_args: []
  <% end %>

  before_script: |
    echo "Running before_script"

  run_script: |-
    # DISPLAY and other important variables into the container
    export APPTAINERENV_DISPLAY=${DISPLAY}
    export APPTAINERENV_XDG_RUNTIME_DIR=/run/user/$(id -u)
  
    echo "=== Starting run_script at $(date) ==="
    echo "DISPLAY=${DISPLAY}"
    echo "APPTAINERENV_DISPLAY=${APPTAINERENV_DISPLAY}"
    echo "APPTAINERENV_XDG_RUNTIME_DIR=${APPTAINERENV_XDG_RUNTIME_DIR}"

    apptainer exec instance://$INSTANCE_NAME <%= xfce_script %>

  after_script: |
    echo "Running after_script"

script:
  native:
    - "--job-name=xfce-<%= os_name %>"
    - "--cpus-per-task=<%= bc_num_cpus %>"
    - "--partition=<%= slurm_partition %>"
    - "--mem=<%= memory_gb.blank? ? 8 : memory_gb %>G"
    - "--time=<%= bc_num_hours %>:00:00"
    <% if use_gpu %>
    - "--gres=gpu:1"
    <% end %>
    <% if email_timer == "1" || email_timer.to_s == "true" %>
    - "--mail-type=TIME_LIMIT_90,TIME_LIMIT_50,TIME_LIMIT"
    - "--mail-user=<%= ENV['USER'] %>@uidaho.edu"
    <% end %>

Here is my template/tmp/xfce-rocky9.sh

#!/bin/bash
# ==============================================================================
# Robust XFCE startup script for Open OnDemand vnc_container + Apptainer +NVidia
# - Private /tmp bind-mounted from host
# - GPU detection + proper compositing for NVIDIA
# - Safe lock cleanup (private /tmp)
# - Proper XDG_RUNTIME_DIR and dbus handling
# - Disable problematic autostart services
# - Backup monitors.xml to avoid multi-monitor issues
# - Graceful fallback and logging
# ==============================================================================

set -euo pipefail

echo "=== Starting XFCE session inside container at $(date) ==="
echo "DISPLAY=${DISPLAY:-NOT_SET}"
echo "XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-NOT_SET}"
echo "User: $(whoami)  UID: $(id -u)"

# Set working directory to home directory
cd "${HOME}"

# ====================== Clean stale X locks ======================
# Safe cleanup of stale X locks (critical with private /tmp bind)
rm -f /tmp/.X*-lock /tmp/.X11-unix/X* 2>/dev/null || true
echo "X locks cleaned."

# Create ICE directory with correct permissions (removes _IceTransmkdir owner warning)
mkdir -p /tmp/.ICE-unix
chmod 1777 /tmp/.ICE-unix 2>/dev/null || true

# # ====================== XDG_RUNTIME_DIR ======================
# # Set up XDG_RUNTIME_DIR (fixes "Cannot open display" and dbus issues)
# export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/tmp/runtime-$(id -u)}"
# mkdir -p "$XDG_RUNTIME_DIR"
# chmod 700 "$XDG_RUNTIME_DIR"
# echo "XDG_RUNTIME_DIR set to: $XDG_RUNTIME_DIR"

# # Ensure runtime subdirs that GVFS expects
# mkdir -p "${XDG_RUNTIME_DIR}/gvfsd" "${XDG_RUNTIME_DIR}/bus"
# chmod 700 "${XDG_RUNTIME_DIR}/gvfsd" "${XDG_RUNTIME_DIR}/bus" 2>/dev/null || true

# Start dbus if needed (helps GVFS and reduces fallback warnings)
if [[ -z "${DBUS_SESSION_BUS_ADDRESS:-}" ]]; then
    eval "$(dbus-launch --sh-syntax --exit-with-session)"
    echo "DBUS_SESSION_BUS_ADDRESS=${DBUS_SESSION_BUS_ADDRESS}"
fi

# ====================== Prevent blank screen issues ======================
# Remove any preconfigured monitors.xml (prevents blank/black screen on some nodes)
if [[ -f "${HOME}/.config/monitors.xml" ]]; then
    mv "${HOME}/.config/monitors.xml" "${HOME}/.config/monitors.xml.bak" 2>/dev/null || true
    echo "Backed up monitors.xml."
fi

# Copy default panel config if missing (prevents first-run prompts)
PANEL_CONFIG="${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml"
if [[ ! -e "${PANEL_CONFIG}" ]]; then
    mkdir -p "$(dirname "${PANEL_CONFIG}")"
    cp "/etc/xdg/xfce4/panel/default.xml" "${PANEL_CONFIG}" 2>/dev/null || true
    echo "Copied default panel config."
fi

# ====================== Disable noisy autostart services ======================
# Disable autostart services that cause noise or failures in containers
AUTOSTART="${HOME}/.config/autostart"
rm -rf "${AUTOSTART}" 2>/dev/null || true
mkdir -p "${AUTOSTART}"

for svc in "pulseaudio" "rhsm-icon" "spice-vdagent" "tracker-extract" \
           "tracker-miner-apps" "tracker-miner-user-guides" \
           "xfce4-power-manager" "xfce-polkit" "nm-applet"; do
    cat > "${AUTOSTART}/${svc}.desktop" << EOF
[Desktop Entry]
Hidden=true
EOF
done
echo "Disabled unnecessary autostart services."

## Set Xfce4 Terminal as login shell (sets proper TERM)
TERM_CONFIG="${HOME}/.config/xfce4/terminal/terminalrc"
if [[ ! -e "${TERM_CONFIG}" ]]; then
    mkdir -p "$(dirname "${TERM_CONFIG}")"
    cat > "${TERM_CONFIG}" << EOF
[Configuration]
CommandLoginShell=TRUE
EOF
fi

# ====================== GPU Acceleration (NVIDIA) ======================
# Enable GPU acceleration in XFCE when available
if command -v nvidia-smi >/dev/null 2>&1; then
    echo "NVIDIA GPU detected via nvidia-smi - enabling hardware acceleration"
    # Turn on compositing so xfwm4 can use GPU (GLX)
    xfconf-query -c xfwm4 -p /general/use_compositing -n -t bool -s true 2>/dev/null || true
    # Set vblank_mode to GLX (usually best for NVIDIA stability)
    xfconf-query -c xfwm4 -p /general/vblank_mode -n -t int -s 1 2>/dev/null || true
else
    # Keep compositor off on CPU-only nodes for performance
    echo "XFWM4 compositor disabled."
    xfconf-query -c xfwm4 -p /general/use_compositing -n -t bool -s false 2>/dev/null || true
fi

# ====================== Basic theme/performance tweaks ======================
# Force software rendering fallback + disable effects
xfconf-query -c xsettings -p /Net/ThemeName -n -t string -s "Adwaita" 2>/dev/null || true
xfconf-query -c xfwm4 -p /general/theme -n -t string -s "Default" 2>/dev/null || true

# Launch Desktop (blocks until user logs out of the desktop)
echo "Launching xfce4-session on DISPLAY=${DISPLAY} (performance mode) ..."

exec xfce4-session

Here is my rocky9_desktop.def

Bootstrap: docker
From: nvidia/cuda:13.2.0-devel-rockylinux9

%labels
  maintainer "James O'Dell"
  version "1.0"
  description "Rocky 9 XFCE for OOD vnc_container"

%environment
export LANGUAGE="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LANG="en_US.UTF-8"
export PATH=/opt/TurboVNC/bin:$PATH

%post
  dnf -y update
  
  # Change locale 
  dnf -y install glibc-langpack-en glibc-locale-source glibc-common
  localedef -i en_US -f UTF-8 en_US.UTF-8 
  echo 'LANG="en_US.UTF-8"' > /etc/locale.conf
  # SEE %environment
  # echo 'export LANG="en_US.UTF-8"' > /environment
  # echo 'export LC_ALL="en_US.UTF-8"' >> /environment
  
  # EPEL
  dnf -y install epel-release
  /usr/bin/crb enable
  
  # Switch from Minimal to Server
  dnf -y groupinstall "Server" --allowerasing 
  
  # XFCE
  dnf -y groupinstall "Xfce" "base-x"
  dnf -y install xterm firefox xorg-x11-xauth dbus-x11 VirtualGL libglvnd-glx
  dnf -y remove xfce4-screensaver
  
  # Development
  dnf -y groupinstall "Development Tools"
  dnf -y install geany cmake
  
  # VNC
  dnf -y install python3-pip
  # ts (timeout) utility
  pip3 install ts
  # OSC OnDemand packages
  dnf install -y https://yum.osc.edu/ondemand/latest/compute/el9Server/x86_64/python3-websockify-0.11.0-2.el9.noarch.rpm
  dnf install -y https://yum.osc.edu/ondemand/latest/compute/el9Server/x86_64/turbovnc-3.2.1-1.el9.x86_64.rpm
  # Critical: Set ownership and strict permissions on security conf
  chown root:root /opt/TurboVNC/etc/turbovncserver-security.conf
  chmod 0600 /opt/TurboVNC/etc/turbovncserver-security.conf
  ls -l /opt/TurboVNC/etc/turbovncserver-security.conf
  
  # Missing utils
  dnf -y install yum-utils compat-openssl11 htop
  
  # Lmod + EasyBuild
  # apptainer --bind /opt/modules
  mkdir -p /opt/modules
  chmod 755 /opt/modules
  dnf -y install Lmod
  pip3 install easybuild
  
  # Clean up to reduce image size
  dnf clean all
  rm -rf /var/cache/dnf/*

%runscript
  exec /bin/bash "$@"

It’s still a work in progress. Hopefully someone finds it useful.