MATLAB on OOD 3.1: App has not been initialized or does not exist

From Interactive Apps like this:

It doesn’t seem to like the form.yml you have and is trying to find another.

I restarted Apache, ran update_ood_portal, and chose ā€œRestart Web Serverā€ so I don’t see those errors anymore but the Desktop is failing to start still:

Launching desktop 'xfce'...
xfce4-session: Cannot open display: .
Type 'xfce4-session --help' for usage.
Desktop 'xfce' ended with 1 status...
Cleaning up...

Here’s job_script_content.sh

cd /home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d

# Export useful connection variables
export host
export port

# Generate a connection yaml file with given parameters
create_yml () {
  echo "Generating connection YAML file..."
  (
    umask 077
    echo -e "host: $host\nport: $port\npassword: $password" > "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/connection.yml"
  )
}

# Cleanliness is next to Godliness
clean_up () {
  echo "Cleaning up..."
  [[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/clean.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/clean.sh"
  [[ ${SCRIPT_PID} ]] && pkill -P ${SCRIPT_PID} || :
  pkill -P $$
  exit ${1:-0}
}

# Source in all the helper functions
source_helpers () {
  # Generate random integer in range [$1..$2]
  random_number () {
    shuf -i ${1}-${2} -n 1
  }
  export -f random_number

  port_used_python() {
    python -c "import socket; socket.socket().connect(('$1',$2))" >/dev/null 2>&1
  }

  port_used_python3() {
    python3 -c "import socket; socket.socket().connect(('$1',$2))" >/dev/null 2>&1
  }

  port_used_nc(){
    nc -w 2 "$1" "$2" < /dev/null > /dev/null 2>&1
  }

  port_used_lsof(){
    lsof -i :"$2" >/dev/null 2>&1
  }

  port_used_bash(){
    local bash_supported=$(strings /bin/bash 2>/dev/null | grep tcp)
    if [ "$bash_supported" == "/dev/tcp/*/*" ]; then
      (: < /dev/tcp/$1/$2) >/dev/null 2>&1
    else
      return 127
    fi
  }

  # Check if port $1 is in use
  port_used () {
    local port="${1#*:}"
    local host=$((expr "${1}" : '\(.*\):' || echo "localhost") | awk 'END{print $NF}')
    local port_strategies=(port_used_nc port_used_lsof port_used_bash port_used_python port_used_python3)

    for strategy in ${port_strategies[@]};
    do
      $strategy $host $port
      status=$?
      if [[ "$status" == "0" ]] || [[ "$status" == "1" ]]; then
        return $status
      fi
    done

    return 127
  }
  export -f port_used

  # Find available port in range [$2..$3] for host $1
  # Default host: localhost
  # Default port range: [2000..65535]
  # returns error code (0: success, 1: failed)
  # On success, the chosen port is echoed on stdout.
  find_port () {
    local host="${1:-localhost}"
    local min_port=${2:-2000}
    local max_port=${3:-65535}
    local port_range=($(shuf -i ${min_port}-${max_port}))
    local retries=1 # number of retries over the port range if first attempt fails
    for ((attempt=0; attempt<=$retries; attempt++)); do
      for port in "${port_range[@]}"; do
        if port_used "${host}:${port}"; then
continue
        fi
        echo "${port}"
        return 0 # success
      done
    done

    echo "error: failed to find available port in range ${min_port}..${max_port}" >&2
    return 1 # failure
  }
  export -f find_port

  # Wait $2 seconds until port $1 is in use
  # Default: wait 30 seconds
  wait_until_port_used () {
    local port="${1}"
    local time="${2:-30}"
    for ((i=1; i<=time*2; i++)); do
      port_used "${port}"
      port_status=$?
      if [ "$port_status" == "0" ]; then
        return 0
      elif [ "$port_status" == "127" ]; then
         echo "commands to find port were either not found or inaccessible."
         echo "command options are lsof, nc, bash's /dev/tcp, or python (or python3) with socket lib."
         return 127
      fi
      sleep 0.5
    done
    return 1
  }
  export -f wait_until_port_used

  # Generate random alphanumeric password with $1 (default: 8) characters
  create_passwd () {
    tr -cd 'a-zA-Z0-9' < /dev/urandom 2> /dev/null | head -c${1:-8}
  }
  export -f create_passwd
}
export -f source_helpers

source_helpers

# Set host of current machine
host=$(hostname)

[[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/before.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/before.sh"

echo "Script starting..."
"/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/script.sh" &
SCRIPT_PID=$!

[[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/after.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/after.sh"

# Create the connection yaml file
create_yml

# Wait for script process to finish
wait ${SCRIPT_PID} || clean_up 1

# Exit cleanly
clean_up

OOD thinks the application is a http server, not a vnc application.

You cloned the repo from us - but did you keep this portion specifying it’s a vnc application?

Thanks @Micket I’ve tried this on RHEL 7 and ran into some difficulties. I was hoping to get the VNC version working and I’m getting close.

Which Jupyter app did you find easy? There are a few posted from several universities.

For the Matlab app yes:

batch_connect:
  template: vnc
script:
  native:
  <%- slurm_args.each do |arg| %>
    - "<%= arg %>"
  <%- end %>

The error I get from MATLAB is:

(xfsettingsd:153295): xfsettingsd-WARNING **: 10:47:59.621: Failed to get the _NET_NUMBER_OF_DESKTOPS property.
Killing Xvnc process ID 153271

OK so from Interactive Apps I still get this error:

This app does not supply a sub app form file under the directory '/var/www/ood/apps/sys/bc_osc_matlab/local'

As for using the MATLAB icon on the left side on the node here are the errors from /var/log/messages:

Oct 16 12:06:50 ax08 systemd[1]: realmd.service: Deactivated successfully.
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=1822857372 pid=153940 comm="xfce4-panel --sm-client-disable ")
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.a11y.Bus'
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.gtk.vfs.Daemon' requested by ':1.6' (uid=1822857372 pid=153937 comm="xfwm4 --compositor=off --sm-client-disable ")
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.xfce.Xfconf' requested by ':1.5' (uid=1822857372 pid=153939 comm="xfsettingsd --sm-client-disable ")
Oct 16 12:06:53 ax08 org.a11y.Bus[153954]: dbus-daemon[153954]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=1822857372 pid=153940 comm="xfce4-panel --sm-client-disable ")
Oct 16 12:06:53 ax08 org.a11y.Bus[153954]: dbus-daemon[153954]: Successfully activated service 'org.a11y.atspi.Registry'
Oct 16 12:06:53 ax08 org.a11y.Bus[153965]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.xfce.Xfconf'
Oct 16 12:06:53 ax08 org.a11y.Bus[153965]: X connection to :1 broken (explicit kill or server shutdown).
Oct 16 12:07:20 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Wow what would cause this from the MATLAB app?

Oct 16 12:28:12 ax08 kernel: xfsettingsd[154358]: segfault at 9 ip 0000150de262f53f sp 00007ffd95a8ce70 error 4 in libGLX_mesa.so.0.0.0[150de2618000+3f000] likely on CPU 0 (core 0, socket 0)
Oct 16 12:28:12 ax08 kernel: Code: c7 9a fe ff 48 8d 15 98 92 02 00 be 07 00 00 00 48 89 c7 48 89 c3 e8 10 9e fe ff 31 d2 48 89 df 89 c6 e8 b4 9a fe ff 48 89 c7 <0f> b6 40 09 89 05 3f 0b 04 00 0f b6 47 08 88 44 24 1f e8 ca 9e fe
Oct 16 12:28:12 ax08 kernel: xfwm4[154355]: segfault at 9 ip 0000146c49fa853f sp 00007ffc3717d700 error 4 in libGLX_mesa.so.0.0.0[146c49f91000+3f000] likely on CPU 24 (core 0, socket 0)
Oct 16 12:28:12 ax08 kernel: Code: c7 9a fe ff 48 8d 15 98 92 02 00 be 07 00 00 00 48 89 c7 48 89 c3 e8 10 9e fe ff 31 d2 48 89 df 89 c6 e8 b4 9a fe ff 48 89 c7 <0f> b6 40 09 89 05 3f 0b 04 00 0f b6 47 08 88 44 24 1f e8 ca 9e fe
Oct 16 12:28:12 ax08 systemd[1]: Started Process Core Dump (PID 154364/UID 0).
Oct 16 12:28:12 ax08 kernel: enp96s0f1 speed is unknown, defaulting to 1000
Oct 16 12:28:12 ax08 systemd[1]: Started Process Core Dump (PID 154365/UID 0).
Oct 16 12:28:12 ax08 kernel: enp96s0f1 speed is unknown, defaulting to 1000
Oct 16 12:28:12 ax08 systemd-coredump[154366]: Process 154358 (xfsettingsd) of user 1822857372 dumped core.#012#012Stack trace of thread 154358:#012#0  0x0000150de262f53f driswCreateScreenDriver (libGLX_mesa.so.0 + 0x3053f)#012#1  0x0000150de2635068 __glXInitialize (libGLX_mesa.so.0 + 0x36068)#012#2  0x0000150de2631db5 glXQueryServerString (libGLX_mesa.so.0 + 0x32db5)#012#3  0x0000150de56a444f epoxy_glx_version (libepoxy.so.0 + 0xc544f)#012#4  0x0000150de6d35730 gdk_x11_screen_init_gl (libgdk-3.so.0 + 0x92730)#012#5  0x0000150de6d3a1c5 _gdk_x11_screen_new (libgdk-3.so.0 + 0x971c5)#012#6  0x0000150de6d26dcd _gdk_x11_display_open (libgdk-3.so.0 + 0x83dcd)#012#7  0x0000150de6cdaadb gdk_display_manager_open_display (libgdk-3.so.0 + 0x37adb)#012#8  0x0000150de660a8fb gtk_init_check (libgtk-3.so.0 + 0x20a8fb)#012#9  0x0000556207c0d930 main (xfsettingsd + 0x8930)#012#10 0x0000150de5c29590 __libc_start_call_main (libc.so.6 + 0x29590)#012#11 0x0000150de5c29640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)#012#12 0x0000556207c0df95 _start (xfsettingsd + 0x8f95)#012ELF object binary architecture: AMD x86-64
Oct 16 12:28:12 ax08 systemd[1]: systemd-coredump@2-154364-0.service: Deactivated successfully.
Oct 16 12:28:12 ax08 systemd-coredump[154368]: Process 154355 (xfwm4) of user 1822857372 dumped core.#012#012Stack trace of thread 154355:
#012#0  0x0000146c49fa853f driswCreateScreenDriver (libGLX_mesa.so.0 + 0x3053f)
#012#1  0x0000146c49fae068 __glXInitialize (libGLX_mesa.so.0 + 0x36068)
#012#2  0x0000146c49faadb5 glXQueryServerString (libGLX_mesa.so.0 + 0x32db5)?
#012#3  0x0000146c4e72e44f epoxy_glx_version (libepoxy.so.0 + 0xc544f)
#012#4  0x0000146c4e5a2730 gdk_x11_screen_init_gl (libgdk-3.so.0 + 0x92730)
#012#5  0x0000146c4e5a71c5 _gdk_x11_screen_new (libgdk-3.so.0 + 0x971c5)
#012#6  0x0000146c4e593dcd _gdk_x11_display_open (libgdk-3.so.0 + 0x83dcd)
#012#7  0x0000146c4e547b0c gdk_display_manager_open_display (libgdk-3.so.0 + 0x37b0c)
#012#8  0x0000146c4de0a8fb gtk_init_check (libgtk-3.so.0 + 0x20a8fb)
#012#9  0x0000146c4de0a92d gtk_init (libgtk-3.so.0 + 0x20a92d)
#012#10 0x0000563fa779bc39 main (xfwm4 + 0x12c39)
#012#11 0x0000146c4d429590 __libc_start_call_main (libc.so.6 + 0x29590)
#012#12 0x0000146c4d429640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)
#012#13 0x0000563fa779e4c5 _start (xfwm4 + 0x154c5)
#012ELF object binary architecture: AMD x86-64

OK this error only happens after the initial login then goes away. We are using CAS and it’s similar to this 2020 RStudio thread. I tried adding CASScope to out conf file but get:

Oct 16 14:30:41 ood.ouruni.edu httpd[1402821]: AH00526: Syntax error on line 17 of /etc/httpd/conf.d/cas.conf:
Oct 16 14:30:41 ood.ouruni.edu httpd[1402821]: CASScope not allowed here

I’m not sure what would cause that core dump. Google searches don’t provide a lot of hits, but if I had to guess it may be a missing library or driver. Like you may be missing a video card driver?

I also don’t know much about CASScope, but the error indicates it’s not in the right place (it’s global?). Like it should be held within all the Location directives, not applied globally.

Which is to say maybe it’s better placed in the auth section of your ood_portal.yml.

Where in the file should it go and what is the value? CAS?

The auth section of your ood_portal.yml. Using / is super liberal, but it’ll account for both /pun stuff and /node and /rnode URIs.

auth:
  - CASScope /
# - the other CAS stuff you have

I think that ā€˜app does not supply a sub app’ error is now resolved. Still seeing these error on Desktop

[websockify]: pid: 158072 (proxying 14087 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
[websockify]: failed to launch!
Cleaning up...
/usr/bin/iceauth:  creating new authority file /run/user/1822857372/ICEauthority
Killing Xvnc process ID 158025
Desktop 'xfce' ended with 1 status...

/var/spool/slurmd/job00117/slurm_script: line 206: /opt/websockify/run: No such file or directory

And on the node:

X connection to :1 broken (explicit kill or server shutdown).

I missed a crucial step to configure Apache to add to /etc/ood/config/ood_portal.yml:

host_regex: '[\w.-]+\.path\.to\.ouruni\.edu'
node_uri: '/node'
rnode_uri: '/rnode'

I also had to put:

auth:
  - 'AuthType CAS'
  - 'CASScope /'

And I had incorrect alignment of:

   job:
   batch_connect:

The latter was indented 2 additional spaces :frowning:

However I’m now getting the No VNC error. The only error in the logs I see is:
[authz_core:error] [pid 1478819:tid 1478974] [client x.x.x.x:57112] AH01630: client denied by server configuration: /etc/httpd/htdocs

I’m not browsing any of these directories and that one does not exist. The other referer log from Apache has this:

[Thu Oct 17 14:52:32.401054 2024] [lua:info] [pid 1478819:tid 1478927] [client 128.59.216.151:56944] res_content_length="0" req_accept_encoding="gzip, deflate, br, zstd" res_content_encoding="" req_method="GET" res_content_location="" res_content_type="" allowed_hosts="ood.ouruni.edu" req_status="302" log_time="2024-10-17T18:52:32.400882.0Z" req_port="443" req_filename="/etc/httpd/htdocs" req_cache_control="" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_accept_language="en-us,en;q=0.9" remote_user="myuser" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_handler="" time_proxy="0" req_is_websocket="false" log_hook="ood" req_origin="" req_accept_charset="" time_user_map="0.004" res_location="/pun/sys/dashboard/" req_protocol="HTTP/1.1" local_user="rk3199" req_content_type="" res_content_language="" req_user_ip="x.x.x.x" req_uri="/nginx/stop" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_server_name="ood.ouruni.edu" req_is_https="true" res_content_disp="" req_hostname="ood.ouruni.edu", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions

Does this sound familiar to anyone?

Thanks @jeff.ohrstrom for your help today.

Edit: the log actually starts with authz_core:error could this be CAS-related?

I tested KDE the errors are differnt, from /var/log/messages, any clues here?

Oct 17 22:51:00 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_search_xapian: Could not obtain lock for Xapian Database. This is bad
Oct 17 22:51:00 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_indexer_agent: "DatabaseLockError" "DatabaseLockError: Unable to get write lock on /home/myuser/.local/share/akonadi/search_db/notes/: already locked"
Oct 17 22:51:05 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Activating service name='org.freedesktop.Tracker3.Miner.Extract' requested by ':1.13' (uid=1822857372 pid=195841 comm="/usr/libexec/tracker-miner-fs-3 ")
Oct 17 22:51:05 ax08 tracker-extract[196982]: tracker_module_manager_load_modules: assertion 'initialized == TRUE' failed
Oct 17 22:51:05 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Successfully activated service 'org.freedesktop.Tracker3.Miner.Extract'
Oct 17 22:51:10 ax08 drkonqi[196298]: org.kde.drkonqi.bugzilla: ProtocolException: "bugs.kde.org: Socket operation timed out" 149 "bugs.kde.org: Socket operation timed out" "bugs.kde.org: Socket operation timed out"
Oct 17 22:51:10 ax08 drkonqi[196298]: org.kde.drkonqi: "bugs.kde.org: Socket operation timed out"
Oct 17 22:51:10 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_search_xapian: Could not obtain lock for Xapian Database. This is bad
Oct 17 22:51:10 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_indexer_agent: "DatabaseLockError" "DatabaseLockError: Unable to get write lock on /home/myuser/.local/share/akonadi/search_db/collections/: already locked"
Oct 17 22:51:10 ax08 akonadiserver[196282]: org.kde.pim.akonadiserver: Subscriber Akonadi::Server::NotificationSubscriber(0x14b82002b530) identified as "AgentBaseChangeRecorder - 94548651063584"
Oct 17 22:51:14 ax08 xdg-desktop-por[195984]: Failed to create secret proxy: Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached
Oct 17 22:51:14 ax08 xdg-desktop-por[195984]: No skeleton to export
Oct 17 22:51:14 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Successfully activated service 'org.freedesktop.portal.Desktop'
Oct 17 22:51:15 ax08 goa-daemon[84858]: secret_password_clear_sync() failed: Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached
Oct 17 22:51:15 ax08 goa-daemon[84858]: Unable to clean-up stale keyring entries: Failed to delete credentials from the keyring
Oct 17 22:51:16 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Here are the compute node system logs when launching a XFCE desktop. Is this GDM keyring an issue? I found a person who mentioned disabling it in a pam module, at least the autostart, which I did. How about the Tracker logs? Should that just be disabled in the XFCE configs?

Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.a11y.Bus'
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.xfce.Xfconf' requested by ':1.2' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.xfce.Xfconf'
Oct 18 09:55:24 ax08 org.a11y.Bus[230035]: dbus-daemon[230035]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 org.a11y.Bus[230035]: dbus-daemon[230035]: Successfully activated service 'org.a11y.atspi.Registry'
Oct 18 09:55:24 ax08 org.a11y.Bus[230046]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.Daemon' requested by ':1.5' (uid=1822857372 pid=230050 comm="xfwm4 ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.thumbnails.Thumbnailer1' requested by ':1.10' (uid=1822857372 pid=230143 comm="xfdesktop ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.UDisks2VolumeMonitor' requested by ':1.11' (uid=1822857372 pid=230151 comm="/usr/lib64/tumbler-1/tumblerd ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.UDisks2VolumeMonitor'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.Sources5' requested by ':1.15' (uid=1822857372 pid=230159 comm="/usr/libexec/evolution-data-server/evolution-alarm")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.thumbnails.Thumbnailer1'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.Metadata' requested by ':1.10' (uid=1822857372 pid=230143 comm="xfdesktop ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.OnlineAccounts' requested by ':1.19' (uid=1822857372 pid=230253 comm="/usr/libexec/evolution-source-registry ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.Metadata'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.Sources5'
Oct 18 09:55:25 ax08 goa-daemon[230307]: goa-daemon version 3.40.0 starting
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.Identity' requested by ':1.23' (uid=1822857372 pid=230307 comm="/usr/libexec/goa-daemon ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.secrets' requested by ':1.23' (uid=1822857372 pid=230307 comm="/usr/libexec/goa-daemon ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.Identity'
Oct 18 09:55:25 ax08 gnome-keyring-daemon[230328]: couldn't connect to control socket at: /tmp/xdg_runtime/keyring/control: Connection refused
Oct 18 09:55:25 ax08 gnome-keyring-d[230328]: couldn't connect to control socket at: /tmp/xdg_runtime/keyring/control: Connection refused
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.secrets'
Oct 18 09:55:25 ax08 gnome-keyring-daemon[230328]: daemon/gkd-main.c:931 Couldn't get object path: GDBus.Error:org.freedesktop.login1.NoSuchSession: No session 'c32' known
Oct 18 09:55:25 ax08 gnome-keyring-d[230328]: daemon/gkd-main.c:931 Couldn't get object path: GDBus.Error:org.freedesktop.login1.NoSuchSession: No session 'c32' known
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.OnlineAccounts'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.Calendar8' requested by ':1.15' (uid=1822857372 pid=230159 comm="/usr/libexec/evolution-data-server/evolution-alarm")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.Calendar8'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='ca.desrt.dconf' requested by ':1.27' (uid=1822857372 pid=230355 comm="/usr/libexec/evolution-calendar-factory ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.AddressBook10' requested by ':1.27' (uid=1822857372 pid=230355 comm="/usr/libexec/evolution-calendar-factory ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'ca.desrt.dconf'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.AddressBook10'
Oct 18 09:55:25 ax08 NetworkManager[1589]: <info>  [1729259725.5020] agent-manager: agent[0b192c68d9ec319d,:1.3081/org.freedesktop.nm-applet/1822857372]: agent registered


Oct 18 09:55:40 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.Tracker3.Miner.Extract' requested by ':1.12' (uid=1822857372 pid=230149 comm="/usr/libexec/tracker-miner-fs-3 ")
Oct 18 09:55:40 ax08 tracker-extract[230518]: tracker_module_manager_load_modules: assertion 'initialized == TRUE' failed
Oct 18 09:55:40 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.Tracker3.Miner.Extract'
Oct 18 09:55:52 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Notice that on the node there is an active xfce session:

ps -ef|grep -i xfce
myuser    230022  229994  0 09:55 ?        00:00:00 xfce4-session
myuser    230131  230022  0 09:55 ?        00:00:01 xfce4-panel
myuser    230173  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libsystray.so 6 12582924 systray Status Tray Plugin Provides status notifier items (application indicators) and legacy systray items
myuser    230174  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libpulseaudio-plugin.so 8 12582925 pulseaudio PulseAudio Plugin Adjust the audio volume of the PulseAudio sound system
myuser    230326  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libactions.so 14 12582926 actions Action Buttons Log out, lock or other system actions

I don’t believe there’s anything wrong on the compute side. When we saw your console logs the websocket connection only lasted 5 ms or so — that’s not long enough to reach XFCE. Such a short duration suggests to me that apache immediately closed the connection and never even tried to forward it to websockify & XFCE on the compute node.

If you’re still getting this error in apache logs - that’s the issue, not XFCE.

AH01630: client denied by server configuration

Curiously, no I no longer see those logs after enabling debug of trace3 level. The last instance was:
[Thu Oct 17 16:41:34.222066 2024] [authz_core:error] [pid 1478819:tid 1478990] [client x.x.x.x:58892] AH01630: client denied by server configuration: /etc/httpd/htdocs

Is there any logging level I can add/increase to the authentication side?

If you have trace3 enabled then it should be saying something about why you’re being disconnected. Look for the websockfiy URLs.

20241018-apache-logs.txt (1.5 MB)
Attached are some trace3 logs from today, 10/18/24. A snip showing websockify, anything stick out here?
[Fri Oct 18 08:59:41.463864 2024] [lua:info] [pid 1511513:tid 1511631] [client x.x.x.x:64979] req_hostname="ood.ouruni.edu" req_origin="" req_is_websocket="false" time_proxy="1.637" req_protocol="HTTP/1.1" time_user_map="0.004" req_port="443" req_content_type="" remote_user="myuser" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_status="200" req_handler="proxy-server" allowed_hosts="ood.ouruni.edu" req_is_https="true" req_method="GET" log_hook="ood" req_server_name="ood.ouruni.edu" res_content_disp="" res_content_encoding="" log_time="2024-10-18T12:59:41.463639.0Z" req_filename="proxy:http://localhost/pun/sys/dashboard/noVNC-1.3.0/vnc.html?autoconnect=true&path=rnode%2Fax.ouruni.edu%2F37917%2Fwebsockify&resize=remote&password=xxx&compression=6&quality=2&commit=Launch+Desktop" req_cache_control="" req_uri="/pun/sys/dashboard/noVNC-1.3.0/vnc.html" local_user="myuser" res_location="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" log_id="ZxJbvQhgr3iVfwHaTZQOIQAAABA" req_accept_language="en-us,en;q=0.9" req_accept_encoding="gzip, deflate, br, zstd" res_content_language="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_user_ip="x.x.x.x" req_accept_charset="" res_content_length="15280" res_content_type="text/html; charset=utf-8" res_content_location="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions

edit: from output.log:

Launching desktop 'xfce'...
[websockify]: pid: 234809 (proxying 20989 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
Gtk-Message: 10:44:11.408: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.409: Failed to load module "window-decorations-gtk-module"

(xfwm4:234839): xfwm4-WARNING **: 10:44:11.527: Unsupported GL renderer (llvmpipe (LLVM 17.0.6, 256 bits)).
Gtk-Message: 10:44:11.580: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.580: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.620: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.620: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.671: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.671: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.698: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.699: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.769: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.769: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.929: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.929: Failed to load module "window-decorations-gtk-module"

** (xfdesktop:234932): WARNING **: 10:44:11.962: Failed to register the newly set background with AccountsService '/usr/share/backgrounds/xfce/xfce-shapes.svg': GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: No such interface ā€œorg.freedesktop.DisplayManager.AccountsServiceā€
Gtk-Message: 10:44:11.991: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.991: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"

(wrapper-2.0:234943): libnotify-WARNING **: 10:44:12.158: Failed to connect to proxy
[websockify]: started successfully (proxying 20989 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...

(tracker-miner-fs-3:234945): Tracker-WARNING **: 10:44:28.485: tracker_resource_set_relation: NULL is not a valid value.