MATLAB on OOD 3.1: App has not been initialized or does not exist

From Interactive Apps like this:

It doesnā€™t seem to like the form.yml you have and is trying to find another.

I restarted Apache, ran update_ood_portal, and chose ā€œRestart Web Serverā€ so I donā€™t see those errors anymore but the Desktop is failing to start still:

Launching desktop 'xfce'...
xfce4-session: Cannot open display: .
Type 'xfce4-session --help' for usage.
Desktop 'xfce' ended with 1 status...
Cleaning up...

Hereā€™s job_script_content.sh

cd /home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d

# Export useful connection variables
export host
export port

# Generate a connection yaml file with given parameters
create_yml () {
  echo "Generating connection YAML file..."
  (
    umask 077
    echo -e "host: $host\nport: $port\npassword: $password" > "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/connection.yml"
  )
}

# Cleanliness is next to Godliness
clean_up () {
  echo "Cleaning up..."
  [[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/clean.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/clean.sh"
  [[ ${SCRIPT_PID} ]] && pkill -P ${SCRIPT_PID} || :
  pkill -P $$
  exit ${1:-0}
}

# Source in all the helper functions
source_helpers () {
  # Generate random integer in range [$1..$2]
  random_number () {
    shuf -i ${1}-${2} -n 1
  }
  export -f random_number

  port_used_python() {
    python -c "import socket; socket.socket().connect(('$1',$2))" >/dev/null 2>&1
  }

  port_used_python3() {
    python3 -c "import socket; socket.socket().connect(('$1',$2))" >/dev/null 2>&1
  }

  port_used_nc(){
    nc -w 2 "$1" "$2" < /dev/null > /dev/null 2>&1
  }

  port_used_lsof(){
    lsof -i :"$2" >/dev/null 2>&1
  }

  port_used_bash(){
    local bash_supported=$(strings /bin/bash 2>/dev/null | grep tcp)
    if [ "$bash_supported" == "/dev/tcp/*/*" ]; then
      (: < /dev/tcp/$1/$2) >/dev/null 2>&1
    else
      return 127
    fi
  }

  # Check if port $1 is in use
  port_used () {
    local port="${1#*:}"
    local host=$((expr "${1}" : '\(.*\):' || echo "localhost") | awk 'END{print $NF}')
    local port_strategies=(port_used_nc port_used_lsof port_used_bash port_used_python port_used_python3)

    for strategy in ${port_strategies[@]};
    do
      $strategy $host $port
      status=$?
      if [[ "$status" == "0" ]] || [[ "$status" == "1" ]]; then
        return $status
      fi
    done

    return 127
  }
  export -f port_used

  # Find available port in range [$2..$3] for host $1
  # Default host: localhost
  # Default port range: [2000..65535]
  # returns error code (0: success, 1: failed)
  # On success, the chosen port is echoed on stdout.
  find_port () {
    local host="${1:-localhost}"
    local min_port=${2:-2000}
    local max_port=${3:-65535}
    local port_range=($(shuf -i ${min_port}-${max_port}))
    local retries=1 # number of retries over the port range if first attempt fails
    for ((attempt=0; attempt<=$retries; attempt++)); do
      for port in "${port_range[@]}"; do
        if port_used "${host}:${port}"; then
continue
        fi
        echo "${port}"
        return 0 # success
      done
    done

    echo "error: failed to find available port in range ${min_port}..${max_port}" >&2
    return 1 # failure
  }
  export -f find_port

  # Wait $2 seconds until port $1 is in use
  # Default: wait 30 seconds
  wait_until_port_used () {
    local port="${1}"
    local time="${2:-30}"
    for ((i=1; i<=time*2; i++)); do
      port_used "${port}"
      port_status=$?
      if [ "$port_status" == "0" ]; then
        return 0
      elif [ "$port_status" == "127" ]; then
         echo "commands to find port were either not found or inaccessible."
         echo "command options are lsof, nc, bash's /dev/tcp, or python (or python3) with socket lib."
         return 127
      fi
      sleep 0.5
    done
    return 1
  }
  export -f wait_until_port_used

  # Generate random alphanumeric password with $1 (default: 8) characters
  create_passwd () {
    tr -cd 'a-zA-Z0-9' < /dev/urandom 2> /dev/null | head -c${1:-8}
  }
  export -f create_passwd
}
export -f source_helpers

source_helpers

# Set host of current machine
host=$(hostname)

[[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/before.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/before.sh"

echo "Script starting..."
"/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/script.sh" &
SCRIPT_PID=$!

[[ -e "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/after.sh" ]] && source "/home/myuser/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/bfd64c96-6a88-4066-a6e9-947382baf13d/after.sh"

# Create the connection yaml file
create_yml

# Wait for script process to finish
wait ${SCRIPT_PID} || clean_up 1

# Exit cleanly
clean_up

OOD thinks the application is a http server, not a vnc application.

You cloned the repo from us - but did you keep this portion specifying itā€™s a vnc application?

Thanks @Micket Iā€™ve tried this on RHEL 7 and ran into some difficulties. I was hoping to get the VNC version working and Iā€™m getting close.

Which Jupyter app did you find easy? There are a few posted from several universities.

For the Matlab app yes:

batch_connect:
  template: vnc
script:
  native:
  <%- slurm_args.each do |arg| %>
    - "<%= arg %>"
  <%- end %>

The error I get from MATLAB is:

(xfsettingsd:153295): xfsettingsd-WARNING **: 10:47:59.621: Failed to get the _NET_NUMBER_OF_DESKTOPS property.
Killing Xvnc process ID 153271

OK so from Interactive Apps I still get this error:

This app does not supply a sub app form file under the directory '/var/www/ood/apps/sys/bc_osc_matlab/local'

As for using the MATLAB icon on the left side on the node here are the errors from /var/log/messages:

Oct 16 12:06:50 ax08 systemd[1]: realmd.service: Deactivated successfully.
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=1822857372 pid=153940 comm="xfce4-panel --sm-client-disable ")
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.a11y.Bus'
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.gtk.vfs.Daemon' requested by ':1.6' (uid=1822857372 pid=153937 comm="xfwm4 --compositor=off --sm-client-disable ")
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Activating service name='org.xfce.Xfconf' requested by ':1.5' (uid=1822857372 pid=153939 comm="xfsettingsd --sm-client-disable ")
Oct 16 12:06:53 ax08 org.a11y.Bus[153954]: dbus-daemon[153954]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=1822857372 pid=153940 comm="xfce4-panel --sm-client-disable ")
Oct 16 12:06:53 ax08 org.a11y.Bus[153954]: dbus-daemon[153954]: Successfully activated service 'org.a11y.atspi.Registry'
Oct 16 12:06:53 ax08 org.a11y.Bus[153965]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 16 12:06:53 ax08 dbus-daemon[153935]: [session uid=1822857372 pid=153933] Successfully activated service 'org.xfce.Xfconf'
Oct 16 12:06:53 ax08 org.a11y.Bus[153965]: X connection to :1 broken (explicit kill or server shutdown).
Oct 16 12:07:20 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Wow what would cause this from the MATLAB app?

Oct 16 12:28:12 ax08 kernel: xfsettingsd[154358]: segfault at 9 ip 0000150de262f53f sp 00007ffd95a8ce70 error 4 in libGLX_mesa.so.0.0.0[150de2618000+3f000] likely on CPU 0 (core 0, socket 0)
Oct 16 12:28:12 ax08 kernel: Code: c7 9a fe ff 48 8d 15 98 92 02 00 be 07 00 00 00 48 89 c7 48 89 c3 e8 10 9e fe ff 31 d2 48 89 df 89 c6 e8 b4 9a fe ff 48 89 c7 <0f> b6 40 09 89 05 3f 0b 04 00 0f b6 47 08 88 44 24 1f e8 ca 9e fe
Oct 16 12:28:12 ax08 kernel: xfwm4[154355]: segfault at 9 ip 0000146c49fa853f sp 00007ffc3717d700 error 4 in libGLX_mesa.so.0.0.0[146c49f91000+3f000] likely on CPU 24 (core 0, socket 0)
Oct 16 12:28:12 ax08 kernel: Code: c7 9a fe ff 48 8d 15 98 92 02 00 be 07 00 00 00 48 89 c7 48 89 c3 e8 10 9e fe ff 31 d2 48 89 df 89 c6 e8 b4 9a fe ff 48 89 c7 <0f> b6 40 09 89 05 3f 0b 04 00 0f b6 47 08 88 44 24 1f e8 ca 9e fe
Oct 16 12:28:12 ax08 systemd[1]: Started Process Core Dump (PID 154364/UID 0).
Oct 16 12:28:12 ax08 kernel: enp96s0f1 speed is unknown, defaulting to 1000
Oct 16 12:28:12 ax08 systemd[1]: Started Process Core Dump (PID 154365/UID 0).
Oct 16 12:28:12 ax08 kernel: enp96s0f1 speed is unknown, defaulting to 1000
Oct 16 12:28:12 ax08 systemd-coredump[154366]: Process 154358 (xfsettingsd) of user 1822857372 dumped core.#012#012Stack trace of thread 154358:#012#0  0x0000150de262f53f driswCreateScreenDriver (libGLX_mesa.so.0 + 0x3053f)#012#1  0x0000150de2635068 __glXInitialize (libGLX_mesa.so.0 + 0x36068)#012#2  0x0000150de2631db5 glXQueryServerString (libGLX_mesa.so.0 + 0x32db5)#012#3  0x0000150de56a444f epoxy_glx_version (libepoxy.so.0 + 0xc544f)#012#4  0x0000150de6d35730 gdk_x11_screen_init_gl (libgdk-3.so.0 + 0x92730)#012#5  0x0000150de6d3a1c5 _gdk_x11_screen_new (libgdk-3.so.0 + 0x971c5)#012#6  0x0000150de6d26dcd _gdk_x11_display_open (libgdk-3.so.0 + 0x83dcd)#012#7  0x0000150de6cdaadb gdk_display_manager_open_display (libgdk-3.so.0 + 0x37adb)#012#8  0x0000150de660a8fb gtk_init_check (libgtk-3.so.0 + 0x20a8fb)#012#9  0x0000556207c0d930 main (xfsettingsd + 0x8930)#012#10 0x0000150de5c29590 __libc_start_call_main (libc.so.6 + 0x29590)#012#11 0x0000150de5c29640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)#012#12 0x0000556207c0df95 _start (xfsettingsd + 0x8f95)#012ELF object binary architecture: AMD x86-64
Oct 16 12:28:12 ax08 systemd[1]: systemd-coredump@2-154364-0.service: Deactivated successfully.
Oct 16 12:28:12 ax08 systemd-coredump[154368]: Process 154355 (xfwm4) of user 1822857372 dumped core.#012#012Stack trace of thread 154355:
#012#0  0x0000146c49fa853f driswCreateScreenDriver (libGLX_mesa.so.0 + 0x3053f)
#012#1  0x0000146c49fae068 __glXInitialize (libGLX_mesa.so.0 + 0x36068)
#012#2  0x0000146c49faadb5 glXQueryServerString (libGLX_mesa.so.0 + 0x32db5)?
#012#3  0x0000146c4e72e44f epoxy_glx_version (libepoxy.so.0 + 0xc544f)
#012#4  0x0000146c4e5a2730 gdk_x11_screen_init_gl (libgdk-3.so.0 + 0x92730)
#012#5  0x0000146c4e5a71c5 _gdk_x11_screen_new (libgdk-3.so.0 + 0x971c5)
#012#6  0x0000146c4e593dcd _gdk_x11_display_open (libgdk-3.so.0 + 0x83dcd)
#012#7  0x0000146c4e547b0c gdk_display_manager_open_display (libgdk-3.so.0 + 0x37b0c)
#012#8  0x0000146c4de0a8fb gtk_init_check (libgtk-3.so.0 + 0x20a8fb)
#012#9  0x0000146c4de0a92d gtk_init (libgtk-3.so.0 + 0x20a92d)
#012#10 0x0000563fa779bc39 main (xfwm4 + 0x12c39)
#012#11 0x0000146c4d429590 __libc_start_call_main (libc.so.6 + 0x29590)
#012#12 0x0000146c4d429640 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29640)
#012#13 0x0000563fa779e4c5 _start (xfwm4 + 0x154c5)
#012ELF object binary architecture: AMD x86-64

OK this error only happens after the initial login then goes away. We are using CAS and itā€™s similar to this 2020 RStudio thread. I tried adding CASScope to out conf file but get:

Oct 16 14:30:41 ood.ouruni.edu httpd[1402821]: AH00526: Syntax error on line 17 of /etc/httpd/conf.d/cas.conf:
Oct 16 14:30:41 ood.ouruni.edu httpd[1402821]: CASScope not allowed here

Iā€™m not sure what would cause that core dump. Google searches donā€™t provide a lot of hits, but if I had to guess it may be a missing library or driver. Like you may be missing a video card driver?

I also donā€™t know much about CASScope, but the error indicates itā€™s not in the right place (itā€™s global?). Like it should be held within all the Location directives, not applied globally.

Which is to say maybe itā€™s better placed in the auth section of your ood_portal.yml.

Where in the file should it go and what is the value? CAS?

The auth section of your ood_portal.yml. Using / is super liberal, but itā€™ll account for both /pun stuff and /node and /rnode URIs.

auth:
  - CASScope /
# - the other CAS stuff you have

I think that ā€˜app does not supply a sub appā€™ error is now resolved. Still seeing these error on Desktop

[websockify]: pid: 158072 (proxying 14087 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
[websockify]: failed to launch!
Cleaning up...
/usr/bin/iceauth:  creating new authority file /run/user/1822857372/ICEauthority
Killing Xvnc process ID 158025
Desktop 'xfce' ended with 1 status...

/var/spool/slurmd/job00117/slurm_script: line 206: /opt/websockify/run: No such file or directory

And on the node:

X connection to :1 broken (explicit kill or server shutdown).

I missed a crucial step to configure Apache to add to /etc/ood/config/ood_portal.yml:

host_regex: '[\w.-]+\.path\.to\.ouruni\.edu'
node_uri: '/node'
rnode_uri: '/rnode'

I also had to put:

auth:
  - 'AuthType CAS'
  - 'CASScope /'

And I had incorrect alignment of:

   job:
   batch_connect:

The latter was indented 2 additional spaces :frowning:

However Iā€™m now getting the No VNC error. The only error in the logs I see is:
[authz_core:error] [pid 1478819:tid 1478974] [client x.x.x.x:57112] AH01630: client denied by server configuration: /etc/httpd/htdocs

Iā€™m not browsing any of these directories and that one does not exist. The other referer log from Apache has this:

[Thu Oct 17 14:52:32.401054 2024] [lua:info] [pid 1478819:tid 1478927] [client 128.59.216.151:56944] res_content_length="0" req_accept_encoding="gzip, deflate, br, zstd" res_content_encoding="" req_method="GET" res_content_location="" res_content_type="" allowed_hosts="ood.ouruni.edu" req_status="302" log_time="2024-10-17T18:52:32.400882.0Z" req_port="443" req_filename="/etc/httpd/htdocs" req_cache_control="" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_accept_language="en-us,en;q=0.9" remote_user="myuser" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_handler="" time_proxy="0" req_is_websocket="false" log_hook="ood" req_origin="" req_accept_charset="" time_user_map="0.004" res_location="/pun/sys/dashboard/" req_protocol="HTTP/1.1" local_user="rk3199" req_content_type="" res_content_language="" req_user_ip="x.x.x.x" req_uri="/nginx/stop" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_server_name="ood.ouruni.edu" req_is_https="true" res_content_disp="" req_hostname="ood.ouruni.edu", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions

Does this sound familiar to anyone?

Thanks @jeff.ohrstrom for your help today.

Edit: the log actually starts with authz_core:error could this be CAS-related?

I tested KDE the errors are differnt, from /var/log/messages, any clues here?

Oct 17 22:51:00 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_search_xapian: Could not obtain lock for Xapian Database. This is bad
Oct 17 22:51:00 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_indexer_agent: "DatabaseLockError" "DatabaseLockError: Unable to get write lock on /home/myuser/.local/share/akonadi/search_db/notes/: already locked"
Oct 17 22:51:05 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Activating service name='org.freedesktop.Tracker3.Miner.Extract' requested by ':1.13' (uid=1822857372 pid=195841 comm="/usr/libexec/tracker-miner-fs-3 ")
Oct 17 22:51:05 ax08 tracker-extract[196982]: tracker_module_manager_load_modules: assertion 'initialized == TRUE' failed
Oct 17 22:51:05 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Successfully activated service 'org.freedesktop.Tracker3.Miner.Extract'
Oct 17 22:51:10 ax08 drkonqi[196298]: org.kde.drkonqi.bugzilla: ProtocolException: "bugs.kde.org: Socket operation timed out" 149 "bugs.kde.org: Socket operation timed out" "bugs.kde.org: Socket operation timed out"
Oct 17 22:51:10 ax08 drkonqi[196298]: org.kde.drkonqi: "bugs.kde.org: Socket operation timed out"
Oct 17 22:51:10 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_search_xapian: Could not obtain lock for Xapian Database. This is bad
Oct 17 22:51:10 ax08 akonadi_indexing_agent[196389]: org.kde.pim.akonadi_indexer_agent: "DatabaseLockError" "DatabaseLockError: Unable to get write lock on /home/myuser/.local/share/akonadi/search_db/collections/: already locked"
Oct 17 22:51:10 ax08 akonadiserver[196282]: org.kde.pim.akonadiserver: Subscriber Akonadi::Server::NotificationSubscriber(0x14b82002b530) identified as "AgentBaseChangeRecorder - 94548651063584"
Oct 17 22:51:14 ax08 xdg-desktop-por[195984]: Failed to create secret proxy: Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached
Oct 17 22:51:14 ax08 xdg-desktop-por[195984]: No skeleton to export
Oct 17 22:51:14 ax08 dbus-daemon[195712]: [session uid=1822857372 pid=195710] Successfully activated service 'org.freedesktop.portal.Desktop'
Oct 17 22:51:15 ax08 goa-daemon[84858]: secret_password_clear_sync() failed: Error calling StartServiceByName for org.freedesktop.secrets: Timeout was reached
Oct 17 22:51:15 ax08 goa-daemon[84858]: Unable to clean-up stale keyring entries: Failed to delete credentials from the keyring
Oct 17 22:51:16 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Here are the compute node system logs when launching a XFCE desktop. Is this GDM keyring an issue? I found a person who mentioned disabling it in a pam module, at least the autostart, which I did. How about the Tracker logs? Should that just be disabled in the XFCE configs?

Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.a11y.Bus' requested by ':1.0' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.a11y.Bus'
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.xfce.Xfconf' requested by ':1.2' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.xfce.Xfconf'
Oct 18 09:55:24 ax08 org.a11y.Bus[230035]: dbus-daemon[230035]: Activating service name='org.a11y.atspi.Registry' requested by ':1.0' (uid=1822857372 pid=230022 comm="xfce4-session ")
Oct 18 09:55:24 ax08 org.a11y.Bus[230035]: dbus-daemon[230035]: Successfully activated service 'org.a11y.atspi.Registry'
Oct 18 09:55:24 ax08 org.a11y.Bus[230046]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.Daemon' requested by ':1.5' (uid=1822857372 pid=230050 comm="xfwm4 ")
Oct 18 09:55:24 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.Daemon'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.thumbnails.Thumbnailer1' requested by ':1.10' (uid=1822857372 pid=230143 comm="xfdesktop ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.UDisks2VolumeMonitor' requested by ':1.11' (uid=1822857372 pid=230151 comm="/usr/lib64/tumbler-1/tumblerd ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.UDisks2VolumeMonitor'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.Sources5' requested by ':1.15' (uid=1822857372 pid=230159 comm="/usr/libexec/evolution-data-server/evolution-alarm")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.thumbnails.Thumbnailer1'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gtk.vfs.Metadata' requested by ':1.10' (uid=1822857372 pid=230143 comm="xfdesktop ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.OnlineAccounts' requested by ':1.19' (uid=1822857372 pid=230253 comm="/usr/libexec/evolution-source-registry ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gtk.vfs.Metadata'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.Sources5'
Oct 18 09:55:25 ax08 goa-daemon[230307]: goa-daemon version 3.40.0 starting
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.Identity' requested by ':1.23' (uid=1822857372 pid=230307 comm="/usr/libexec/goa-daemon ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.secrets' requested by ':1.23' (uid=1822857372 pid=230307 comm="/usr/libexec/goa-daemon ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.Identity'
Oct 18 09:55:25 ax08 gnome-keyring-daemon[230328]: couldn't connect to control socket at: /tmp/xdg_runtime/keyring/control: Connection refused
Oct 18 09:55:25 ax08 gnome-keyring-d[230328]: couldn't connect to control socket at: /tmp/xdg_runtime/keyring/control: Connection refused
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.secrets'
Oct 18 09:55:25 ax08 gnome-keyring-daemon[230328]: daemon/gkd-main.c:931 Couldn't get object path: GDBus.Error:org.freedesktop.login1.NoSuchSession: No session 'c32' known
Oct 18 09:55:25 ax08 gnome-keyring-d[230328]: daemon/gkd-main.c:931 Couldn't get object path: GDBus.Error:org.freedesktop.login1.NoSuchSession: No session 'c32' known
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.OnlineAccounts'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.Calendar8' requested by ':1.15' (uid=1822857372 pid=230159 comm="/usr/libexec/evolution-data-server/evolution-alarm")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.Calendar8'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='ca.desrt.dconf' requested by ':1.27' (uid=1822857372 pid=230355 comm="/usr/libexec/evolution-calendar-factory ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.gnome.evolution.dataserver.AddressBook10' requested by ':1.27' (uid=1822857372 pid=230355 comm="/usr/libexec/evolution-calendar-factory ")
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'ca.desrt.dconf'
Oct 18 09:55:25 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.gnome.evolution.dataserver.AddressBook10'
Oct 18 09:55:25 ax08 NetworkManager[1589]: <info>  [1729259725.5020] agent-manager: agent[0b192c68d9ec319d,:1.3081/org.freedesktop.nm-applet/1822857372]: agent registered


Oct 18 09:55:40 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Activating service name='org.freedesktop.Tracker3.Miner.Extract' requested by ':1.12' (uid=1822857372 pid=230149 comm="/usr/libexec/tracker-miner-fs-3 ")
Oct 18 09:55:40 ax08 tracker-extract[230518]: tracker_module_manager_load_modules: assertion 'initialized == TRUE' failed
Oct 18 09:55:40 ax08 dbus-daemon[230018]: [session uid=1822857372 pid=230016] Successfully activated service 'org.freedesktop.Tracker3.Miner.Extract'
Oct 18 09:55:52 ax08 systemd[1]: systemd-hostnamed.service: Deactivated successfully.

Notice that on the node there is an active xfce session:

ps -ef|grep -i xfce
myuser    230022  229994  0 09:55 ?        00:00:00 xfce4-session
myuser    230131  230022  0 09:55 ?        00:00:01 xfce4-panel
myuser    230173  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libsystray.so 6 12582924 systray Status Tray Plugin Provides status notifier items (application indicators) and legacy systray items
myuser    230174  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libpulseaudio-plugin.so 8 12582925 pulseaudio PulseAudio Plugin Adjust the audio volume of the PulseAudio sound system
myuser    230326  230131  0 09:55 ?        00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libactions.so 14 12582926 actions Action Buttons Log out, lock or other system actions

I donā€™t believe thereā€™s anything wrong on the compute side. When we saw your console logs the websocket connection only lasted 5 ms or so ā€” thatā€™s not long enough to reach XFCE. Such a short duration suggests to me that apache immediately closed the connection and never even tried to forward it to websockify & XFCE on the compute node.

If youā€™re still getting this error in apache logs - thatā€™s the issue, not XFCE.

AH01630: client denied by server configuration

Curiously, no I no longer see those logs after enabling debug of trace3 level. The last instance was:
[Thu Oct 17 16:41:34.222066 2024] [authz_core:error] [pid 1478819:tid 1478990] [client x.x.x.x:58892] AH01630: client denied by server configuration: /etc/httpd/htdocs

Is there any logging level I can add/increase to the authentication side?

If you have trace3 enabled then it should be saying something about why youā€™re being disconnected. Look for the websockfiy URLs.

20241018-apache-logs.txt (1.5 MB)
Attached are some trace3 logs from today, 10/18/24. A snip showing websockify, anything stick out here?
[Fri Oct 18 08:59:41.463864 2024] [lua:info] [pid 1511513:tid 1511631] [client x.x.x.x:64979] req_hostname="ood.ouruni.edu" req_origin="" req_is_websocket="false" time_proxy="1.637" req_protocol="HTTP/1.1" time_user_map="0.004" req_port="443" req_content_type="" remote_user="myuser" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_status="200" req_handler="proxy-server" allowed_hosts="ood.ouruni.edu" req_is_https="true" req_method="GET" log_hook="ood" req_server_name="ood.ouruni.edu" res_content_disp="" res_content_encoding="" log_time="2024-10-18T12:59:41.463639.0Z" req_filename="proxy:http://localhost/pun/sys/dashboard/noVNC-1.3.0/vnc.html?autoconnect=true&path=rnode%2Fax.ouruni.edu%2F37917%2Fwebsockify&resize=remote&password=xxx&compression=6&quality=2&commit=Launch+Desktop" req_cache_control="" req_uri="/pun/sys/dashboard/noVNC-1.3.0/vnc.html" local_user="myuser" res_location="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" log_id="ZxJbvQhgr3iVfwHaTZQOIQAAABA" req_accept_language="en-us,en;q=0.9" req_accept_encoding="gzip, deflate, br, zstd" res_content_language="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_user_ip="x.x.x.x" req_accept_charset="" res_content_length="15280" res_content_type="text/html; charset=utf-8" res_content_location="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions

edit: from output.log:

Launching desktop 'xfce'...
[websockify]: pid: 234809 (proxying 20989 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
Gtk-Message: 10:44:11.408: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.409: Failed to load module "window-decorations-gtk-module"

(xfwm4:234839): xfwm4-WARNING **: 10:44:11.527: Unsupported GL renderer (llvmpipe (LLVM 17.0.6, 256 bits)).
Gtk-Message: 10:44:11.580: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.580: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.620: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.620: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.671: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.671: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.698: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.699: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.769: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.769: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:11.929: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.929: Failed to load module "window-decorations-gtk-module"

** (xfdesktop:234932): WARNING **: 10:44:11.962: Failed to register the newly set background with AccountsService '/usr/share/backgrounds/xfce/xfce-shapes.svg': GDBus.Error:org.freedesktop.DBus.Error.InvalidArgs: No such interface ā€œorg.freedesktop.DisplayManager.AccountsServiceā€
Gtk-Message: 10:44:11.991: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:11.991: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "colorreload-gtk-module"
Gtk-Message: 10:44:12.138: Failed to load module "window-decorations-gtk-module"

(wrapper-2.0:234943): libnotify-WARNING **: 10:44:12.158: Failed to connect to proxy
[websockify]: started successfully (proxying 20989 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...

(tracker-miner-fs-3:234945): Tracker-WARNING **: 10:44:28.485: tracker_resource_set_relation: NULL is not a valid value.