I’m working on getting OnDemand fully set up on a new cluster. Right now, we have everything installed as per usual on the login node and I’m finalizing the TurboVNC integration. However, we’re getting an issue when launching XFCE4 (I haven’t tried the other desktops yet, though I’ll probably try MATE soon).
Everything in the usual logs looks fine, except the output.log file:
Setting VNC password...
Starting VNC server...
Desktop 'TurboVNC: hcocs001:1 (jpg00017)' started on display hcocs001:1
Log file is vnc.log
Successfully started VNC server on hcocs001:5901...
Script starting...
Starting websocket server...
Launching desktop 'xfce'...
[websockify]: pid: 123972 (proxying 46779 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
grep: ./websockify.log: No such file or directory
_IceTransmkdir: Owner of /tmp/.ICE-unix should be set to root
/usr/bin/iceauth: creating new authority file /run/user/560378/ICEauthority
(xfwm4:124003): xfwm4-WARNING **: 20:07:41.159: Unsupported GL renderer (llvmpipe (LLVM 20.1.8, 256 bits)).
[websockify]: started successfully (proxying 46779 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...
** (wrapper-2.0:124039): WARNING **: 20:07:42.493: No outputs have backlight property
(wrapper-2.0:124039): Gtk-CRITICAL **: 20:07:42.537: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:124039): Gtk-CRITICAL **: 20:07:42.562: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:124039): Gtk-CRITICAL **: 20:07:42.562: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:124036): libnotify-WARNING **: 20:07:42.568: Failed to connect to proxy
(wrapper-2.0:124039): Gtk-CRITICAL **: 20:07:42.618: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:124036): pulseaudio-plugin-WARNING **: 20:07:49.247: Disconnected from the PulseAudio server. Attempting to reconnect in 5 seconds...
(xfdesktop:124029): GVFS-RemoteVolumeMonitor-WARNING **: 20:07:49.251: Owner of volume monitor org.gtk.vfs.UDisks2VolumeMonitor disconnected from the bus; removing drives/volumes/mounts
(tracker-miner-fs-3:124531): GVFS-RemoteVolumeMonitor-WARNING **: 20:07:49.252: Owner of volume monitor org.gtk.vfs.UDisks2VolumeMonitor disconnected from the bus; removing drives/volumes/mounts
xfsettingsd: Another instance took over. Leaving...
(Thunar:124024): thunar-WARNING **: 20:07:49.268: Name 'org.xfce.FileManager' lost on the message dbus.
(Thunar:124024): thunar-WARNING **: 20:07:49.268: Name 'org.freedesktop.FileManager1' lost on the message dbus.
(xfce4-panel:124020): xfce4-panel-CRITICAL **: 20:07:49.269: Name org.xfce.Panel lost on the message dbus, exiting.
/usr/bin/iceauth: error in locking authority file /run/user/560378/ICEauthority
(xfce4-session:123987): xfce4-session-WARNING **: 20:07:49.319: Failed to execute "/usr/bin/iceauth source /tmp/.xfsm-ICE-X1KTS3"
Terminated
Desktop 'xfce' ended with 143 status...
Cleaning up...
Killing Xvnc process ID 123939
OK
@micket iceauth is on both the host machine for OOD and the compute node, its weird that its trying to use /tmp/ instead of /run/user, which is what I see in my other system like you.
Mimic-ing your set-up @jeff.ohrstrom , I cannot get the before_script to run more than just the first line of exporting the modules (which is something we want to do as well). Adding the submit: submit/harpers.yml.erb to the bc_desktop app and then adding said file with the following:
batch_connect:
before_script: |
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module
# MATE acts strange in pitzer-exp and doesn't like /var/run/$(id -u)
export XDG_RUNTIME_DIR="$TMPDIR/xdg_runtime"
# reset SLURM_EXPORT_ENV so that things like srun & sbatch work out of the box
export SLURM_EXPORT_ENV=ALL
my before.sh only shows:
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module
With submit/harpers.yml.erb simply being (with the plan to add the extra slurm stuff before the --- later):
---
batch_connect:
before_script: |
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module
# MATE acts strange in pitzer-exp and doesn't like /var/run/$(id -u)
export XDG_RUNTIME_DIR="$TMPDIR/xdg_runtime"
# reset SLURM_EXPORT_ENV so that things like srun & sbatch work out of the box
export SLURM_EXPORT_ENV=ALL
YML looks good, I wonder if it’s a more simple issue like the file/directory isn’t readable or the relative location of submit/ is not exactly right? (i.e., can’t find the file)
Yeah it’s pretty puzzling. I am also running the /opt/ood/nginx_stage/sbin/update_nginx_stage and /opt/ood/ood-portal-generator/sbin/update_ood_portal just to be on the safe side.
Nope; tried that. Also tried a new Incognito session. Changed the app’s name to “Harpers Desktop” to confirm its running a new app. Updated nginx, restarted httpd, and removed my AD user’s ~/ondemand/ to really make sure. Still not seeing the before script getting updated. I’m also not seeing really anything in the /var/log/ondemand-nginx/jpg00017/error.log to indicate anything is failing (like permission issues).
So I went into /var/www/ood/apps/sys/bc_desktop/template/before.sh.erb and edited it to be:
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module
# MATE acts strange in pitzer-exp and doesn't like /var/run/$(id -u)
export XDG_RUNTIME_DIR="$TMPDIR/xdg_runtime"
Doing so correctly shows the before.sh matching this (so its definitely not setting the override correctly).
However, I am still getting the error. I jumped on the node and noticed that /tmp/.ICE-unix was owned by jpg00017:its-rc-harpers. Changed that to root:root. Now we are getting:
/usr/bin/iceauth: error in locking authority file /tmp/xdg_runtime/ICEauthority
I’ll note that on this is what things look like on the node:
I set up /tmp/xdg_runtime/ on a node with the 1777 permissions. Doing so, results in the following output.log
Setting VNC password...
Starting VNC server...
Desktop 'TurboVNC: hcocx001:1 (jpg00017)' started on display hcocx001:1
Log file is vnc.log
Successfully started VNC server on hcocx001:5901...
Script starting...
Starting websocket server...
Launching desktop 'xfce'...
[websockify]: pid: 163605 (proxying 51146 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
grep: ./websockify.log: No such file or directory
/usr/bin/iceauth: unable to link authority file /tmp/xdg_runtime/ICEauthority, use /tmp/xdg_runtime/ICEauthority-n
(xfwm4:163636): xfwm4-WARNING **: 17:25:29.591: Unsupported GL renderer (llvmpipe (LLVM 20.1.8, 256 bits)).
(xfce4-session:163620): xfce4-session-WARNING **: 17:25:29.595: ICE connection 0x55e8c06a4f20 rejected
(xfwm4:163636): xfwm4-WARNING **: 17:25:29.595: Failed to connect to session manager: Failed to connect to the session manager: Authentication Rejected, reason : None of the authentication protocols specified are supported and host-based authentication failed
[websockify]: started successfully (proxying 51146 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...
Failed to connect to session manager: Failed to connect to the session manager: Authentication Rejected, reason : None of the authentication protocols specified are supported and host-based authentication failed
(xfce4-session:163620): xfce4-session-WARNING **: 17:25:37.551: ICE connection 0x55e8c06ce800 rejected
xfsettingsd: Another instance took over. Leaving...
Terminated
Desktop 'xfce' ended with 143 status...
Cleaning up...
Killing Xvnc process ID 163572
So this is quite confusing. Is there an authentication app that needs to be installed? For reference, this is stock Rocky 9.7 5.14.0-611.55.1.el9_7.x86_64 on the compute node with only the following installs done on the warewulf image:
I don’t think you need additional packages, but I would maybe try XDG_RUNTIME_DIR="$(mktemp -d)/xdg_runtime" instead just to avoid more issues with that directory creation.
A quick google search of the error seems to suggest it’s trying to do things as a different user, root maybe?
I just tried on my systems and this file should be owned by you, not root:root. That chown could be causing more issues.
Every system I have checked has /tmp/.ICE-unix/ owned by root with 1777 permissions, then the files inside are owned by the given user.
While the before.sh never changes, I do see that in the job_script_content.sh that gets created, the contents of submit/harpers.yml.erb’s before_script does get added into that .sh:
I corrected a mistake with the XDG_RUNTIME_DIR="$(mktemp -d)/xdg_runtime" not having export (that’s what I get for bouncing around on projects) and ensured it is created by the user with mkdir $XDG_RUNTIME_DIR after the the export. Now we get everything set up correctly:
However, we are still failing to start the VNC, seemingly because the proxy is failing to connect after a bunch of gvfs errors.
Setting VNC password...
Starting VNC server...
Desktop 'TurboVNC: hcocx001:1 (jpg00017)' started on display hcocx001:1
Log file is vnc.log
Successfully started VNC server on hcocx001:5901...
Script starting...
Starting websocket server...
Launching desktop 'xfce'...
[websockify]: pid: 166632 (proxying 59909 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
grep: ./websockify.log: No such file or directory
/usr/bin/iceauth: creating new authority file /tmp/tmp.zbBkJ1IxI4/xdg_runtime/ICEauthority
(xfwm4:166663): xfwm4-WARNING **: 18:05:41.119: Unsupported GL renderer (llvmpipe (LLVM 20.1.8, 256 bits)).
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.709: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.710: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.710: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.711: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.711: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
(xfdesktop:166688): GVFS-WARNING **: 18:05:41.711: The peer-to-peer connection failed: Error when getting information for file “/tmp/tmp.zbBkJ1IxI4/xdg_runtime/gvfsd”: No such file or directory. Falling back to the session bus. Your application is probably missing --filesystem=xdg-run/gvfsd privileges.
[websockify]: started successfully (proxying 59909 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...
** (wrapper-2.0:166698): WARNING **: 18:05:42.463: No outputs have backlight property
(wrapper-2.0:166698): Gtk-CRITICAL **: 18:05:42.513: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:166698): Gtk-CRITICAL **: 18:05:42.534: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:166698): Gtk-CRITICAL **: 18:05:42.534: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(wrapper-2.0:166695): libnotify-WARNING **: 18:05:42.541: Failed to connect to proxy
(wrapper-2.0:166698): Gtk-CRITICAL **: 18:05:42.588: gtk_icon_theme_has_icon: assertion 'icon_name != NULL' failed
(xfdesktop:166688): GVFS-RemoteVolumeMonitor-WARNING **: 18:05:49.193: Owner of volume monitor org.gtk.vfs.UDisks2VolumeMonitor disconnected from the bus; removing drives/volumes/mounts
(tracker-miner-fs-3:167221): GVFS-RemoteVolumeMonitor-WARNING **: 18:05:49.194: Owner of volume monitor org.gtk.vfs.UDisks2VolumeMonitor disconnected from the bus; removing drives/volumes/mounts
xfsettingsd: Another instance took over. Leaving...
(Thunar:166683): thunar-WARNING **: 18:05:49.197: Name 'org.xfce.FileManager' lost on the message dbus.
(Thunar:166683): thunar-WARNING **: 18:05:49.197: Name 'org.freedesktop.FileManager1' lost on the message dbus.
(xfce4-panel:166679): xfce4-panel-CRITICAL **: 18:05:49.197: Name org.xfce.Panel lost on the message dbus, exiting.
Terminated
Desktop 'xfce' ended with 143 status...
Cleaning up...
Killing Xvnc process ID 166597
OK
This is also an issue websockify doesn’t seem to start. Though it does not seem to be the main issue.
That said - I’m not sure what the GVFS issue could be or those messages about “lost on the message dbus.” Seems like DBUS isn’t working quite right for you X session, but I could not say why.
Are there any logs in journalctl or other system logs that may have some information?
I would strongly suggest to simplifying and just try starting vncserver directly inside and interactive job;
# Isolate to it's own XDG environment
export XDG_RUNTIME_DIR=$TMPDIR/xdg-runtime-test
export XDG_CACHE_DIR=$TMPDIR/xdg-cache-test
mkdir -p "$XDG_CACHE_DIR" "$XDG_RUNTIME_DIR"
# probably want to isolate this into it's own DBUS environment as well:
unset DBUS_SESSION_BUS_ADDRESS
unset DBUS_SYSTEM_BUS_ADDRESS
eval $(dbus-launch --sh-syntax)
# Should launch without to many errors in vnc log:
vncserver -xstartup xfce4-session
I suspect that this might be a very new Xfce4, and they start relying more on DBus. I know when I built Xfce4 from source, this started to become a requirement to isolate the dbus environments when i tested stuff. Probably always a good idea to do this.
This worked right out of the gate. Here is the final state:
before_script: |
# Isolate the XDG Runtime Directories
export TMPDIR = "$(mktemp -d)"
export XDG_RUNTIME_DIR=$TMPDIR/xdg_runtime
export XDG_CACHE_DIR=$TMPDIR/xdg_cache
mkdir -p $XDG_RUNTIME_DIR $XDG_CACHE_DIR
# Unset DBUS Hooks to allow TurboVNC to Work with XFCE4
unset DBUS_SESSION_BUS_ADDRESS
unset DBUS_SYSTEM_BUS_ADDRESS
eval $(dbus-launch --sh-syntax)
# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module
# reset SLURM_EXPORT_ENV so that things like srun & sbatch work out of the box
export SLURM_EXPORT_ENV=ALL
For @jeff.ohrstrom , nothing illuminating was in the journalctl or other system logs. This system is running RHEL 9.7 and kernel 5.14.0-611.55.1.el9_7.x86_64 with xfce4 4.18.3-1.el9.
The one issue I am running into is that the screensaver comes up and requires a password, which it doesn’t accept from the user. If I remember right, a solution for this is to just uninstall xfce4-screensaver.
Just as a side-note; what’s happening here is we create a new dbus environment for this new isolated xfce. It isn’t turbovnc specific, it’s an desktop-environment thing (Xfce and many others that rely on dbus). If we ran a simpler desktop environment, say, some basic OpenBox environment, this wouldn’t be needed.
No matter if we run turbovnc (Xvnc), Xvfb, Xorg, or even wayland, you want this isolated dbus thing environment (from dbus-launch) to launch multiple Xfce’s.
try restarting your webserver in the help menu. The app has likely been cached without the 