Stale websockify processes running

[kenny.hanson]

Hey folks,
I’m finding stale /bin/python3 /usr/local/bin/websockify -D localhost:5901 processes running on almost every compute node. It looks like they were started with Remote Desktop.
I’m guessing these were jobs that I was forced to kill while testing new configurations. Anybody else run into this? Thoughts?

Here’s output for ‘ps aux | grep w55c’:

w55c785 1937495 0.0 0.0 306876 5132 ? S Jan06 2:13 /bin/python3 /usr/local/bin/websockify -D 25333 localhost:5901
w55c785 1937546 0.0 0.0 150504 2028 ? Ssl Jan06 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 1937553 0.0 0.0 372140 0 ? Sl Jan06 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 1937558 0.0 0.0 150400 2732 ? Sl Jan06 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 1937566 0.0 0.0 291048 152 ? Sl Jan06 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 1943153 0.0 0.0 306876 4488 ? S Jan06 2:04 /bin/python3 /usr/local/bin/websockify -D 3399 localhost:5901
w55c785 1943219 0.0 0.0 150472 2004 ? Ssl Jan06 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 1943223 0.0 0.0 372140 0 ? Sl Jan06 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 1943228 0.0 0.0 150396 3392 ? Sl Jan06 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 1943235 0.0 0.0 291068 188 ? Sl Jan06 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 2738465 0.0 0.0 306548 1300 ? S 2022 5:13 /bin/python3 /usr/local/bin/websockify -D 41835 localhost:5901
w55c785 2738498 0.0 0.0 150484 2024 ? Ssl 2022 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 2738502 0.0 0.0 372172 0 ? Sl 2022 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 2738507 0.0 0.0 130056 2296 ? Sl 2022 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 2738514 0.0 0.0 291068 12 ? Sl 2022 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 2842036 0.0 0.0 306916 8472 ? S 2022 4:41 /bin/python3 /usr/local/bin/websockify -D 45277 localhost:5901
w55c785 2842063 0.0 0.0 150488 796 ? Ssl 2022 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 2842067 0.0 0.0 372164 996 ? Sl 2022 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 2842072 0.0 0.0 130048 2200 ? Sl 2022 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 2842079 0.0 0.0 291068 1160 ? Sl 2022 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 2842637 0.0 0.0 306800 5436 ? S 2022 4:44 /bin/python3 /usr/local/bin/websockify -D 18776 localhost:5901
w55c785 2842665 0.0 0.0 150488 796 ? Ssl 2022 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 2842669 0.0 0.0 372156 1004 ? Sl 2022 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 2842674 0.0 0.0 130044 2092 ? Sl 2022 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 2842681 0.0 0.0 291068 3196 ? Sl 2022 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 2843224 0.0 0.0 306804 6492 ? S 2022 4:46 /bin/python3 /usr/local/bin/websockify -D 19282 localhost:5901
w55c785 2843254 0.0 0.0 150492 788 ? Ssl 2022 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 2843258 0.0 0.0 372172 1136 ? Sl 2022 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 2843263 0.0 0.0 130056 1340 ? Sl 2022 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 2843270 0.0 0.0 291068 3208 ? Sl 2022 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 3357883 0.0 0.0 306800 8584 ? S Jan07 2:03 /bin/python3 /usr/local/bin/websockify -D 9866 localhost:5901
w55c785 3357954 0.0 0.0 150508 948 ? Ssl Jan07 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 3357958 0.0 0.0 372164 944 ? Sl Jan07 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 3357963 0.0 0.0 150392 1004 ? Sl Jan07 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 3357970 0.0 0.0 291068 180 ? Sl Jan07 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 3362175 0.0 0.0 306800 8124 ? S Jan07 1:50 /bin/python3 /usr/local/bin/websockify -D 6015 localhost:5901
w55c785 3362239 0.0 0.0 150480 1872 ? Ssl Jan07 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 3362244 0.0 0.0 372172 1056 ? Sl Jan07 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 3362250 0.0 0.0 150392 996 ? Sl Jan07 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 3362257 0.0 0.0 291056 256 ? Sl Jan07 0:00 /usr/lib64/xfce4/xfconf/xfconfd

[gbyrket (Gerald Byrket)]

Hi Kenny.

Thanks for your post.

The port (5901) is used by VNC.

You said that you believe the stale processes are showing up due to something that happens when you change a config. Is that accurate?

I guess my thought is. Are you sure they are stale? Are you user “w55c785”?
If you are not, it could be that user “w55c785” is actually running the apps that use VNC.

I would recommend that you do the following.

1. If you are not user “w55c785”, then contact that user to see what they are doing.
2. If you are user “w55c785”, then continue
   a) Kill those processes
   b) Start a job that uses VNC using Open OnDemand.
   c) Do your ps and ensure that your job is running. Take note of the PID.
   d) Cancel your job using Open OnDemand.
   e) Check your process again to see if it is running.

Let us know the results, please.

Thanks,
-gerald

[kenny.hanson]

Yeah, w55c785 is me. I’ll do that but I have to go through every node and run a killall -u w55c785 in order to start with a clean slate. I’ll post results tomorrow.

Thanks Gerald

Kenny

[kenny.hanson]

It does appear that killing the job does some orphaned processes behind.

This first section is after I launched xfce remote desktop on a GPU node.

[root@tempest-gpu003 ~]# ps aux | grep w55c
w55c785 766305 0.0 0.0 222796 3496 ? S 11:52 0:00 /bin/bash /var/spool/slurm/d/job965082/slurm_script
w55c785 766348 0.2 0.0 533780 93136 ? Sl 11:52 0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop TurboVNC: tempest-gpu003:1 (w55c785) -auth /home/w55c785/.Xauthority -geometry 800x600 -depth 24 -rfbwait 120000 -rfbauth vnc.passwd -x509cert /home/w55c785/.vnc/x509_cert.pem -x509key /home/w55c785/.vnc/x509_private.pem -rfbport 5901 -fp catalogue:/etc/X11/fontpath.d -deferupdate 1 -dridir /usr/lib64/dri -registrydir /usr/lib64/xorg -idletimeout 0
w55c785 766356 0.0 0.0 222756 3488 ? S 11:52 0:00 bash /home/w55c785/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/tempest/output/31755876-7d52-43e7-8b40-b0dfba40dbba/script.sh
w55c785 766435 0.0 0.0 84968 3424 ? Ssl 11:52 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 766437 0.1 0.0 840908 64644 ? Sl 11:52 0:00 xfce4-session
w55c785 766439 0.0 0.0 372172 9212 ? Sl 11:52 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 766444 0.0 0.0 84736 5812 ? Sl 11:52 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 766448 0.0 0.0 248832 8984 ? Sl 11:52 0:00 /usr/libexec/at-spi2-registryd --use-gnome-session
w55c785 766451 0.0 0.0 291064 8196 ? Sl 11:52 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 766472 0.0 0.0 593340 24032 ? Sl 11:52 0:00 /usr/bin/xfce4-screensaver
w55c785 766485 0.1 0.0 3211536 85060 ? Sl 11:52 0:00 xfwm4
w55c785 766521 0.0 0.0 595996 26092 ? Sl 11:52 0:00 xfsettingsd
w55c785 766524 0.6 0.0 703752 33880 ? Sl 11:52 0:00 xfce4-panel
w55c785 766528 0.0 0.0 742932 24248 ? Sl 11:52 0:00 Thunar --daemon
w55c785 766540 0.2 0.0 762484 42372 ? Sl 11:52 0:00 xfdesktop
w55c785 766543 0.0 0.0 728616 24676 ? Sl 11:52 0:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libsystray.so 6 14680072 systray Status Tray Plugin Provides status notifier items (application indicators) and legacy systray items
w55c785 766544 0.1 0.0 675464 29188 ? Sl 11:52 0:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libpulseaudio-plugin.so 8 14680073 pulseaudio PulseAudio Plugin Adjust the audio volume of the PulseAudio sound system
w55c785 766545 0.0 0.0 580248 28068 ? Sl 11:52 0:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libxfce4powermanager.so 9 14680074 power-manager-plugin Power Manager Plugin Display the battery levels of your devices and control the brightness of your display
w55c785 766547 0.0 0.0 466744 15384 ? SNl 11:52 0:00 /usr/lib64/tumbler-1/tumblerd
w55c785 766556 0.0 0.0 587960 26568 ? Sl 11:52 0:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libactions.so 14 14680075 actions Action Buttons Log out, lock or other system actions
w55c785 766566 0.0 0.0 511336 8580 ? Sl 11:52 0:00 /usr/libexec/geoclue-2.0/demos/agent
w55c785 766623 0.2 0.0 2396716 26560 ? S 11:52 0:00 /bin/python3 /usr/local/bin/websockify -D 19610 localhost:5901
w55c785 766624 0.0 0.0 222796 2616 ? S 11:52 0:00 /bin/bash /var/spool/slurm/d/job965082/slurm_script
w55c785 766625 0.0 0.0 222796 2144 ? S 11:52 0:00 /bin/bash /var/spool/slurm/d/job965082/slurm_script
w55c785 766627 0.0 0.0 217128 892 ? S 11:52 0:00 tail -f --pid=766356 vnc.log
w55c785 766635 0.0 0.0 2401764 30136 ? S 11:52 0:00 /bin/python3 /usr/local/bin/websockify -D 19610 localhost:5901
root 766815 0.0 0.0 221944 1100 pts/1 S+ 11:53 0:00 grep --color=auto w55c

This [below] is after I killed the job. I did not exit from within session properly. I gave it a few runs to see if there was any delay in closing down processes, but there’s been no change for several minutes.

[root@tempest-gpu003 ~]# ps aux | grep w55c
w55c785 766435 0.0 0.0 84968 3424 ? Ssl 11:52 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 766439 0.0 0.0 372172 9212 ? Sl 11:52 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 766444 0.0 0.0 84736 5812 ? Sl 11:52 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 766451 0.0 0.0 291064 8196 ? Sl 11:52 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 766547 0.0 0.0 466744 15384 ? SNl 11:52 0:00 /usr/lib64/tumbler-1/tumblerd
w55c785 766623 0.1 0.0 2396716 26564 ? S 11:52 0:00 /bin/python3 /usr/local/bin/websockify -D 19610 localhost:5901
root 766870 0.0 0.0 221944 1100 pts/1 S+ 11:54 0:00 grep --color=auto w55c

[gbyrket (Gerald Byrket)]

Hi @kenny.hanson .

Thanks for the information. I’m going to try and reproduce this problem. To confirm, the websocket process is an unexpected process? There is no reason that it should be running, as far as you are concerned?

Thanks,
-gerald

[kenny.hanson]

I"m running Xfce Remote Desktoop from OOD, so the websocket is definitely expected. I ran this again and used the logout feature in Xfce which shuts the job down gracefully, but I still have artifacts leftover from the job. websockify isn’t being killed when the job is complete, along with xfce processes.

To reproduce, launch xfce desktop (this happens on cpu jobs too, not just gpu):

Exit gracefully using Log out (job shuts down as Completed):

Go to node where job ran:

ssh tempest-gpu001
ps aux | grep

I still have:

[root@tempest-gpu001 ~]# ps aux | grep w55c
w55c785 1624084 0.0 0.0 84980 3412 ? Ssl 14:54 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 1624097 0.0 0.0 372148 8760 ? Sl 14:54 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 1624102 0.0 0.0 84740 5668 ? Sl 14:54 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 1624109 0.0 0.0 291064 8004 ? Sl 14:54 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 1624342 0.0 0.0 2396716 26468 ? S 14:55 0:00 /bin/python3 /usr/local/bin/websockify -D 35119 localhost:5901
w55c785 1625715 0.0 0.0 84984 3460 ? Ssl 15:08 0:00 /usr/bin/dbus-daemon --syslog --fork --print-pid 6 --print-address 8 --session
w55c785 1625722 0.0 0.0 372172 9260 ? Sl 15:08 0:00 /usr/libexec/at-spi-bus-launcher
w55c785 1625727 0.0 0.0 84872 5864 ? Sl 15:08 0:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3
w55c785 1625734 0.0 0.0 291068 8180 ? Sl 15:08 0:00 /usr/lib64/xfce4/xfconf/xfconfd
w55c785 1625833 0.0 0.0 466684 15348 ? SNl 15:08 0:00 /usr/lib64/tumbler-1/tumblerd
w55c785 1625948 0.0 0.0 2396716 26844 ? S 15:08 0:00 /bin/python3 /usr/local/bin/websockify -D 25794 localhost:5901
root 1626354 0.0 0.0 221944 1092 pts/2 S+ 15:11 0:00 grep --color=auto w55c

bc_desktop/form.yml.erb:

[root@tempest-web bc_desktop]# cat form.yml.erb

/var/www/ood/apps/sys/bc_desktop/form.yml.erb

<%-

Get list of groups user belongs to

g_cmd = “/var/www/ood/apps/sys/jupyter/user-groups.sh $USER”
begin
output, status = Open3.capture2e(g_cmd)
if status.success?
accounts = output.split(“\n”).map(&:strip).reject(&:blank?).sort
else
raise output
end
rescue => e
accounts =
error = e.message.strip
end
-%>

Batch Connect app configuration file

@note Used to define the submitted cluster, title, description, and

hard-coded/user-defined attributes that make up this Batch Connect app.

---

attributes:
desktop: “xfce”
bc_vnc_idle: 0
bc_vnc_resolution:
required: true
node_type: null
num_cpus:
label: “Number of cpus”
widget: “number_field”
help: “You can request up to 64 cpus for your remote desktop session”
value: “2”
min: 2
max: 64

num_gpus:
label: “Number of GPU’s”
value: 0
min: 0
max: 2
help: “GPU assignment requires selecting GpuPriority partion”

gpu_model:
label: “Select GPU model”
widget: select
options:
- [ ‘NVidia A40’, ‘a40’ ]
- [ ‘NVidia A100’, ‘a100’ ]

custom_account:
label: “Account”
widget: select
options:
<%- accounts.each do |a| -%>
- [ “<%= a %>”, “<%= a %>” ]
<%- end -%>

custom_queue:
label: “Partition”
help: “Select the partition to allocate Jupyter job to.”
widget: select
options:
- [ ‘Priority’, ‘priority’ ]
- [ ‘GpuPriority’, ‘gpupriority’ ]
- [ ‘Legacy’, ‘legacy’ ]

custom_mem:
widget: “text_field”
value: “4G”
label: “Amount of RAM for session”
help: |
- The default is 4 GB
- 512M - Use 512 MB
- 32G - Use 32 GB

form:

• bc_vnc_idle
• desktop
• custom_account
• custom_queue
• custom_mem
• bc_num_hours
• num_cpus
• num_gpus
• gpu_model
• node_type
• bc_vnc_resolution
• bc_email_on_started

bc_desktop/submit.yml.erb:

[root@tempest-web bc_desktop]# cat submit.yml.erb

batch_connect:
template: vnc

script:
accounting_id: “<%= custom_account %>”
queue_name: “<%= custom_queue %>”
native: # … array of command line arguments …
- “-N 1”
- “-c”
- “<%= num_cpus %>”
- “–mem=<%= custom_mem %>”
<%- if num_gpus.to_i != 0 -%>
- “–gres=gpu:<%= num_gpus.to_i %>”
<%- end -%>
<%- if num_gpus.to_i != 0 -%>
- “–gres=gpu:<%= gpu_model %>:<%= num_gpus.to_i %>”
<%- end -%>

bc_desktop/template/desktops/xfce.sh:

Remove any preconfigured monitors

if [[ -f “${HOME}/.config/monitors.xml” ]]; then
mv “${HOME}/.config/monitors.xml” “${HOME}/.config/monitors.xml.bak”
fi

Copy over default panel if doesn’t exist, otherwise it will prompt the user

PANEL_CONFIG=“${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml”
if [[ ! -e “${PANEL_CONFIG}” ]]; then
mkdir -p “$(dirname “${PANEL_CONFIG}”)”
cp “/etc/xdg/xfce4/panel/default.xml” “${PANEL_CONFIG}”
fi

SCRLOCK_CONFIG=“${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-screensaver.xml”
if [[ ! -f ${SCRLOCK_CONFIG} ]]; then

Disable lock screen

cat < ${SCRLOCK_CONFIG}

<?xml version="1.0" encoding="UTF-8"?> EOF else xfconf-query -c xrce4-screensaver -p /lock/enabled -n -t bool -s false fi

DESKTOP_CONFIG=“${HOME}/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-desktop.xml”
if [[ ! -f ${DESKTOP_CONFIG} ]]; then

Disable lock screen

cat < ${DESKTOP_CONFIG}

<?xml version="1.0" encoding="UTF-8"?> EOF else xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-filesystem -n -t bool -s false xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-home -n -t bool -s true xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-removable -n -t bool -s false fi

Disable startup services

xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false
xfconf-query -c xfce4-session -p /startup/gpg-agent/enabled -n -t bool -s false

#xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-filesystem -n -t bool -s false
#xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-home -n -t bool -s true
#xfconf-query -c xfce4-desktop -p /desktop-icons/file-icons/show-removable -n -t bool -s false
#xfconf-query -c xrce4-screensaver -p /lock/enabled -n -t bool -s false

Disable useless services on autostart

AUTOSTART=“${HOME}/.config/autostart”
rm -fr “${AUTOSTART}” # clean up previous autostarts
mkdir -p “${AUTOSTART}”
for service in “pulseaudio” “rhsm-icon” “spice-vdagent” “tracker-extract” “tracker-miner-apps” “tracker-miner-user-guides” “xfce4-power-manag er” “xfce-polkit”; do
echo -e “[Desktop Entry]\nHidden=true” > “${AUTOSTART}/${service}.desktop”
done

Run Xfce4 Terminal as login shell (sets proper TERM)

TERM_CONFIG=“${HOME}/.config/xfce4/terminal/terminalrc”
if [[ ! -e “${TERM_CONFIG}” ]]; then
mkdir -p “$(dirname “${TERM_CONFIG}”)”
sed ‘s/^ {4}//’ > “${TERM_CONFIG}” << EOL
[Configuration]
CommandLoginShell=TRUE
EOL
else
sed -i
‘/^CommandLoginShell=/{h;s/=.*/=TRUE/};${x;/^$/{s//CommandLoginShell=TRUE/;H};x}’
“${TERM_CONFIG}”
fi

launch dbus first through eval becuase it can conflict with a conda environment

see conda installed dbus-launch causes XFCE to fail to start · Issue #700 · OSC/ondemand · GitHub

eval $(dbus-launch --sh-syntax)

Start up xfce desktop (block until user logs out of desktop)

xfce4-session

[gbyrket (Gerald Byrket)]

Hi @kenny.hanson .

Thanks for the information.

-gerald

[jeff.ohrstrom]

Are you running an LSF cluster? I think this issue has come up with LSF.

From that topic below you need these settings.

LSB_RESOURCE_ENFORCE=“cpu gpu memory”
LSF_PROCESS_TRACKING=Y
LSF_LINUX_CGROUP_ACCT=Y

LSF job termination on VNC logout

Hi All, Why is the corresponding LSF job not getting terminated when logging out from an interactive desktop session? Logging out stops X and VNC server on the worker node, but won’t stop the corresponding LSF job. The job would still show up in OOD, but if one tries to reconnect to that same session it fails. See screenshots below: [1] [2] [3] It works as expected with Slurm, but not with LSF. Does anyone have an idea? Thanks and best regards

[kenny.hanson]

Hi Jeff, We run slurm.

Kenny

[jeff.ohrstrom]

Good! This could be solvable then.

What’s your ProctrackType. We use proctrack/cgroup.

From the slurm conf docs: Slurm Workload Manager - slurm.conf

NOTE: "proctrack/linuxproc" and "proctrack/pgid" can fail to identify all processes associated with a job since processes can become a child of the init process (when the parent process terminates) or change their process group. To reliably track all processes, "proctrack/cgroup" is highly recommended.

[kenny.hanson]

Apparently we use ProctrackType=proctrack/pgid, which explains why it keeps leaving job artifacts upon RmtDesktop exit. We have a maintenance window coming up soon, I can switch it over then.

Kenny