Nothing that i can see…
lsof -i :0 -t doesn’t give any answers.
I can’t reboot the server as there are running jobs, but might try with another one.
I think you’d want to try a ps -elf | grep ':0' maybe? It’s not an actual file that lsof would return I don’t think.
It shows a lot of files… I’ve killed all processes owned by me just in case. Same issue…
I was starting xfce as part of script.sh.erb as well. No errors, but now it just fails… The search continues. 
Script starting...
Starting websocket server...
[websockify]: pid: 875462 (proxying 36088 ==> localhost:5901)
[websockify]: log file: ./websockify.log
[websockify]: waiting ...
[websockify]: started successfully (proxying 36088 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...
Cleaning up...
Killing Xvnc process ID 875444
$ cat /etc/ood/config/clusters.d/ionic.yml
---
v2:
metadata:
title: Ionic Cluster
hidden: false
login:
host: loginnode
job:
adapter: slurm
cluster: ionic
lib: "/usr/lib64"
bin: "/usr/bin"
conf: "/etc/slurm/slurm.conf"
batch_connect:
basic:
script_wrapper: |
module purge
%s
vnc:
min_port: 10000
max_port: 60000
script_wrapper: |
module purge
export PATH="/opt/TurboVNC/bin:$PATH"
export WEBSOCKIFY_CMD="/usr/bin/websockify"
export XDG_RUNTIME_DIR="/tmp/${UID}"
%s
does this look correct…? Thank you again!
Yea that cluster.d file looks OK.
From that log it seems to be exiting immediately with no errors. Very strange indeed.
It seems like this stopped immediately? What’s the content of vnc.log?
$ cat vnc.log
TurboVNC Server (Xvnc) 64-bit v3.1.1 (build 20240127.sdl9)
Copyright (C) 1999-2024 The VirtualGL Project and many others (see README.md)
Visit http://www.TurboVNC.org for more information on TurboVNC
13/11/2024 13:53:47 Using security configuration file /etc/turbovncserver-security.conf
13/11/2024 13:53:47 Enabled security type 'tlsvnc'
13/11/2024 13:53:47 Enabled security type 'tlsotp'
13/11/2024 13:53:47 Enabled security type 'tlsplain'
13/11/2024 13:53:47 Enabled security type 'x509vnc'
13/11/2024 13:53:47 Enabled security type 'x509otp'
13/11/2024 13:53:47 Enabled security type 'x509plain'
13/11/2024 13:53:47 Enabled security type 'vnc'
13/11/2024 13:53:47 Enabled security type 'otp'
13/11/2024 13:53:47 Enabled security type 'unixlogin'
13/11/2024 13:53:47 Enabled security type 'plain'
_XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root
13/11/2024 13:53:47 Desktop name 'TurboVNC: node805.host.edu:1 (aadvorki)' (node908.ionic.cs.princeton.edu:1)
13/11/2024 13:53:47 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
13/11/2024 13:53:47 Listening for VNC connections on TCP port 5901
13/11/2024 13:53:47 Interface 0.0.0.0
13/11/2024 13:53:47 Framebuffer: BGRX 8/8/8/8
13/11/2024 13:53:47 New desktop size: 800 x 600
13/11/2024 13:53:47 New screen layout:
13/11/2024 13:53:47 0x00000040 (output 0x00000040): 800x600+0+0
13/11/2024 13:53:47 Maximum clipboard transfer size: 1048576 bytes
13/11/2024 13:53:47 VNC extension running!
OK. How about websockify.log? You should have seen something like this out of vnc.log, but you seem to have ended before that.
13/11/2024 13:57:15 Got connection from client 127.0.0.1
13/11/2024 13:57:15 Using protocol version 3.8
13/11/2024 13:57:15 Enabling TightVNC protocol extensions
13/11/2024 13:57:15 Advertising Tight auth cap 'VENCRYPT'
13/11/2024 13:57:15 Advertising Tight auth cap 'VNCAUTH_'
13/11/2024 13:57:15 Advertising Tight auth cap 'ULGNAUTH'
13/11/2024 13:57:15 Full-control authentication enabled for 127.0.0.1
$ cat websockify.log
WebSocket server settings:
- Listen on :42152
- No SSL/TLS support (no cert file)
- proxying from :42152 to localhost:5901
Hello, Jeff.
I am wondering if there is anything I can check, please? Is there a verbose mode? I’m totally stuck at this point.
Thank you!
Yea i’m a bit at a loss too. I guess I’d check /var/log for anything that could point us in the right direction.
Is your output.log the same as it’s above? where it just goes from running to Cleaning up... or is there anything more in that log file?
Also I guess I’d ask to see your ood_portal.yml just to check you went through these steps:
Compute node: node805.ionic.cs.host.edu
Are you seeing any issues with my configuration? Again, thank you very much for all your help.
$ more ood_portal.yml
# File managed by Puppet - do not edit!
---
listen_addr_port:
- '443'
- '80'
servername: ondemand.ionic.cs.host.edu
port: '443'
ssl:
- SSLCertificateFile /etc/acme.sh/certs/ondemand.ionic.cs.host.edu/fullchain.pem
- SSLCertificateKeyFile /etc/acme.sh/keys/ondemand.ionic.cs.host.edu/private.key
- SSLCertificateChainFile /etc/acme.sh/certs/ondemand.ionic.cs.host.edu/cert.pem
disable_logs: false
logroot: logs
use_rewrites: true
use_maintenance: true
maintenance_ip_allowlist: []
security_strict_transport: true
lua_root: "/opt/ood/mod_ood_proxy/lib"
user_map_match: ".*"
pun_stage_cmd: sudo /opt/ood/nginx_stage/sbin/nginx_stage
auth:
- AuthType CAS
- Require valid-user
- RequestHeader unset Authorization
custom_vhost_directives: []
custom_location_directives: []
root_uri: "/pun/sys/dashboard"
public_uri: "/public"
public_root: "/var/www/ood/public"
logout_uri: "/logout"
logout_redirect: "/pun/sys/dashboard/logout"
host_regex: "[\\w.-]+\\.cs\\.host\\.edu"
node_uri: "/node"
rnode_uri: "/rnode"
nginx_uri: "/nginx"
pun_uri: "/pun"
pun_socket_root: "/var/run/ondemand-nginx"
pun_max_retries: 5
oidc_remote_user_claim: preferred_username
oidc_scope: openid profile email
oidc_session_inactivity_timeout: 28800
oidc_session_max_duration: 28800
oidc_state_max_number_of_cookies: 10 true
oidc_settings: {}
dex_uri: "/dex"
Might could be. The only thing I think of is host_regex: "[\\w.-]+\\.cs\\.host\\.edu" where you maybe only need 1 escape .
Have you verified it works in this step:
$ nc -l 5432
GET /node/node805.ionic.cs.host.edu/5432 HTTP/1.1
Host: node805.ionic.cs.host.edu:5432
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:132.0) Gecko/20100101 Firefox/132.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Cookie: MOD_AUTH_CAS_S=ea12516e277e513336ae981b074c5593
Upgrade-Insecure-Requests: 1
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Priority: u=0, i
X-Forwarded-Proto: https
X-Forwarded-User: aadvorki
X-Forwarded-Escaped-Uri: %2fnode%2fnode805.ionic.cs.host.edu%2f5432
Via: 1.1 ondemand.ionic.cs.princeton.edu
X-Forwarded-For: 128.11.111.111
X-Forwarded-Host: ondemand.ionic.cs.host.edu
X-Forwarded-Server: ondemand.ionic.cs.host.edu
Connection: close
Even though in the browser I get error message:
Failed to connect to node805.ionic.cs.host.edu:5432
Is this what’s expected? Thank you!
Yea that seems OK - maybe it faild on the browser because NC doesn’t return any response.
Hmmmmmmm…
There’s nothing more in output.log, vnc.out or websockify.log that would indicate any error?
Nothing.
I tried to increase debug level under my pam.d configs and seeing:
pam_xauth(su-l:session): user has no DISPLAY, doing nothing
I haven’t looked into it further, but that doesn’t seem right…
So my issue had to do with the apache configuration. Below line fixed the problem.
CASScope /
Thank you for all your help! Have a wonderful day.