OK - let’s try this, does the shell application work? I don’t know if you have websockets enabled.
You should have mod_proxy and mod_proxy_wstunnel enabled.
If you mean these then yes:
I added these to httpd.conf:
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
I updated OOD and restarted web server in browser, same issue. Anything here in this log snip?
[Fri Oct 18 11:16:00.334490 2024] [lua:info] [pid 1549515:tid 1549669] [client x.x.x.x:51905] res_content_length="" remote_user="myuser" req_server_name="ood.ouruni.edu" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_content_type="" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_is_https="true" res_content_disp="" req_port="443" req_protocol="HTTP/1.1" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions" req_accept_charset="" req_uri="/pun/sys/dashboard/batch_connect/sessions" log_time="2024-10-18T15:16:00.334165.0Z" res_content_language="" log_hook="ood" time_proxy="707.896" req_origin="" req_user_ip="x.x.x.x" req_handler="proxy-server" req_accept_language="en-us,en;q=0.9" time_user_map="0.004" res_location="" req_cache_control="max-age=0" req_accept_encoding="gzip, deflate, br, zstd" res_content_type="text/html; charset=utf-8" req_status="200" req_hostname="ood.ouruni.edu" local_user="myuser" res_content_encoding="" req_is_websocket="false" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" allowed_hosts="ood.ouruni.edu" req_method="GET" log_id="ZxJ7rwjLAJEZcsr5qdr3cQAAABA" res_content_location="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:02.443772 2024] [rewrite:trace2] [pid 1549515:tid 1549663] mod_rewrite.c(504): [client x.x.x.x:51905] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558013a70/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:02.443843 2024] [rewrite:trace3] [pid 1549515:tid 1549663] mod_rewrite.c(504): [client x.x.x.x:51905] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558013a70/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:02.443871 2024] [rewrite:trace3] [pid 1549515:tid 1549663] mod_rewrite.c(504): [client x.x.x.x:51905] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558013a70/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:02.443914 2024] [rewrite:trace1] [pid 1549515:tid 1549663] mod_rewrite.c(504): [client x.x.x.x:51905] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558013a70/initial] pass through /pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:02.512816 2024] [lua:info] [pid 1549515:tid 1549663] [client x.x.x.x:51905] req_uri="/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new" req_server_name="ood.ouruni.edu" req_user_ip="x.x.x.x" log_id="ZxJ7sgjLAJEZcsr5qdr3cgAAAA8" res_content_length="" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new" req_cache_control="" allowed_hosts="ood.ouruni.edu" req_accept_encoding="gzip, deflate, br, zstd" req_accept_language="en-us,en;q=0.9" req_accept_charset="" req_is_websocket="false" res_content_location="" req_is_https="true" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_protocol="HTTP/1.1" local_user="myuser" req_method="GET" req_origin="" res_content_type="text/html; charset=utf-8" res_content_encoding="" log_hook="ood" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" log_time="2024-10-18T15:16:02.512517.0Z" res_location="" time_user_map="0.004" time_proxy="66.566" res_content_language="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_content_type="" req_handler="proxy-server" remote_user="myuser" req_port="443" req_status="200" req_hostname="ood.ouruni.edu" res_content_disp="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:04.313693 2024] [rewrite:trace2] [pid 1549515:tid 1549679] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff520032e70/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.313868 2024] [rewrite:trace3] [pid 1549515:tid 1549679] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff520032e70/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.313899 2024] [rewrite:trace3] [pid 1549515:tid 1549679] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff520032e70/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.313941 2024] [rewrite:trace1] [pid 1549515:tid 1549679] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff520032e70/initial] pass through /pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.958447 2024] [lua:info] [pid 1549515:tid 1549679] [client x.x.x.x:51909] req_accept_encoding="gzip, deflate, br, zstd" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts" res_location="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_user_ip="x.x.x.x" req_cache_control="max-age=0" req_hostname="ood.ouruni.edu" res_content_disp="" res_content_length="" req_protocol="HTTP/1.1" log_id="ZxJ7tAjLAJEZcsr5qdr3cwAAABU" time_user_map="0.005" req_origin="https://ood.ouruni.edu" req_port="443" time_proxy="641.49" remote_user="myuser" res_content_location="" req_handler="proxy-server" res_content_type="text/html; charset=utf-8" req_method="POST" res_content_language="" res_content_encoding="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" req_server_name="ood.ouruni.edu" req_uri="/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts" req_accept_language="en-us,en;q=0.9" log_hook="ood" log_time="2024-10-18T15:16:04.958183.0Z" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new" req_is_https="true" local_user="myuser" req_is_websocket="false" req_accept_charset="" req_status="302" allowed_hosts="ood.ouruni.edu" req_content_type="application/x-www-form-urlencoded", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.975803 2024] [rewrite:trace2] [pid 1549515:tid 1549686] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.975840 2024] [rewrite:trace3] [pid 1549515:tid 1549686] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.975865 2024] [rewrite:trace3] [pid 1549515:tid 1549686] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:04.975896 2024] [rewrite:trace1] [pid 1549515:tid 1549686] mod_rewrite.c(504): [client x.x.x.x:51909] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] pass through /pun/sys/dashboard/batch_connect/sessions, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:06.036527 2024] [lua:info] [pid 1549515:tid 1549686] [client x.x.x.x:51909] req_accept_charset="" res_content_length="" req_origin="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" log_hook="ood" req_accept_language="en-us,en;q=0.9" log_id="ZxJ7tAjLAJEZcsr5qdr3dAAAABY" req_is_https="true" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions" req_is_websocket="false" req_method="GET" res_content_encoding="" local_user="myuser" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new" log_time="2024-10-18T15:16:06.36223.0Z" time_user_map="0.003" req_status="200" allowed_hosts="ood.ouruni.edu" req_protocol="HTTP/1.1" req_hostname="ood.ouruni.edu" remote_user="myuser" req_accept_encoding="gzip, deflate, br, zstd" res_content_location="" req_uri="/pun/sys/dashboard/batch_connect/sessions" req_cache_control="max-age=0" res_content_language="" req_user_ip="x.x.x.x" req_content_type="" res_content_disp="" req_server_name="ood.ouruni.edu" res_location="" req_handler="proxy-server" req_port="443" res_content_type="text/html; charset=utf-8" time_proxy="1058.548", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new
[Fri Oct 18 11:16:16.152633 2024] [rewrite:trace2] [pid 1549717:tid 1549754] mod_rewrite.c(504): [client x.x.x.x:51914] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558031ac0/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:16.152720 2024] [rewrite:trace3] [pid 1549717:tid 1549754] mod_rewrite.c(504): [client x.x.x.x:51914] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558031ac0/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:16.152748 2024] [rewrite:trace3] [pid 1549717:tid 1549754] mod_rewrite.c(504): [client x.x.x.x:51914] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558031ac0/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:16.152788 2024] [rewrite:trace1] [pid 1549717:tid 1549754] mod_rewrite.c(504): [client x.x.x.x:51914] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558031ac0/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:17.426199 2024] [lua:info] [pid 1549717:tid 1549755] [client x.x.x.x:51914] res_content_length="" req_port="443" res_content_type="text/javascript; charset=utf-8" req_handler="proxy-server" req_protocol="HTTP/1.1" req_content_type="" req_accept_language="en-us,en;q=0.9" req_accept_charset="" log_id="ZxJ7wNVbLNY1pBJGLzLqAgAAAMk" req_origin="" time_user_map="0.004" req_accept_encoding="gzip, deflate, br, zstd" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" log_hook="ood" req_user_ip="x.x.x.x" req_status="200" res_content_language="" req_method="GET" req_is_websocket="false" req_cache_control="" allowed_hosts="ood.ouruni.edu" res_content_location="" req_server_name="ood.ouruni.edu" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" time_proxy="1271.103" res_location="" local_user="myuser" req_is_https="true" log_time="2024-10-18T15:16:17.425841.0Z" req_hostname="ood.ouruni.edu" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" res_content_disp="" remote_user="myuser" res_content_encoding="" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729264566070" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.467792 2024] [rewrite:trace2] [pid 1549515:tid 1549674] mod_rewrite.c(504): [client x.x.x.x:51917] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff55800a390/initial] init rewrite engine with requested uri /pun/sys/dashboard/noVNC-1.3.0/vnc.html, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.467866 2024] [rewrite:trace3] [pid 1549515:tid 1549674] mod_rewrite.c(504): [client x.x.x.x:51917] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff55800a390/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/noVNC-1.3.0/vnc.html', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.467893 2024] [rewrite:trace3] [pid 1549515:tid 1549674] mod_rewrite.c(504): [client x.x.x.x:51917] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff55800a390/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/noVNC-1.3.0/vnc.html', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.467935 2024] [rewrite:trace1] [pid 1549515:tid 1549674] mod_rewrite.c(504): [client x.x.x.x:51917] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff55800a390/initial] pass through /pun/sys/dashboard/noVNC-1.3.0/vnc.html, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.472110 2024] [lua:info] [pid 1549515:tid 1549674] [client x.x.x.x:51917] time_proxy="1.88" req_origin="" res_content_location="" req_accept_language="en-us,en;q=0.9" req_accept_encoding="gzip, deflate, br, zstd" log_hook="ood" allowed_hosts="ood.ouruni.edu" req_uri="/pun/sys/dashboard/noVNC-1.3.0/vnc.html" res_content_type="text/html; charset=utf-8" res_content_length="15280" req_status="200" req_protocol="HTTP/1.1" req_user_ip="x.x.x.x" req_server_name="ood.ouruni.edu" res_location="" req_is_https="true" log_id="ZxJ7xAjLAJEZcsr5qdr3dQAAABI" res_content_disp="" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_cache_control="" time_user_map="0.004" req_port="443" local_user="myuser" res_content_encoding="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_accept_charset="" req_is_websocket="false" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" remote_user="myuser" res_content_language="" req_handler="proxy-server" req_method="GET" log_time="2024-10-18T15:16:20.471876.0Z" req_filename="proxy:http://localhost/pun/sys/dashboard/noVNC-1.3.0/vnc.html?autoconnect=true&path=rnode%2Fcomputenode.ouruni.edu%2F63696%2Fwebsockify&resize=remote&password=xxx&compression=6&quality=2&commit=Launch+Desktop" req_content_type="" req_hostname="ood.ouruni.edu", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:20.677002 2024] [rewrite:trace2] [pid 1549516:tid 1549630] mod_rewrite.c(504): [client x.x.x.x:51919] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550021990/initial] init rewrite engine with requested uri /rnode/computenode.ouruni.edu/63696/websockify
[Fri Oct 18 11:16:20.677075 2024] [rewrite:trace3] [pid 1549516:tid 1549630] mod_rewrite.c(504): [client x.x.x.x:51919] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550021990/initial] applying pattern '^(.*)' to uri '/rnode/computenode.ouruni.edu/63696/websockify'
[Fri Oct 18 11:16:20.677102 2024] [rewrite:trace3] [pid 1549516:tid 1549630] mod_rewrite.c(504): [client x.x.x.x:51919] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550021990/initial] applying pattern '^.*$' to uri '/rnode/computenode.ouruni.edu/63696/websockify'
[Fri Oct 18 11:16:20.677139 2024] [rewrite:trace1] [pid 1549516:tid 1549630] mod_rewrite.c(504): [client x.x.x.x:51919] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550021990/initial] pass through /rnode/computenode.ouruni.edu/63696/websockify
[Fri Oct 18 11:16:27.483395 2024] [rewrite:trace2] [pid 1549717:tid 1549758] mod_rewrite.c(504): [client x.x.x.x:51922] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558017d40/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:27.483518 2024] [rewrite:trace3] [pid 1549717:tid 1549758] mod_rewrite.c(504): [client x.x.x.x:51922] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558017d40/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:27.483548 2024] [rewrite:trace3] [pid 1549717:tid 1549758] mod_rewrite.c(504): [client x.x.x.x:51922] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558017d40/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:27.483606 2024] [rewrite:trace1] [pid 1549717:tid 1549758] mod_rewrite.c(504): [client x.x.x.x:51922] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff558017d40/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:28.800131 2024] [lua:info] [pid 1549717:tid 1549758] [client x.x.x.x:51922] req_accept_language="en-us,en;q=0.9" req_method="GET" req_content_type="" req_origin="" req_protocol="HTTP/1.1" log_id="ZxJ7y9VbLNY1pBJGLzLqAwAAAM0" res_location="" remote_user="myuser" req_server_name="ood.ouruni.edu" res_content_location="" res_content_type="text/javascript; charset=utf-8" req_hostname="ood.ouruni.edu" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_handler="proxy-server" res_content_disp="" req_port="443" req_accept_charset="" res_content_encoding="" req_is_websocket="false" log_time="2024-10-18T15:16:28.799881.0Z" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729264566071" req_cache_control="" local_user="myuser" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_user_ip="x.x.x.x" req_accept_encoding="gzip, deflate, br, zstd" allowed_hosts="ood.ouruni.edu" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" log_hook="ood" time_user_map="0.004" time_proxy="1314.298" res_content_language="" req_status="200" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" req_is_https="true" res_content_length="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:38.866935 2024] [rewrite:trace2] [pid 1549516:tid 1549644] mod_rewrite.c(504): [client x.x.x.x:51924] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550010b80/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:38.867017 2024] [rewrite:trace3] [pid 1549516:tid 1549644] mod_rewrite.c(504): [client x.x.x.x:51924] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550010b80/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:38.867045 2024] [rewrite:trace3] [pid 1549516:tid 1549644] mod_rewrite.c(504): [client x.x.x.x:51924] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550010b80/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:38.867088 2024] [rewrite:trace1] [pid 1549516:tid 1549644] mod_rewrite.c(504): [client x.x.x.x:51924] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff550010b80/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:40.070759 2024] [lua:info] [pid 1549516:tid 1549643] [client x.x.x.x:51924] req_is_websocket="false" req_protocol="HTTP/1.1" req_port="443" req_server_name="ood.ouruni.edu" req_status="200" req_accept_language="en-us,en;q=0.9" local_user="myuser" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" res_content_length="" log_time="2024-10-18T15:16:40.70516.0Z" res_location="" req_hostname="ood.ouruni.edu" time_user_map="0.004" allowed_hosts="ood.ouruni.edu" res_content_encoding="" log_id="ZxJ71lJ0Ezb2mlb2a3z9EQAAAEk" time_proxy="1201.332" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729264566072" log_hook="ood" res_content_language="" req_user_ip="x.x.x.x" req_accept_encoding="gzip, deflate, br, zstd" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" remote_user="myuser" res_content_location="" req_method="GET" req_handler="proxy-server" req_accept_charset="" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_is_https="true" res_content_disp="" req_content_type="" req_origin="" req_cache_control="" res_content_type="text/javascript; charset=utf-8", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:50.118380 2024] [rewrite:trace2] [pid 1549515:tid 1549688] mod_rewrite.c(504): [client x.x.x.x:51926] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:50.118488 2024] [rewrite:trace3] [pid 1549515:tid 1549688] mod_rewrite.c(504): [client x.x.x.x:51926] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:50.118516 2024] [rewrite:trace3] [pid 1549515:tid 1549688] mod_rewrite.c(504): [client x.x.x.x:51926] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:50.118555 2024] [rewrite:trace1] [pid 1549515:tid 1549688] mod_rewrite.c(504): [client x.x.x.x:51926] x.x.x.x - - [ood.ouruni.edu/sid#559bdbf958c0][rid#7ff554029e90/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 11:16:51.543697 2024] [lua:info] [pid 1549515:tid 1549676] [client x.x.x.x:51926] req_port="443" log_id="ZxJ74gjLAJEZcsr5qdr3dgAAABg" time_user_map="0.004" req_server_name="ood.ouruni.edu" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" allowed_hosts="ood.ouruni.edu" req_is_websocket="false" res_location="" log_time="2024-10-18T15:16:51.543451.0Z" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" req_content_type="" res_content_location="" res_content_disp="" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_status="200" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729264566073" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" res_content_type="text/javascript; charset=utf-8" req_is_https="true" local_user="myuser" req_method="GET" time_proxy="1422.969" res_content_language="" req_user_ip="x.x.x.x" req_origin="" req_protocol="HTTP/1.1" req_accept_language="en-us,en;q=0.9" req_hostname="ood.ouruni.edu" log_hook="ood" req_cache_control="" req_handler="proxy-server" req_accept_charset="" req_accept_encoding="gzip, deflate, br, zstd" res_content_encoding="" res_content_length="" remote_user="myuser", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
I do see these logs in /var/log/httpd/error_log
[Fri Oct 18 11:14:30.201493 2024] [so:warn] [pid 1549513:tid 1549513] AH01574: module proxy_module is already loaded, skipping
[Fri Oct 18 11:14:30.202023 2024] [so:warn] [pid 1549513:tid 1549513] AH01574: module proxy_balancer_module is already loaded, skipping
[Fri Oct 18 11:14:30.202553 2024] [so:warn] [pid 1549513:tid 1549513] AH01574: module proxy_http_module is already loaded, skipping
[Fri Oct 18 11:14:30.202905 2024] [so:warn] [pid 1549513:tid 1549513] AH01574: module proxy_wstunnel_module is already loaded, skipping
Indeed they have been loaded from this file:
# cat /etc/httpd/conf.modules.d/00-proxy.conf
# This file configures all the proxy modules:
LoadModule proxy_module modules/mod_proxy.so
LoadModule lbmethod_bybusyness_module modules/mod_lbmethod_bybusyness.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule lbmethod_bytraffic_module modules/mod_lbmethod_bytraffic.so
LoadModule lbmethod_heartbeat_module modules/mod_lbmethod_heartbeat.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_express_module modules/mod_proxy_express.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule proxy_fdpass_module modules/mod_proxy_fdpass.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_hcheck_module modules/mod_proxy_hcheck.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_uwsgi_module modules/mod_proxy_uwsgi.so
LoadModule proxy_wstunnel_module modules/mod_proxy_wstunnel.so
I see the rewrite rules for something like /rnode/ax.ouruni.edu/35538/websockify but I can’t see any log lines for the response code for this URL. That is, I see it being processed, but can’t see how the request ended.
Note that these URLs are all that we’re interested in. /rnode/<host>/<port>/websockify, everything else is noise.
If apache is in fact proxying this request to the origin (the compute node) then you’d see something in the websockify.log of the job to that effect. I guess at this point we just need to confirm that apache is in fact proxying back to websockfiy.
OK how do I confirm that?
httpd -M
[Fri Oct 18 12:24:29.627195 2024] [so:warn] [pid 1555129:tid 1555129] AH01574: module proxy_module is already loaded, skipping
[Fri Oct 18 12:24:29.628517 2024] [so:warn] [pid 1555129:tid 1555129] AH01574: module proxy_balancer_module is already loaded, skipping
[Fri Oct 18 12:24:29.629664 2024] [so:warn] [pid 1555129:tid 1555129] AH01574: module proxy_http_module is already loaded, skipping
[Fri Oct 18 12:24:29.630377 2024] [so:warn] [pid 1555129:tid 1555129] AH01574: module proxy_wstunnel_module is already loaded, skipping
Loaded Modules:
core_module (static)
so_module (static)
http_module (static)
proxy_module (shared)
proxy_balancer_module (shared)
proxy_http_module (shared)
proxy_wstunnel_module (shared)
auth_cas_module (shared)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
allowmethods_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_anon_module (shared)
authn_core_module (shared)
authn_dbd_module (shared)
authn_dbm_module (shared)
authn_file_module (shared)
authn_socache_module (shared)
authz_core_module (shared)
authz_dbd_module (shared)
authz_dbm_module (shared)
authz_groupfile_module (shared)
authz_host_module (shared)
authz_owner_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cache_module (shared)
cache_disk_module (shared)
cache_socache_module (shared)
data_module (shared)
dbd_module (shared)
deflate_module (shared)
dir_module (shared)
dumpio_module (shared)
echo_module (shared)
env_module (shared)
expires_module (shared)
ext_filter_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
info_module (shared)
log_config_module (shared)
logio_module (shared)
macro_module (shared)
mime_magic_module (shared)
mime_module (shared)
negotiation_module (shared)
remoteip_module (shared)
reqtimeout_module (shared)
request_module (shared)
rewrite_module (shared)
setenvif_module (shared)
slotmem_plain_module (shared)
slotmem_shm_module (shared)
socache_dbm_module (shared)
socache_memcache_module (shared)
socache_redis_module (shared)
socache_shmcb_module (shared)
status_module (shared)
substitute_module (shared)
suexec_module (shared)
unique_id_module (shared)
unixd_module (shared)
userdir_module (shared)
version_module (shared)
vhost_alias_module (shared)
watchdog_module (shared)
brotli_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_lock_module (shared)
lua_module (shared)
mpm_event_module (shared)
lbmethod_bybusyness_module (shared)
lbmethod_byrequests_module (shared)
lbmethod_bytraffic_module (shared)
lbmethod_heartbeat_module (shared)
proxy_ajp_module (shared)
proxy_connect_module (shared)
proxy_express_module (shared)
proxy_fcgi_module (shared)
proxy_fdpass_module (shared)
proxy_ftp_module (shared)
proxy_hcheck_module (shared)
proxy_scgi_module (shared)
proxy_uwsgi_module (shared)
ssl_module (shared)
systemd_module (shared)
cgid_module (shared)
ldap_module (shared)
authnz_ldap_module (shared)
http2_module (shared)
proxy_http2_module (shared)
What’s in your websockify.log?
WebSocket server settings:
- Listen on :45418
- No SSL/TLS support (no cert file)
- proxying from :45418 to localhost:5901
If apache actually connected to it, you’d see log lines like this for the same.
WebSocket server settings:
- Listen on :8491
- No SSL/TLS support (no cert file)
- proxying from :8491 to localhost:5901
some.ip.address - - [18/Oct/2024 13:43:25] some.ip.address: Plain non-SSL (ws://) WebSocket connection
some.ip.address - - [18/Oct/2024 13:43:25] some.ip.address: Path: '/websockify'
some.ip.address - - [18/Oct/2024 13:43:25] connecting to: localhost:5901
While that job is running on the compute node, can you curl/telnet to that ip:port combination? I know we saw that you could ping the machine, but now I’m wondering if there’s network connectivity to that port specifically.
Hm interesting even with firewalld disabled on the OOD server and compute node I’m seeing this:
nmap -sT -p 5901 compute
Starting Nmap 7.92 ( https://nmap.org ) at 2024-10-18 14:32 EDT
Nmap scan report for ax08 (10.198.24.59)
Host is up (0.00035s latency).
rDNS record for 10.198.24.59: compute.ouruni.edu
PORT STATE SERVICE
5901/tcp closed vnc-1
MAC Address: AC:1F:6B:XX:XX::XX Super Micro Computer)
telnet computenode 5901
Trying x.x.x.x...
telnet: connect to address x.x.x.x: Connection refused
I’m checking if something on RHEL 9 blocks this…
Should I just start TigerVNC as a service on the compute node?
It’s not 5901, it’s the other port. If you saw this log line - you’d try to hit port 8491.
So why is it localhost here:
WebSocket server settings:
- Listen on :18922
- No SSL/TLS support (no cert file)
- proxying from :18922 to localhost:5901
Shouldn’t this be from 18922 on the OOD server to TCP port 5901 on the compute node?
I have firewalld stopped on both so not sure what I’m looking for here but it does sound like you are on to something here.
And on the compute node when I start the desktop:
netstat -ano | grep 5901
tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN off (0.00/0/0)
ps -ef|grep xfce
myuser 254863 227808 0 14:42 ? 00:00:00 /usr/lib64/xfce4/xfconf/xfconfd
myuser 254885 254846 0 14:42 ? 00:00:00 xfce4-session
myuser 254896 1 0 14:42 ? 00:00:00 /usr/lib64/xfce4/xfconf/xfconfd
myuser 254988 254885 0 14:42 ? 00:00:00 xfce4-panel
myuser 255004 254988 0 14:42 ? 00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libsystray.so 6 12582924 systray Status Tray Plugin Provides status notifier items (application indicators) and legacy systray items
myuser 255007 254988 0 14:42 ? 00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libpulseaudio-plugin.so 8 12582925 pulseaudio PulseAudio Plugin Adjust the audio volume of the PulseAudio sound system
myuser 255147 254988 0 14:42 ? 00:00:00 /usr/lib64/xfce4/panel/wrapper-2.0 /usr/lib64/xfce4/panel/plugins/libactions.so 14 12582926 actions Action Buttons Log out, lock or other system actions
root 255726 253532 0 14:45 pts/0 00:00:00 grep --color=auto xfce
This look correct, no?
[websockify]: started successfully (proxying 18922 ==> localhost:5901)
Scanning VNC log file for user authentications...
Generating connection YAML file...
(tracker-miner-fs-3:255018): Tracker-WARNING **: 14:43:06.257: tracker_resource_set_relation: NULL is not a valid value.
The VNC server is on 5901. If you wanted to use a VNC client, you could connect to it through that port. We’re, however, using websockify so that we can transfer that VNC data over the web (over HTTP transport, not just TCP). And websockify is listening on 18922.
Thus apache is routing a request like /rnode/<host>/18922/websockify to that host and that port - attempting to connect to websockify.
Everything on the compute side appears to work OK. What does not appear to be working is actually having apache proxy back to that compute node.
Using the nc test from the doc,
Forbidden
You don't have permission to access this resource.
I’m trying the short name and FQDN.
Logs:
[Fri Oct 18 15:20:21.656492 2024] [rewrite:trace3] [pid 1565087:tid 1565220] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e800a8f0/initial] applying pattern '^.*$' to uri '/node/ournode/5432/'
[Fri Oct 18 15:20:21.656601 2024] [rewrite:trace1] [pid 1565087:tid 1565220] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e800a8f0/initial] pass through /node/ournode/5432/
[Fri Oct 18 15:20:21.755158 2024] [rewrite:trace2] [pid 1565087:tid 1565234] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e8008380/initial] init rewrite engine with requested uri /favicon.ico, referer: https://ood.ouruni.edu/node/ournode/5432/
[Fri Oct 18 15:20:21.755340 2024] [rewrite:trace3] [pid 1565087:tid 1565234] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e8008380/initial] applying pattern '^(.*)' to uri '/favicon.ico', referer: https://ood.ouruni.edu/node/ournode/5432/
[Fri Oct 18 15:20:21.755394 2024] [rewrite:trace3] [pid 1565087:tid 1565234] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e8008380/initial] applying pattern '^.*$' to uri '/favicon.ico', referer: https://ood.ouruni.edu/node/ournode/5432/
[Fri Oct 18 15:20:21.755808 2024] [rewrite:trace1] [pid 1565087:tid 1565234] mod_rewrite.c(504): [client 10.192.157.127:58399] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24e8008380/initial] pass through /favicon.ico, referer: https://ood.ouruni.edu/node/ournode/5432/
[Fri Oct 18 15:20:24.184343 2024] [rewrite:trace2] [pid 1565275:tid 1565317] mod_rewrite.c(504): [client 10.192.157.127:58400] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24d401ddf0/initial] init rewrite engine with requested uri /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:20:24.184561 2024] [rewrite:trace3] [pid 1565275:tid 1565317] mod_rewrite.c(504): [client 10.192.157.127:58400] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24d401ddf0/initial] applying pattern '^(.*)' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:20:24.184943 2024] [rewrite:trace3] [pid 1565275:tid 1565317] mod_rewrite.c(504): [client 10.192.157.127:58400] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24d401ddf0/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:20:24.185134 2024] [rewrite:trace1] [pid 1565275:tid 1565317] mod_rewrite.c(504): [client 10.192.157.127:58400] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24d401ddf0/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:20:25.483191 2024] [lua:info] [pid 1565275:tid 1565317] [client x.x.x.x:58400] res_content_disp="" local_user="myuser" req_status="200" req_is_https="true" res_content_encoding="" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729278758260" req_hostname="ood.ouruni.edu" req_user_ip="10.192.157.127" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" log_id="ZxK0-CEOXfT1Mq-v-EQNHQAAAMw" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" req_accept_language="en-us,en;q=0.9" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" res_content_type="text/javascript; charset=utf-8" time_proxy="1295.673" req_method="GET" time_user_map="0.004" res_location="" res_content_language="" req_origin="" req_port="443" req_content_type="" req_handler="proxy-server" req_server_name="ood.ouruni.edu" allowed_hosts="ood.ouruni.edu" res_content_length="" req_is_websocket="false" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_accept_encoding="gzip, deflate, br, zstd" remote_user="myuser" req_protocol="HTTP/1.1" res_content_location="" log_time="2024-10-18T19:20:25.482957.0Z" req_cache_control="" log_hook="ood" req_accept_charset="", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
ood.ouruni.edu/sid#557859f2d8c0][rid#7f24b00240a0/initial] applying pattern '^.*$' to uri '/pun/sys/dashboard/batch_connect/sessions.js', referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:31:23.197276 2024] [rewrite:trace1] [pid 1565275:tid 1565321] mod_rewrite.c(504): [client 10.192.157.127:58656] 10.192.157.127 - - [ood.ouruni.edu/sid#557859f2d8c0][rid#7f24b00240a0/initial] pass through /pun/sys/dashboard/batch_connect/sessions.js, referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
[Fri Oct 18 15:31:23.919705 2024] [lua:info] [pid 1565275:tid 1565323] [client x.x.x.x:58656] req_is_https="true" req_content_type="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" req_hostname="ood.ouruni.edu" req_user_ip="x.x.x.x" req_handler="proxy-server" req_cache_control="" req_port="443" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" res_content_location="" res_content_encoding="" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" local_user="myuser" req_accept_language="en-us,en;q=0.9" time_proxy="719.278" req_is_websocket="false" allowed_hosts="ood.ouruni.edu" log_hook="ood" res_content_length="" remote_user="myuser" time_user_map="0.005" res_content_disp="" log_time="2024-10-18T19:31:23.919389.0Z" res_location="" log_id="ZxK3iyEOXfT1Mq-v-EQNVAAAANA" res_content_type="text/javascript; charset=utf-8" req_status="200" res_content_language="" req_method="GET" req_server_name="ood.ouruni.edu" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1729279597402" req_referer="https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions" req_accept_charset="" req_origin="" req_accept_encoding="gzip, deflate, br, zstd" req_protocol="HTTP/1.1", referer: https://ood.ouruni.edu/pun/sys/dashboard/batch_connect/sessions
There’s a req_status="200" isn’t that usually a good thing?
Hmm is this CAS?
Edit:
Well I’m not sure here as the test says to use http which forwards to https.
Well, some success! I had to update the host_regex in /etc/ood/config/ood_portal.yml to use multiple sub domain names via a wildcard. I wish the errors were more indicative of that so perhaps this thread will help someone down the line…
@jeff.ohrstrom I got host based auth to work, hat tip to this RHEL article. However, my colleague gets the below errors. Any insight here?
Failed to submit session with the following error:Warning: Permanently added 'slurm.ouruni.edu' (ED25519) to the list of known hosts.
colleague@slurm.ouruni.edu: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased).
If this job failed to submit because of an invalid job name please ask your administrator to configure OnDemand to set the environment variable OOD_JOB_NAME_ILLEGAL_CHARS.
The Desktop session data for this session can be accessed under the staged root directory.
App 1883184 output: [2024-10-23 09:59:07 -0400 ] INFO "method=GET path=/pun/sys/dashboard/batch_connect/sys/bc_osc_matlab/session_contexts/new format=html controller=BatchConnect::SessionContextsController action=new status=200 allocations=20720 duration=61.69 view=38.15"
App 1883184 output: [2024-10-23 09:59:12 -0400 ] INFO "execve = [\"git\", \"describe\", \"--always\", \"--tags\"]"
App 1883184 output: [2024-10-23 09:59:12 -0400 ] INFO "method=GET path=/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new format=html controller=BatchConnect::SessionContextsController action=new status=200 allocations=19205 duration=51.62 view=33.20"
App 1883184 output: [2024-10-23 09:59:14 -0400 ] INFO "execve = [\"git\", \"describe\", \"--always\", \"--tags\"]"
App 1883184 output: [2024-10-23 09:59:14 -0400 ] INFO "method=GET path=/pun/sys/dashboard/batch_connect/sys/bc_osc_matlab/session_contexts/new format=html controller=BatchConnect::SessionContextsController action=new status=200 allocations=20621 duration=57.62 view=35.51"
App 1883184 output: [2024-10-23 09:59:20 -0400 ] INFO "execve = [\"git\", \"describe\", \"--always\", \"--tags\"]"
App 1883184 output: [2024-10-23 09:59:20 -0400 ] INFO "method=GET path=/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts/new format=html controller=BatchConnect::SessionContextsController action=new status=200 allocations=19214 duration=56.00 view=35.89"
App 1883184 output: [2024-10-23 09:59:27 -0400 ] INFO "execve = [{\"SLURM_CONF\"=>\"/etc/slurm/slurm.conf\"}, \"ssh\", \"-p\", \"22\",\"-o\", \"BatchMode=yes\", \"-o\", \"UserKnownHostsFile=/dev/null\", \"-o\", \"StrictHostKeyChecking=no\", \"slurm.ouruni.edu\", \"/usr/bin/sbatch\", \"-D\", \"/home/colleague/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/ab90ef88-d397-413a-a72c-48eb5c5dff3b\", \"--mail-type\", \"BEGIN\", \"-J\", \"sys/dashboard/sys/bc_desktop\", \"-o\", \"/home/colleague/ondemand/data/sys/dashboard/batch_connect/sys/bc_desktop/output/ab90ef88-d397-413a-a72c-48eb5c5dff3b/output.log\", \"-A\", \"zrc\", \"-t\", \"01:00:00\", \"--export\", \"NONE\", \"-N\", \"1\", \"--parsable\"]"
App 1883184 output: [2024-10-23 09:59:27 -0400 ] ERROR "ERROR: OodCore::JobAdapterError - Warning: Permanently added 'slurm.ouruni.edu' (ED25519) to the list of known hosts.\r\ncolleague@slurm.ouruni.edu: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password,hostbased)."
App 1883184 output: [2024-10-23 09:59:27 -0400 ] INFO "execve = [\"git\", \"describe\", \"--always\", \"--tags\"]"
App 1883184 output: [2024-10-23 09:59:27 -0400 ] INFO "method=POST path=/pun/sys/dashboard/batch_connect/sys/bc_desktop/session_contexts format=html controller=BatchConnect::SessionContextsController action=create status=200 allocations=24652 duration=279.86 view=38.06"
[ N 2024-10-23 10:04:27.6310 1883160/T3 age/Cor/CoreMain.cpp:1146 ]: Checking whether to disconnect long-running connections for process 1883184, application /var/www/ood/apps/sys/dashboard (production)
[ N 2024-10-23 12:00:02.2089 1883160/T8 age/Cor/CoreMain.cpp:670 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown)
[ N 2024-10-23 12:00:02.2090 1883160/T1 age/Cor/CoreMain.cpp:1245 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected...
[ N 2024-10-23 12:00:02.2090 1883160/T8 Ser/Server.h:901 ]: [ServerThr.1] Freed 0 spare client objects
[ N 2024-10-23 12:00:02.2091 1883160/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished
[ N 2024-10-23 12:00:02.2091 1883160/Ta Ser/Server.h:901 ]: [ServerThr.2] Freed 0 spare client objects
[ N 2024-10-23 12:00:02.2091 1883160/Te Ser/Server.h:901 ]: [ServerThr.4] Freed 0 spare client objects
[ N 2024-10-23 12:00:02.2091 1883160/Ta Ser/Server.h:558 ]: [ServerThr.2] Shutdown finished
[ N 2024-10-23 12:00:02.2092 1883160/Te Ser/Server.h:558 ]: [ServerThr.4] Shutdown finished
[ N 2024-10-23 12:00:02.2092 1883160/Tc Ser/Server.h:901 ]: [ServerThr.3] Freed 0 spare client objects
[ N 2024-10-23 12:00:02.2092 1883160/Tc Ser/Server.h:558 ]: [ServerThr.3] Shutdown finished
[ N 2024-10-23 12:00:02.2094 1883160/Tg Ser/Server.h:901 ]: [ApiServer] Freed 0 spare client objects
[ N 2024-10-23 12:00:02.2094 1883160/Tg Ser/Server.h:558 ]: [ApiServer] Shutdown finished
[ E 2024-10-23 12:00:07.2134 1883160/T1 age/Cor/TelemetryCollector.h:454 ]: Error contacting anonymous telemetry server: Connection timed out after 5001 milliseconds
[ N 2024-10-23 12:00:07.2665 1883160/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished
This is the error. You either need to provide HostBasedAuthentication so folks can seemlessly ssh into your submit_host or install the slurm binaries on the web node itself so that you can just issue commands like sbatch without having to ssh.
Yes HBA is working. The user can ssh without a password in both directions.
I do have these on the ondemand node:
pcp-pmda-slurm-6.2.0-5.el9_4.x86_64
slurm-20.02.7-1.el9.x86_64
slurm-pam_slurm-20.02.7-1.el9.x86_64
slurm-perlapi-20.02.7-1.el9.x86_64
Does slurmd need to be running?
Yeah I see Slurmd commands fail with:
squeue
slurm_load_jobs error: Zero Bytes were transmitted or received
Munge is installed and working.
The user cannot ssh to the submit_host. That’s what this error means, they failed to ssh.
slurmd doesn’t need to be running on the node, but commands like squeue or sbatch need to correctly communicate with the controller.
So do all new users need to ssh first between the 2 nodes to get that trust on first use key uploaded?
I had an old slurm.conf. Using the same one on the slurm node got Slurm commands to work.
With HostBasedAuthentication they don’t need their own keys. The error message says that hostbased was attempted, so I’m not sure what’s going on there.
OK, then the sshing to the submit_host is moot. Just reconfigure the cluster.d file to use commands locally by removing the submit_host section.
After a reboot of the server why is
Oct 23 16:33:09 ood.ouruni.edu httpd[11139]: AH00526: Syntax error on line 83 of /etc/httpd/conf.d/ood-portal.conf:
Oct 23 16:33:09 ood.ouruni.edu httpd[11139]: Invalid command 'LuaRoot', perhaps misspelled or defined by a module not included in the server configuration
Do these Lua options need to be set?