Setting up interactive apps - enable reverse proxy

[maikenp (Maiken Pedersen)]

Hi!

I have successfully set up an ondemand server with the basics, and can log in using oidc, see the home dir, and I can access shell.

Now I am trying to set up interactive apps and am following the seemingly straightforward instructions Setup Interactive Apps — Open OnDemand 2.0.20 documentation

I have installed the required software on the compute-nodes
I have defined the cluster and I see it (and can open a shell, which I think means the config is correct)
I have generated a new apache config after adding

host_regex: '[\w]+\.hpc\.uio\.no'
node_uri: '/node'
rnode_uri: '/rnode'

and checked that the regex is correct for my compute-nodes (e.g. hepp03.hpc.uio.no)

I do not have dex installed as I do not think I need this (?).
Also, it did not work together with my oidc configuration, so I completely removed the ondemand-dex package.

I have generated the new apache config and restarted apache, so basically gone through all the steps in the instructions step 1, 2 and until 3.2

When following procedure to check that it works:
https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/enable-reverse-proxy.html#verify-it-works

I see

I am not managing to find out what is the problem and how to fix.

In the ood error log I see:

[Mon Apr 03 14:29:00.966702 2023] [proxy:error] [pid 827057:tid 139674282620672] (113)No route to host: AH00957: HTTP: attempt to connect to <ip-to-compute-node>:5432 (*) failed
[Mon Apr 03 14:29:00.966740 2023] [proxy_http:error] [pid 827057:tid 139674282620672] [client 193.157.181.81:55450] AH01114: HTTP: failed to make connection to backend: hepp03.hpc.uio.no, referer: https://auth.dataporten.no/
[Mon Apr 03 14:29:00.967109 2023] [lua:info] [pid 827057:tid 139674282620672] [client <client-ip>:55450] req_user_ip="<client-ip>" res_location="" log_time="2023-04-03T12:29:00.967034.0Z" req_status="503" req_handler="proxy-server" remote_user="<username>@uio.no" log_hook="ood" req_accept_language="en-us,en;q=0.5" log_id="ZCrGjFCEpR39ZNKXcpDKwwAAAFM" res_content_encoding="" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" req_content_type="" req_accept_charset="" res_content_disp="" req_method="GET" req_is_websocket="false" local_user="<user-name>" req_port="443" req_accept_encoding="gzip, deflate, br" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/111.0" req_is_https="true" req_origin="" req_referer="https://auth.dataporten.no/" time_user_map="0.002" time_proxy="0.763" res_content_language="" req_server_name="ood-hepp.uio.no" req_cache_control="" res_content_location="" req_protocol="HTTP/1.1" req_uri="/node/hepp03.hpc.uio.no/5432" res_content_type="text/html; charset=iso-8859-1" res_content_length="43" req_hostname="ood-hepp.uio.no" req_filename="proxy:http://hepp03.hpc.uio.no:5432/node/hepp03.hpc.uio.no/5432", referer: https://auth.dataporten.no/
[Mon Apr 03 14:33:32.433243 2023] [proxy:error] [pid 828484:tid 139862992746240] (113)No route to host: AH00957: HTTP: attempt to connect to <compute-node-ip>:5432 (*) failed
[Mon Apr 03 14:33:32.433277 2023] [proxy_http:error] [pid 828484:tid 139862992746240] [client <client-ip>:55511] AH01114: HTTP: failed to make connection to backend: hepp03.hpc.uio.no, referer: https://auth.dataporten.no/
[Mon Apr 03 14:33:32.433667 2023] [lua:info] [pid 828484:tid 139862992746240] [client <client-ip>:55511] req_user_ip="<user-ip>" req_filename="proxy:http://hepp03.hpc.uio.no:5432/node/hepp03.hpc.uio.no/5432" req_handler="proxy-server" log_hook="ood" req_referer="https://auth.dataporten.no/" req_cache_control="" req_port="443" res_location="" req_is_https="true" res_content_location="" log_time="2023-04-03T12:33:32.433580.0Z" local_user="<user-name>" log_id="ZCrHnEA3ybwd3vy0wGGVUgAAABc" res_content_type="text/html; charset=iso-8859-1" req_server_name="ood-hepp.uio.no" req_protocol="HTTP/1.1" req_accept="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8" req_accept_charset="" req_uri="/node/hepp03.hpc.uio.no/5432" time_proxy="0.924" req_hostname="ood-hepp.uio.no" req_accept_encoding="gzip, deflate, br" req_status="503" res_content_encoding="" time_user_map="0.002" res_content_language="" req_content_type="" req_user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/111.0" req_method="GET" res_content_disp="" req_is_websocket="false" req_accept_language="en-us,en;q=0.5" remote_user="<username>@uio.no" res_content_length="43" req_origin="", referer: https://auth.dataporten.no/

(removed user-name,id etc in the above with etc

Any hints to help me along the way?

Thanks!

[maikenp (Maiken Pedersen)]

Update: pin-pointed it down to it being due to firewalld rules on the compute-node.
So then it is a matter of just finding the correct set up rules to get it working.

[jeff.ohrstrom]

Hi and welcome! Sorry we didn’t get to your topic in time as this is your first time posting, that’s on us. Glad to hear you solved it and sorry again for not replying sooner.

[maikenp (Maiken Pedersen)]

Hi, reopening this as have some more questions/info.

I have not manged to figure out what firewall rules to set to get VCN working.

If firewall is off on the compute nodes everything works perfectly (just following the [https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/enable-reverse-proxy.html](https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/enable-reverse-proxy.html#verify-it-works but changing the default desktop from mate to xfcv) and I see the bc_desktop as expected

Our compute nodes have firewall on normally.

I do not see any information in the documentation about firewall settings on the compute nodes. Do you assume they do not have firewall on?

I have tried to add the ood-server as source, and adding port 5901/tcp specifically - this was not the solution.

Any suggestions what I need to change?

Thanks!

[maikenp (Maiken Pedersen)]

Solved.

The trick was adding the source to the zone=trusted instead of public. This will allow all connections from the source (ood-server).

firewall-cmd --permanent --zone=trusted --add-source <ip-address-of-ood-server>