Reverse proxy (rnode or node) to SSL service

Get Help
1

Hi,

I’m trying to set up an app which will proxy to a service that requires SSL. I found this post:

and tried the solution there, but I still get

Bad Request
Your browser sent a request that this server could not understand.
Reason: You’re speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

In case it matters, the app in question (NoMachine) handles all authentication so all I really need to do here is reserve the node and produce a link that will reverse proxy to the web interface. Restricting the service to a specific user is something we’ll add later via Slurm prolog/epilog scripts to manage the nxserver config.

My current view.html.erb is

<p>
   The link below will launch your NoMachine Workstation desktop in a browser
   window/tab. If you'd prefer to use the NoMachine native client, create an
   ssh tunnel to the session:
</p>
<ol>
  <li><pre>ssh login-01 -L 24000:<%= host %>:4000</pre></li>
  <li>Add a NoMachine connection to "localhost" using port "24000" and the NX protocol.</li>
</ol>
<hr>
<a href="/node/<%= host %>/<%= port %>"  target="_blank" rel="noreferrer noopener">NoMachine Web Interface (node)</a>
<hr>
<a href="/rnode/<%= host %>/<%= port %>"  target="_blank" rel="noreferrer noopener">NoMachine Web Interface (rnode)</a>

The submitted job just grabs the node with --exclusive to make sure no one else can start jobs on it, starting sleep 90d.

In /opt/ood/mod_ood_proxy/lib/ood/proxy.lua I have added this:

function set_reverse_proxy(r, conn)
  -- find protocol used by parsing the request headers
  local protocol = (r.headers_in['Upgrade'] and "ws://" or "http://")
  if upstreamPort then
    -- If specified port was used, then use secure protocols
    if upstreamPort == '4443' then
      protocol = (r.headers_in['Upgrade'] and "wss://" or "https://")
    end
  end

And SSLProxyEngine On is in the VirtualHost config.

I set the port in before.sh.erb:

# Export the module function if it exists
[[ $(type -t module) == "function" ]] && export -f module

export port=4443

When a session starts, the generated URLs in the view are:

https://ondemand.bruno.czbiohub.org/node/gpu-sm01-14.clusternet/4443
https://ondemand.bruno.czbiohub.org/rnode/gpu-sm01-14.clusternet/4443

Both producing the error I mentioned above.

I’m not sure where to look next for where this is going off the rails.

griznog

2

Could you share the entirety of your modified proxy.lua file as a txt file? That would help me to do a little bit of digging.

3

The complete file is:

--[[
  set_reverse_proxy

  Modify a given request to utilize mod_proxy for reverse proxying.
--]]
function set_reverse_proxy(r, conn)
  -- find protocol used by parsing the request headers
  local protocol = (r.headers_in['Upgrade'] and "ws://" or "http://")
  if upstreamPort then
    -- If specified port was used, then use secure protocols
    if upstreamPort == '4443' then
      protocol = (r.headers_in['Upgrade'] and "wss://" or "https://")
    end
  end

  -- setup request to use mod_proxy for the reverse proxy
  r.handler = "proxy-server"
  r.proxyreq = apache2.PROXYREQ_REVERSE

  -- define reverse proxy destination using connection object
  if conn.socket then
    r.filename = "proxy:unix:" .. conn.socket .. "|" .. protocol .. "localhost" .. conn.uri
  else
    r.filename = "proxy:" .. protocol .. conn.server .. conn.uri
  end

  -- include useful information for the backend server

  -- provide the protocol used
  r.headers_in['X-Forwarded-Proto'] = r.is_https and "https" or "http"

  -- provide the authenticated user name
  r.headers_in['X-Forwarded-User'] = conn.user or ""

  -- **required** by PUN when initializing app
  r.headers_in['X-Forwarded-Escaped-Uri'] = r:escape(conn.uri)

  -- set timestamp of reverse proxy initialization as CGI variable for later hooks (i.e., analytics)
  r.subprocess_env['OOD_TIME_BEGIN_PROXY'] = r:clock()
end

return {
  set_reverse_proxy = set_reverse_proxy
}
4

I’m wondering about the bit of code that’s missing that’s in the solution you linked to, but not your file - upstreamPort isn’t declared or set anywhere. What happens when you include this:

-- find protocol used by parsing the request headers
  -- Check if an upstream port was set for reverse proxies.
  local upstreamPort = nil
  local isUpstreamPortSet = (r.subprocess_env['MATCH_PORT'] and 'true' or 'false')
  if isUpstreamPortSet == 'true' then
    upstreamPort = r.subprocess_env['MATCH_PORT']
  end
5

Thanks @hrandquist , I had missed getting all the change they made into it. Adding that has some impact, the error I get is now:

Proxy Error
The proxy server could not handle the request
Reason: Error during SSL Handshake with remote server

Which I’m going to assume is a problem with the NoMachine self-signed cert and start digging into the Apache config.

6

And with the addition of

  - 'SSLVerifyClient none'
  - 'SSLProxyVerify none'
  - 'SSLProxyProtocol TLSv1.2'
  - 'SSLProxyCheckPeerName off'

to my portal/httpd config, I now get an error from the NoMachine client:

Error
An unexpected error was encountered during the installation of the Web Player component.

Which means, I think, that the proxying is working but the extra stuff in the URL is breaking the NoMachine web client.

7

I contacted NoMachine support, and their recommendation for putting it behind a reverse proxy using Apache is the config:

SSLProxyEngine On
SSLProxyVerify none
SSLProxyCheckPeerCN Off
SSLProxyCheckPeerName Off
SSLProxyCheckPeerExpire Off


ProxyPass  "/" "https://192.168.3.201:4443/"
ProxyPassReverse "/"  "https://192.168.3.201:4443/"
ProxyPass "/nxplayer"  "https://192.168.3.201:4443/nxplayer"
ProxyPassReverse "/nxplayer"  "https://192.168.3.201:4443/nxplayer"

How do I get the equivalent of those ProxyPass* directives into OnDemand?

8

After speaking with a colleague about this, I believe we already implement the behavior you’re looking for, and it’s possible you just need to add the correct settings to the SSL config. What happens if you add

 SSLVerifyClient none
 SSLProxyVerify none
 SSLProxyProtocol TLSv1.2
 SSLProxyCheckPeerName off
 SSLProxyVerify none
 SSLProxyCheckPeerCN Off
 SSLProxyCheckPeerName Off
 SSLProxyCheckPeerExpire Off

to the ssl config?

9

The SSL part is all sorted now, I think, at least I no longer get any SSL related errors and instead get a problem with the URLs the NoMachine webplayer expects to work through the proxy.

For anyone coming across this in the future, I opened this issue on github:

Which I then closed as it more or less duplicates these two issues:

  1. For addressing origin servers that use SSL:
    naive support for secure origins · Issue #3179 · OSC/ondemand · GitHub

  2. For handling things that need extra help proxying to specifc URLS:
    mod_subsitute to correct assets/links on proxied applications · Issue #2311 · OSC/ondemand · GitHub

It doesn’t seem possible to get NoMachine to work without the functionality of 2311 and I’m kind of reluctant to release something to users that depend on me locally modifying the /opt/ood/mod_ood_proxy/lib/ood/proxy.lua file as I’m certain to forget or break that in the future :slight_smile:

Thanks to everyone who looked (and is looking into) this, also for Open OnDemand in general which is invaluable to us and our users and the work that goes into it is greatly appreciated.