# Reverse Proxy on Separate Host?

**URL:** <https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499>\
**Category:** Get Help\
**Created:** [May 24, 2021, 8:29pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499 "2021-05-24T20:29:12Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nathan.gregg](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nathan.gregg](https://discourse.openondemand.org/u/nathan.gregg)\
**Post date:** [May 24, 2021, 8:29pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/1 "2021-05-24T20:29:12Z")

</div>

Hello All,

We are looking to deploy OOD and have a question regarding the Reverse Proxy Setup. We are curious is it possible to configure the reverse proxy on a separate host from our main OOD server? The reason for this is for security reasons we have our OOD isolated from the rest of our environment and it currently cannot talk to compute nodes directly which I believe is needed for the reverse proxy to work. We are curious if we can shift this part of the workflow to another host that does have the necessary connectivity to the compute nodes?

Sorry in advance if this question seems confusing and thanks in advance for the support!

Nate

---

<div class="post-metadata">

**Author:** ![rsettlag](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rsettlag/32/78_2.png) [@rsettlag](https://discourse.openondemand.org/u/rsettlag)\
**Post date:** [May 25, 2021, 3:25am UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/2 "2021-05-25T03:25:44Z")

</div>

Hi,  
yes, you can. I had this setup at our site where I had two OODs running, each within the networks of different clusters, but wanted users to be able to submit and launch jobs across the clusters so I had to pass off the proxy from one OOD server to the other. Once the job was launched, the other OOD server was just handling the proxy. IIRC, I think you can, for instance, add an ONDEMAND\_HOST.

Eric gave me this suggestion when I was doing this, change the submit.yml.erb file, for instance:

```auto
 batch_connect:
   template: vnc
  conn_params:
    - ONDEMAND_HOST
 script:
   accounting_id: "<%= account %>"
  job_environment:
    ONDEMAND_HOST: "this.is.a.custom.set.value"

```

Which results in the connection.yml file containing:

```auto
ONDEMAND_HOST: this.is.a.custom.set.value

```

---

<div class="post-metadata">

**Author:** ![nathan.gregg](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nathan.gregg](https://discourse.openondemand.org/u/nathan.gregg)\
**Post date:** [May 25, 2021, 2:48pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/3 "2021-05-25T14:48:12Z")

</div>

Thanks! We will give this a shot and see if we can get it to work. I much appreciate the feedback.

Nate

---

<div class="post-metadata">

**Author:** ![nathan.gregg](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nathan.gregg](https://discourse.openondemand.org/u/nathan.gregg)\
**Post date:** [May 27, 2021, 1:19pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/4 "2021-05-27T13:19:46Z")

</div>

@rsettlag One quick and maybe silly question … does the host that is running the reverse proxy (in your case inside each cluster) need to be exposed publicly for the proxy to work correctly? I suspect the answer is yes, and when users open something just a Jupyter notebook they are handed over to the host running the reverse proxy.

Thanks in advance for your insight.

Nate

---

<div class="post-metadata">

**Author:** ![rsettlag](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rsettlag/32/78_2.png) [@rsettlag](https://discourse.openondemand.org/u/rsettlag)\
**Post date:** [May 27, 2021, 1:37pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/5 "2021-05-27T13:37:25Z")

</div>

Not sure what you mean by publicly, we are public within the VT network and access controls apply. In our case, this is behind the VT VPN.

---

<div class="post-metadata">

**Author:** ![nathan.gregg](https://avatars.discourse-cdn.com/v4/letter/n/5daacb/32.png) [@nathan.gregg](https://discourse.openondemand.org/u/nathan.gregg)\
**Post date:** [May 27, 2021, 1:48pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/6 "2021-05-27T13:48:15Z")

</div>

Sorry @rsettlag, I should have been more clear … does the user need to access the reverse proxy directly via port 443 (behind VPN is fine)?

For example does the following occur:

Assumptions:

- Main OOD Server (192.168.100.1)
- Proxy Server running on Cluster (10.0.0.1)
- Compute Node (10.0.0.2)

When a user wants to access a Jupyter Notebook, does the following workflow occur:

- User launches an Jupyter Notebook on 192.168.100.1
- Scheduler spins up a Jupyter Notebook on a compute node
- User select “Connect to Jupyter” on 192.168.100.1
- User is redirected to 10.0.0.1 which proxies out to 10.0.0.2

Hopefully that makes more sense? But it still might be as clear as mud?

Basically what I am getting as if the the Proxy Server is firewalled off so external users can’t access it directly, does the proxy process still work? I assume it does not.

---

<div class="post-metadata">

**Author:** ![rsettlag](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rsettlag/32/78_2.png) [@rsettlag](https://discourse.openondemand.org/u/rsettlag)\
**Post date:** [May 27, 2021, 3:11pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/7 "2021-05-27T15:11:45Z")

</div>

I think the workflow is stated a little differently for clarity:

- 

```
User uses the Jupyter Notebook APP on 192.168.100.1 which submits a job to scheduler

```

- 

```
Scheduler spins up a Jupyter Notebook on a compute node

```

- 

```
User select “Connect to Jupyter” on 192.168.100.1

```

- 

```
User is redirected to 10.0.0.1 which proxies out to 10.0.0.2

```

So, now the question is what do users have access to.

if your users don’t have access to the internal proxy, they need a proxy from 192.168.100.1 to 10.0.0.1 which in turn proxies to 10.0.0.2. But it sounds like you can redirect.

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [September 30, 2021, 11:23pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/8 "2021-09-30T23:23:18Z")

</div>

As Nate mentioned, we are working to deploy OOD using the reverse proxy on a separate host because of security reasons. Our main OOD server is isolated from the compute nodes.

However so far, I’ve not been successful in enabling the Interactive Desktop using this configuration. I’m looking for advice regarding what OOD or BC Desktop files I need to change to make this happen.

Example:

Main OOD Server: [ood.example.org](http://ood.example.org)  
Proxy Server: [apache.example.org](http://apache.example.org) (just Apache is installed in here but it has network access to the compute node).  
Compute Node: cn01

I followed the advice of @rsettlag and generate the variable ONDEMAND\_HOST in the connection.yml file. However, apparently, the only effect this has is creating the following variables in the compute node environment:

ONDEMAND\_HOST=[apache.example.org](http://apache.example.org)  
SLURM\_EXPORT\_ENV=ONDEMAND\_HOST

When the Dashboard parses the connection.yml file, the generate link to connect to the running Desktop session still looks like:

[https://ood.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=rnode/cn01/61518/websockify&resize=downscale](https://ood.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=rnode/cn01/61518/websockify&resize=downscale)

Thus, of course, the reverse proxy fails because the OOD server cannot communicate with cn01.

How can I generate the right connection file so the link generated goes though the host running the Apache server (to perform the OOD reverse proxy) and then come back to the OOD host to be able to display the Desktop session through noVNC?

Is this even possible using the latest OOD release?

Thanks in advance for the support!

---

<div class="post-metadata">

**Author:** ![rsettlag](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rsettlag/32/78_2.png) [@rsettlag](https://discourse.openondemand.org/u/rsettlag)\
**Post date:** [October 1, 2021, 12:10pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/9 "2021-10-01T12:10:30Z")

</div>

If your link were to change to

[https://apache.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=rnode/cn01/61518/websockify&resize=downscale](https://apache.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=rnode/cn01/61518/websockify&resize=downscale)

Would your app connect?

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [October 1, 2021, 2:42pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/10 "2021-10-01T14:42:33Z")

</div>

No. That host only has Apache installed. Thus I got the error:

Not Found  
The requested URL /pun/sys/dashboard/noVNC-1.1.0/vnc.html was not found on this server.

Thanks!

---

<div class="post-metadata">

**Author:** ![mario](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mario/32/662_2.png) [@mario](https://discourse.openondemand.org/u/mario)\
**Post date:** [October 4, 2021, 7:19pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/11 "2021-10-04T19:19:01Z")

</div>

@HPCworks `noVNC-1.1.0/vnc.html` is all client side, noVNC’s `RFB.js` on the client is used to directly authenticate the connection from the query parameters. Try setting the `path` as the FQDN of the remote WebSockify server that’s running instead of a relative one.

---

<div class="post-metadata">

**Author:** ![rsettlag](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rsettlag/32/78_2.png) [@rsettlag](https://discourse.openondemand.org/u/rsettlag)\
**Post date:** [October 5, 2021, 9:22pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/12 "2021-10-05T21:22:50Z")

</div>

Mario, am I correct in that he needs WebSockify on the server he has Apache on if that ultimately the proxy server? The OOD server will simply redirect to that server via the view button. Perhaps I am totally off on the scenario.

---

<div class="post-metadata">

**Author:** ![mario](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mario/32/662_2.png) [@mario](https://discourse.openondemand.org/u/mario)\
**Post date:** [October 7, 2021, 2:14pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/13 "2021-10-07T14:14:48Z")

</div>

@HPCworks @rsettlag Not quite, if the ultimate goal is to run WebSockify on another server then WebSockify is the only thing that needs to be reachable. OOD will redirect to wherever `novnc_link` references:

> **[sessions\_helper.rb?L179-184](https://sourcegraph.com/github.com/OSC/ondemand/-/blob/apps/dashboard/app/helpers/batch_connect/sessions_helper.rb?L179-184)**
>
> https://sourcegraph.com/github.com/OSC/ondemand

Take this for example, `path` is set to `rnode/cn01/61518/websockify` then when the client is redirected to `noVNC-1.1.0/vnc.html` and then the clients browser will take over completely from there and make a WebSocket connection to `rnode/cn01/61518/websockify` from the client itself.

`rnode/cn01/61518/websockify` is a relative path so if our domain was localhost the FQDN is `https://localhost/rnode/cn01/61518/websockify`

The starts a WebSocket connection to `wss://localhost/rnode/cn01/61518/websockify` but that path can be anything, as long as it’s a valid WS endpoint running WebSockify.

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [October 7, 2021, 9:54pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/14 "2021-10-07T21:54:12Z")

</div>

Mario,

Thanks for your support and thanks to the OSC team for OOD, a fantastic HPC solution.

Our main roadblock to enable OOD interactive apps is that our OOD server is isolated from the compute nodes.

I followed your advice and even if I use the FQDN instead of a relative one the Desktop won’t display on the client browser as there’s NOT network connection between the OOD server and the compute node cn01. Thus, the WebSockify server cannot be reached (error: Failed to connect to server):

[https://ood.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=https://cn01.example.org/rnode/cn01/61518/websockify&resize=downscale](https://ood.example.org/pun/sys/dashboard/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=https://cn01.example.org/rnode/cn01/61518/websockify&resize=downscale)

However, we have configured an Apache server in a host that can access cn01. We configured this Apache server using the _ood-portal.conf_ file from our OOD server which contains the OOD reverse proxy instructions. However, this host doesn’t have OOD because that would defeat the purpose of our security requirements.

To try your advice, we installed noVNC in the host that runs Apache and after the Desktop Slurm Job starts at cn01, I tried to connect using a link like this:

[https://apache.example.org/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path= https://cn01.example.org/ rnode/cn01/61518/websockify&resize=downscale](https://apache.example.org/noVNC-1.1.0/vnc.html?autoconnect=true&password=xxxxxxx&path=%20https://cn01.example.org/%20rnode/cn01/61518/websockify&resize=downscale)

This time noVNC connects but instead of displaying the Mate Desktop running on the compute node, it presents a login shell for the OOD server:

_CentOS Linux 7 (Core)_  
_Kerner 3.10.0-1160.31.1.e17.centos.plus.x86\_64 on an x86\_64_

_ood login:_

Again, we’re looking for advice on what OOD core or batch connect YML files, we need to customize to enable this set up, **if at all possible**. For example, I know that the Dashboard parses the _connection.yml_ file to generate the Dashboard link to connect to the running Desktop session in the compute node. Also, that the parameters in the _connection.yml_ file can be set up in the _cluster.yml.erb_ “embedded Ruby” batch connect file.

Do your team at OSC has any _cluster.yml.erb_ example files with similar reverse proxy complex set ups?

Thanks!

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [October 8, 2021, 5:25pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/15 "2021-10-08T17:25:29Z")

</div>

Hi, I think I may have something, but I think we’re breaking new ground here. Which is to say, I don’t believe anyone’s ever done this.

(1) You want to set the host as your intermediary. This means every `connection.yml` will have your intermediary proxy as the host.

```yaml
set_host: "host=$(echo my.intermediary.proxy)"

```

You probably want to set this for the entire cluster instead of in every submit.yml. Here are docs for the same.  
[https://osc.github.io/ood-documentation/latest/reference/files/submit-yml-erb.html#setting-batch-connect-options-globally](https://osc.github.io/ood-documentation/latest/reference/files/submit-yml-erb.html#setting-batch-connect-options-globally)

(2) Then you’ll need to pass back the real host through `conn_params`. Here’s how that works. You should also set these to be cluster wide (as a pose to setting it for every single app)

> [@How does conn\_params work?](https://discourse.openondemand.org/t/how-does-conn-params-work/1711):
>
> I’m trying to sort out an issue with rstudio-server 1.4 authentication, which I posted about here: [RStudio when launched without Singularity is having strange troubles with authentication - #53 by griznog](https://discourse.openondemand.org/t/rstudio-when-launched-without-singularity-is-having-strange-troubles-with-authentication/1213/53) but in troubleshooting that I realized conn\_params isn’t working as I’d expect it to. What I expect is that if I set in the app config # submit.yml.erb conn\_params: - bubba then in template/before.sh.erb I add export bubba=BUBBAWASHERE that I can then use in view.html.erb \<input type="hi…

(3) Now at this point, your apps will know all the information they need, you need to use them in the form. I’d suggest starting with an app like Jupyter instead of desktop, just to defer the whole noVNC situation, because that may need additional stuff that I don’t have offhand.

Here’s a view.html.erb I took from our Rstudio (changing rnode to node). You see the URL we’re redirecting to has `host` which you’ve set as a static string in #1 and `the_real_host` you’ve configured in #2.

**I believe you’ll have to always use _/node_ as a post to _/rnode_. Our proxy will remove /rnode/host/port portions of the URL when sending to rnode (because it’s relative). You’ll need to keep these intact so your intermediary can then determine what to do/remove.**

```html
<form action="/node/<%= host %>/<%= the_real_host %>/<%= port %>/auth-do-sign-in" method="post" target="_blank">
  <input type="hidden" name="username" value="<%= ENV["USER"] %>">
  <input type="hidden" name="password" value="<%= password %>">
  <input type="hidden" name="staySignedIn" value="1">
  <input type="hidden" name="appUri" value="">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-registered"></i> Connect to RStudio Server
  </button>
</form>

```

(4) At this point we’re talking to your intermediary, we’re proxying to it. Now comes the hard bit because you likely have to modify the URL to fit what the app is expecting and do the actual proxying. I don’t believe you can just use another OOD installation as the URL has an extra host in it so it may throw off our parsing expectations.

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [October 8, 2021, 6:32pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/16 "2021-10-08T18:32:58Z")

</div>

Thanks Jeff! I will give this a shot and see if I can get it to work with Jupyter Notebooks.

We much appreciate your feedback.

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [October 20, 2021, 7:24pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/17 "2021-10-20T19:24:29Z")

</div>

Jeff and others,

We’re still working on trying to make this intermediary proxy set up functional for Jupyter Notebooks.

While working on this, I found another way to execute Jupyter Notebooks on our compute nodes (cn01) but it involves a couple of “manual” steps that I would like to eliminate, if possible.

I access one of our Cluster login nodes and run Firefox to access our OOD server. The login nodes have network access to our compute nodes. After Slurm start the Jupyter job, (1) I use the Dashboard Files App to open the _output.log_ and _connection.yml_ files to get the Web link and password for the Jupyter Server in the compute node. (2) I copy and paste the link in Firefox to access Jupyter and execute Notebooks.

The links at the _output.log_ are similar to this. But the port varies (between 2000 to 65535) for each Slurm job:

[http://cn01:54020/node/cn01/54020/](http://cn01:54020/node/cn01/54020/)

**Questions:**

Is it possible to modify the _view.html.erb_ file so it generates a connection button with just this URL, without including the preceding OOD URL?

Or perhaps modifying some other Batch Connect or OOD configuration files?

Remember that our OOD server doesn’t have network access to the compute nodes, thus the standard link to access the Jupyter Server will fail:

[https://ood.example.org/node/c01/54020/](https://ood.example.org/node/c01/54020/)

_Failed to connect to c01:54020_

We much appreciate your feedback!

---

<div class="post-metadata">

**Author:** ![HPCworks](https://avatars.discourse-cdn.com/v4/letter/h/9f8e36/32.png) [@HPCworks](https://discourse.openondemand.org/u/HPCworks)\
**Post date:** [October 21, 2021, 5:19pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/18 "2021-10-21T17:19:08Z")

</div>

All,  
I just solved my own question. The _view.html.erb_ for this case looks like this:

```auto
<form action="http://<%= host %>:<%= port %>/node/<%= host %>/<%= port %>/login" method="post" target="_blank">
  <input type="hidden" name="password" value="<%= password %>">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-eye"></i> Connect to Jupyter
  </button>
</form>

```

In addition, I need to secure the Jupyter Server using SSL/HTTPS so that the password is not sent unencrypted by the browser to the web server. Right now, I’m running on a private network but when running the notebook on the public internet this is absolutely required.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 5, 2021, 7:18pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/19 "2021-11-05T19:18:41Z")

</div>

Hi all - just to give an update to this thread - We’re aware of the security issues around the current proxy. Given time and resources, I’d love to replace it.

We’re tracking the work in this project here - and as you may see there’s not much movement on it.

> **[New Proxy · OSC/ondemand](https://github.com/OSC/ondemand/projects/11)**
>
> Supercomputing. Seamlessly. Open, Interactive HPC Via the Web - New Proxy · OSC/ondemand

Here’s our thinking. First We need to document the current problem space.

Then we need to explore some proxy solutions. Another apache on another instance may be in this.

We’re thinking about writing an apache module to give that layer a bit of authorization where only I can get proxied to _my_ sessions, and I can’t even _see_ others. This is the main problem we need to solve. Then we can start to solve SSL termination to the compute nodes and more.

We’re open to looking at existing proxies but the administrative overhead of an Envoy or similar seems like a lot, perhaps too much especially for small sites.

Hope that helps! Let me know what sort of progress you make on this.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [December 6, 2021, 8:58pm UTC](https://discourse.openondemand.org/t/reverse-proxy-on-separate-host/1499/20 "2021-12-06T20:58:41Z")

</div>

Glad to hear you all set up Jupyter, at least a demonstration of what/how this works.

I know we have issues with `/rnode` URLs that I don’t have a solution for off the top, but I’ll keep thinking on it and hacking. I’ll let you know what I come up with.

Re-reading that comment I had said this. So we’ll see how to hack `/node` to get this to work if it can.

> [@jeff.ohrstrom](#):
>
> I believe you’ll have to always use _/node_ as a post to _/rnode_ . Our proxy will remove /rnode/host/port portions of the URL when sending to rnode (because it’s relative). You’ll need to keep these intact so your intermediary can then determine what to do/remove.