Putting an entire OOD setup behind a web proxy?

Get Help
1

I seem to have my test setup of OnDemand 1.6 working quite well; since we’re extremely limited in routable IP addresses, I have been using ‘redir’ to just redirect connections from a high port on the head node to the (otherwise non-routable) system OOD is installed on with good success.

However, I realized that every logged connection in OOD has the address of the proxying server and not the actual origin. If I’m going to open OOD to internet logins, I expect I’ll need to use fail2ban to keep brute force attacks to a minimum and need to log the actual source address.

Is anyone successfully using a web proxy to do this with apache or nginx? If so, could you possibly share the configuration you’re using to do so?

Thank you.

2

I gave setting up a proxy from scratch a shot. I tried with apache 2.2 first, which seemed to work until I tried websocket apps; it’s too old to include support for websockets; however it does appear that the mod_proxy_wstunnel module resolves that problem for apache 2.4, and it apparently is backported to 2.2.

Since I had nginx 1.14.1 available, I gave that a try, and it appears to work nicely for proxying to Open OnDemand so far. My configuration was a modified version of what I had used when testing JupyterLab, so it may be less than ideal:

user nginx;
worker_processes  1;
error_log  logs/error.log  info;
events {
    worker_connections  1024;
}
http {
    include       mime.types;
    include /nginx/1.14.1/sites-available/*.conf;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
    }
server {
        listen x.y.z.a:8443 ssl;
        ssl on;
        server_name www.my.org;
	ssl_certificate /etc/pki/tls/certs/www.my.org_2019.crt;
	ssl_certificate_key /etc/pki/tls/private/www.my.org_2019.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_session_timeout 1d;
        ssl_stapling on;
        ssl_stapling_verify on;
        add_header Strict-Transport-Security max-age=15768000;
	location / {
            proxy_pass https://10.10.10.21:443;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
	}
    }
}

I believe all I need to do now is figure out how to properly set the Apache LogFormat via update_ood_portal to include the X-Forwarded-For (and perhaps related) headers to show the original source address. Then it’s a matter of configuring fail2ban.

1 Like
3

Thanks for updating the topic with what you’ve done! We at OSC don’t run OOD behind a load balancer/proxy/VIP so we’re very interested in folks who do, and there have been several topics opened asking us how to do it.

Let us know how it turns out, what updates you make and how well it works for you! If you save the config or deployment scripts on Github let us know.

4

Chris - it sounds like you are set with Nginx. We have just started testing with Proxy/ReverseProxy for Apache. For Apache 2.4 you need to add some proxy rules for web sockets. There are some useful tips here https://www.happyassassin.net/2018/11/23/reverse-proxying-websockets-with-apache-a-generic-approach-that-works-even-with-firefox/ . We follow this recipe and have updated our mod_proxy configuration from

Before

     ProxyPreserveHost On
     RewriteEngine on
     ProxyPass        "/" "http://localhost:2080/"
     ProxyPassReverse "/" "http://localhost:2080/"
     ProxyRequests off

to After

     ProxyPreserveHost On
     RewriteEngine on
     RewriteCond %{HTTP:Upgrade} websocket [NC]
     RewriteCond %{HTTP:Connection} upgrade [NC]
     RewriteRule .* "ws://localhost:2080%{REQUEST_URI}" [P]
     ProxyPass        "/" "http://localhost:2080/"
     ProxyPassReverse "/" "http://localhost:2080/"
     ProxyRequests off

At moment the proxy is just against a Docker container with OOD on same host. We have not tried combining with a Proxy balancer yet.

Chris

5

At my site we are in the progress of setting up a proxy for ondemand with haproxy. It works very well for now, with just a small addition to ood-portal-generator (https://github.com/OSC/ondemand/pull/74)

if anyone is interested I can share parts of our config.

1 Like
6

We’re very interested @wdpypere. I’d suggest open sourcing some templated configs in ruby’s ERB or python jinja2, that way you don’t have to show anything that’s secret while still showing the entire configuration file(s).

7

we have a setup with oidc, 2 haproxy servers (although it also works with ony one) and several ood backend servers.

With version 1.6.19 we set in /etc/ood/config/ood_portal.yml

servername: <ood_servername>
proxy_server: <haproxy_public_name>
logout_redirect: /oidc?logout=https%3A%2F%2F<haproxy_public_name>

in /opt/rh/httpd24/root/etc/httpd/conf.modules.d/auth_openidc.conf we set:

OIDCRedirectURI https://<haproxy_public_name>/oidc

for the haproxy setup we have specifically for ood (not including ssl settings etc):

frontend ood-in
    bind <haproxy_server_public_ip>:443 crt <certificate_file> ssl
    mode http
    default_backend ood-bk

backend ood-bk
    balance roundrobin
    mode http
    option forwardfor
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server <ood_backend_server1> <IP>:443 ca-file <cafile> check ssl
    server <ood_backend_server2> <IP>:443 ca-file <cafile> check ssl
    server <ood_backend_server3> <IP>:443 ca-file <cafile> check ssl
    stick-table type ip size 1m peers mypeers
    stick on src

peers mypeers
    peer <haproxy_server_1> <IP>:<port>
    peer <haproxy_server_2> <IP>:<port>
8

Awesome, this topic now seems to have proxy configs for Apache httpd, Nginx and HAProxy. Thanks so much! We’d love to see more like github repos or blog posts about how to set this up, what your architecture looks like, challenges, failure modes, etc. Whatever you want to share, we’d love to see it.

9

When using Haproxy as LoadBalancer, certain applications like “shell” timeout with the message “your connection to remote server has been terminated” with-in a minute of inactivity

This is due to the default timeout configured for websockets in haproxy is 50s

you can increase the timeout by adding the following to your backend config in /etc/haproxy/haproxy.cfg file

timeout tunnel 7200s
10

what you did to get proper values for:

  • authorization_endpoint
  • issuer
  • etc.

in https://{hostname}:5554/.well-known/openid-configuration ?

It seems to me it is generated from field issuer defined in dex/config.yaml beeing autogenerated from FQDN of dex portal host. Instead of that, it should be, IMO, the FQDN of reverse proxy.

or, did I miss something ?

EDIT: I believe the possibility to configure that is missing, and it could be possibly usable, so I created ticket to discuss that topic: field "issuer" in generated dex config should be configurable · Issue #1101 · OSC/ondemand · GitHub .

11

@jose-d yes thank you so much for the issue on Github! I think you’re right, if proxy_server is set we should try to use it in the issuer.