Simple Portal Auth from Header (Provided by reverse Proxy) [SOLVED]

Hi,

I’m currently in the beginning stages of bringing OOD to our campus-wide multi-cluster HPC service at the University of Washington. Cheers on a great product and an active community.

The OOD deployment will live in pods on our local kubernetes cluster. I’ve borrowed/extended an flask/nginx reverse proxy for other user-facing services which acts as a saml IDP, and can inject the username of an authenticated user as a header down to the service which it forwards traffic to.

Several of our services, most notably grafana and jenkins are able to consume this arbitrarily named header and create/auth users by trusting it.

Examples here:
Grafana: https://grafana.com/docs/auth/auth-proxy/
Jenkins: https://plugins.jenkins.io/reverse-proxy-auth-plugin

My question is as follows: would an existing mod_auth_* module allow for the same behavior, which could be used in the ood_portal.yaml config generator? And if not, how difficult would it be to implement that sort of behavior?

Thanks reading,

Felix

If the header “X-WEBAUTH-USER” is already being set to the username by your proxy then it should just be a matter of properly configuring Apache in OnDemand. I don’t think you need any existing mod_auth_*. My guess the ood_portal.yaml would get this option set:

user_env: "X_WEBAUTH_USER"

The reason for the underscores instead of dashes is in this document: https://httpd.apache.org/docs/trunk/env.html

You would of course have to unfortunately regenerate the Apache config after changing this option.

This essentially says, instead of using REMOTE_USER for the argument to the mapping command, use X_WEBAUTH_USER. REMOTE_USER is what the mod_auth_* modules typically set after a user is authenticated.

I’ll be out Fri and Mon but happy to help more with this next week if you run into issues.

1 Like

@felix-russell have you had any success with the proposed solution?

Hey @efranz thanks for the response last week!

Sorry for getting back to you so late. We were still in the middle of setting up jenkins on kubernetes, our registries, etc.

So far I’ve done the following:

  1. generated the ood-portal.conf using only two options in the portal generator: Namely:
    servername: ‘test.test.washington.edu’
    user_env: ‘X_WEBAUTH_USER’

  2. Modified the resulting ood-portal.conf by

    1. Turning off the RewriteEngine (ood portal generator’s rewriterule tried to redirect:301 a http request back through our kubeproxy -> samlproxy -> ood-portal on an infinite loop
    2. Changing RedirectMatch:
      From: RedirectMatch ^/ "/pun/sys/dashboard" To: RedirectMatch ^/http://test.test.washington.edu/pun/sys/dashboard
    3. Upping the log verbosity of the VirtualHost to: LogLevel Trace8
    4. Changing Error and CustomLog to output to stdout so our docker logging facility would pick them up with: ErrorLog “/dev/stdout” and ErrorLog “/dev/stdout”

What’s great is that our redirects are all bringing us to the dashboard, but the htpasswd prompt still asks for a basicauth.

Here are the relevant locations in the ood-portal.conf that contain the old htpasswd stuff…

SetEnv OOD_PUN_URI "/pun"
  <Location "/pun">
    AuthType Basic
    AuthName "Private"
    AuthUserFile "/opt/rh/httpd24/root/etc/httpd/.htpasswd"
    RequestHeader unset Authorization
    Require valid-user

    ProxyPassReverse "http://localhost/pun"

    # ProxyPassReverseCookieDomain implementation (strip domain)
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation (less restrictive)
    Header edit* Set-Cookie ";\s*(?i)Path\s*=(?-i)(?!\s*/pun)[^;]*" "; Path=/pun"

    SetEnv OOD_PUN_SOCKET_ROOT "/var/run/ondemand-nginx"
    SetEnv OOD_PUN_MAX_RETRIES "5"
    LuaHookFixups pun_proxy.lua pun_proxy_handler

  </Location>

And

  SetEnv OOD_NGINX_URI "/nginx"
  <Location "/nginx">
    AuthType Basic
    AuthName "Private"
    AuthUserFile "/opt/rh/httpd24/root/etc/httpd/.htpasswd"
    RequestHeader unset Authorization
    Require valid-user

    LuaHookFixups nginx.lua nginx_handler
  </Location>

I’m wondering which of these I should be getting rid of to allow the pun dashboard app to pick up on the X_WEBAUTH_USER, I can see from my trace logs that http_request.c is seeing header passed into the portal, I’m a little lost where to go from here.

Thanks again!

Felix

@efranz
I recently attempted to replace the following stanzas in both the /nginx and the /pun locations from the above configs:

AuthType Basic
AuthName “Private”
AuthUserFile “/opt/rh/httpd24/root/etc/httpd/.htpasswd”
RequestHeader unset Authorization
Require valid-user

with:

AuthType None
Require all granted

This didn’t work so well and caused an internal server error as far as what was readable through the returned HTML, I’m going to try and dive through the nginx,passenger,and apache portal logs to see where its unwinding itself, but if you know of a way to just do blind-trusted auth from the header, to the PUN apps, it would be massively appreciated.

Cheers,
Felix

Did you try removing all the auth directives? That might work.

But then that introduces the problem of requests made to Apache from the host running Apache itself. We typically treat the OnDemand host like a login node and assume users are provided shell access to that node, so extra security measures are taken as a result.

So it would seem like you would want some type of trust setup between Apache and the authenticating proxy so that Apache doesn’t accept requests from any other source. I’m not an expert in this part of Apache. @tdockendorf might have some ideas.

This is an example of where it would be beneficial if Apache were option in OnDemand, and the main server component that handled starting the PUNs could listen on a Unix socket.

@efranz @tdockendorf since Our auth proxy (That’s also handling 2fa) and the ood container are all running inside the same kubernetes namespace with NetworkSecurityPolicies enabled, and no end-user shell access we’re comfortable implicitly/blindly trusting the auth header.

Update from removing the auth header references:
I removed the following from the /nginx and /pun locations on the virtualhost:

    # AuthType Basic
    # AuthName "Private"
    # AuthUserFile "/opt/rh/httpd24/root/etc/httpd/.htpasswd"
    # RequestHeader unset Authorization
    # Require valid-user

I’m getting a code 500 server internal error. I have a strong feeling that the lua handler is translating the output from the mod_auth_* (whichever one is used, like htpasswd, ldap, shib, etc…) and passing it as a variable down to the PUN (passenger?) process. I’m going to have a look at the lua code at:
/opt/ood/mod_ood_proxy/lib/pun_proxy.lua, func: pun_proxy_handler and see if I can substitute the header string contents into the variable that the downstream PUN is consuming… If you guys can think of anything else it would be appreciated.

@efranz @tdockendorf
Figured it out, had to hack on this file:

Replaced the function with this

function map(r, user_map_cmd, remote_user)
  -- run user_map_cmd and read in stdout
  local now = r:clock()
  -- local handle = io.popen(user_map_cmd .. " '" .. r:escape(remote_user) .. "'")
  sys_user = r.subprocess_env["PROX_MAPPED_USER"]
  -- handle:close()
  time_user_map = (r:clock() - now)/1000.0
  -- r:debug("Mapped '" .. remote_user .. "' => '" .. (sys_user or "") .. "' [" .. time_user_map .. " ms]")

  -- failed to map if returns empty string
  if not sys_user or sys_user == "" then
    return nil
  end

  r.subprocess_env['MAPPED_USER'] = sys_user -- set as CGI variable for later hooks (i.e., analytics)
  r.subprocess_env['OOD_TIME_USER_MAP'] = time_user_map -- set as CGI variable for later hooks (i.e., analytics)
  return sys_user
end

return {
  map = map
}

and on my ood-portal.conf I had to add the following SetEnvIf:

SetEnv OOD_USER_ENV "X_WEBAUTH_USER"
SetEnvIf X_WEBAUTH_USER "(.*)" PROX_MAPPED_USER=$1

PUNS are now launching without complaint. Thanks for gently steering me in the right direction. Super excited to make more progress now…

Will likely fork upstream and make my changes more clean, and try to merge the changes in with some documentation for other users who want to follow the same path.

I wonder, if you did this instead, would the original Lua script work unmodified:

SetEnv OOD_USER_ENV "PROX_MAPPED_USER"
SetEnvIf X_WEBAUTH_USER "(.*)" PROX_MAPPED_USER=$1