I am trying to set up Jupyter notebook with ondemand and I have some issues with URL:
ondemand routes to
http:///node/compute-x-x/27006/login
which is not found
when checking the logs from the running job I see that the correct path should be:
http://compute-x-x:27006/node/compute-x-x/27006/login
which routes to the jupyter notebook but asks for a password?
How should I go about fixing the URL routing
and the password request?
You probably need to configure youâre reverse proxy. The password business will be taken care of by OOD.
In your /etc/ood/config/ood_portal.yml youâll need these configs.
# this regex you should probably change to be more specific
host_regex: "[^/]+"
# these should be fine for you
node_uri: "/node"
rnode_uri: "/rnode"
Thanks for the quick response/
Any suggestions on how the host_regex should be?
our hosts are set with compute-x-x and gpu-x-x convention
we can add the <my_center.edu> part to the hostname if needed.
Right now it looks like this:
host_regex: â[\w.-]+.mycenter.eduâ
node_uri: â/nodeâ
rnode_uri: â/rnodeâ
Ok I was able to get that part fixed with adding the mycenter extension to the hostnames.
Whatâs remaining is the password prompt which says âinvalidâ
First it goes to login authentication page again and then âinvalidâ password for jupyter.
Any thoughts?
First are you running CAS authentication? If so itâs an issue of your CASScope. Even if you arenât Hereâs a very lengthy thread on how to debug this issue.
We donât use CAS. authentication.
Will do, thanksâŚ
It all seems like that the password from jupyter is not correctly passed into the instance through ondemand
Bummer, that would have been a quick fix. In any case, that topic has a lot of debugging steps that I would just replicate here. If you do look through it and still canât find what youâre looking for update this topic with information and weâll look into it more.
Things weâd want to know are
view.html.erb looks like (if itâs different from ours).connection.yml file (the directory for a given session is described in the other topic).Looking at that topic, one obvious difference in our case is that,
when we click the âconnect to Jupyterâ in the Sandbox environment, it prompts to the ondemand web authentications which users have already done when they logged into the website.
Then we pass that authentication, and then âjupyter passwordâ webpage appears with a message âinvalid authenticationâ
For the sake of testing, I tried to copy the password that is generated in
âconnection.ymlâ file under <user_path>/jupyter-notebook/output/<jupyter_instance>
and when enter that, a blank page appears. It seems the same with entering any other password.
OK yea somethingâs very wrong with that. You should not have to authenticate twice. Youâre going to need to use Chromeâs web dev tools to âopen on new tabâ and see how youâre being redirected once you click âconnect to jupyterâ.
Iâm not sure why/how youâre being redirected like that. What authentication method do you use. And are you being redirected to a different host? like sandbox.myorg -> production.myorg?
Here is what I see when clicking the âconnect to Jupyterâ:
This is the authentication pop-up that appears and I try to cancel â
The override.css looks like
<path_to_jupyter>/static/auth/css/override.cssâŚ
Is that the CAS that you were referring to??
CAS is Central Authentication Service. I donât know much about it, but itâs a single sign on that some sites configure their apache for user login.
Can you take a second look at that thread Iâve linked. Specifically the 8th and 9th comments.
What does this screen shot look like for you? Specifically, the response headers (theyâre a little cut off in this image) of the very first login request.
My response headers look like this. The big difference here is the âLocationâ. It should be relative (thereâs no hostname in the location, itâs only the path) and from Jupyter itself (the âserverâ is TornadoServer, thatâs Jupyter).
How youâre being redirected seems very off to me, especially if you need to re-login.
Hi Jeff,
Thanks for bearing with us as we investigate the issue. We use LDAP for authentication not CAS.
I will revisit those steps and provide the web screenshot you requested. In the meantime, can you remind me what path should we expect as the output of each jupyter launch?
For some partitions, I get
"http://compute-x-x.<mycenter.edu>:16863/node/compute-x-x.<mycenter.edu>/16863/
which seems to be wrong?
Any hints on what to check to get that part fixed are appreciated
Thatâs the right path, but the wrong host. https://<OOD HOST>/node/compute-x-x.<mycenter.edu>/16863/ should be the host/path.
If you look at our view.html.erb (the box that has the âconnect to jupyterâ button) the action is relative meaning you should still land on the open ondemand host and not be redirected directly to the compute node, we (OOD) should be proxying to the compute node for you.
Thanks for the information. Strangely, our nodes still directly use the compute node rather than the OOD HOST.
. Is there another config file that we should look for to route this correctly?
"
user@host$ cat view.html.erb <form action="/node/<%= host %>/<%= port %>/login" method="post" target="_blank"> <input type="hidden" name="password" value="<%= password %>"> <button class="btn btn-primary" type="submit"> <i class="fa fa-eye"></i> Connect to Jupyter </button> </form>
"
In your /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf what does this LocationMatch look like?
<LocationMatch "^/node/(?<host>[\w.-]+\.osc\.edu)/(?<port>\d+)">
AuthType openid-connect
Require valid-user
# ProxyPassReverse implementation
Header edit Location "^[^/]+//[^/]+" ""
# ProxyPassReverseCookieDomain implemenation
Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""
# ProxyPassReverseCookiePath implementation
Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"
LuaHookFixups node_proxy.lua node_proxy_handler
</LocationMatch>
user@host$ cat /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf
<LocationMatch "^/node/(?<host>[\w.-]+\.my.center\.edu)/(?<port>\d+)">
AuthType Basic
AuthName "private"
AuthBasicProvider ldap
AuthLDAPURL "ldaps://ldaps.my.center.edu:portnumber/OU=users,DC=dcname,DC=orgname?name"
AuthLDAPGroupAttribute cn
AuthLDAPGroupAttributeIsDN on
Require valid-user
AuthLDAPBindDN "cn=cnname,ou=users,dc=dcname,dc=orgname"
AuthLDAPBindPassword ***
# ProxyPassReverse implementation
Header edit Location "^[^/]+//[^/]+" ""
# ProxyPassReverseCookieDomain implemenation
Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""
# ProxyPassReverseCookiePath implementation
Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"
LuaHookFixups node_proxy.lua node_proxy_handler
</LocationMatch>Maybe it could be an issue of rewrites?
In your /etc/ood/config/ood_portal.yml do you have this set (or left it to the default)?
use_rewrites: true
Honestly I have no idea why youâre being redirected like that. Youâre location - this bit here Header edit Location "^[^/]+//[^/]+" "" - doesnât seem to be working.
That very first request you make to Jupyter should have a Location header, can you let me know what it is?
the use_rewrites is set to default.
See attached.
As mentioned, one difference with the other thread is that it routes to authentication again before getting to jupyter, i.e., after âconnect to Jupyterâ the authentication prompt window appears again. If I authenticate that, then I am prompted to the jupyter notebook with an âinvalid credentialsâ message. Then, attached is the screenshot you asked for.
As a side note,
I just see something in the jupyter output file from one of the launches:
/home/user/ondemand/data/sys/dashboard/batch_connect/dev/jupyter_notebook/output/jupyter-id/before.sh: line 38: openssl: command not found which means traffic wonât be encrypted. I donât think thatâs related but just wanted to bring that.
Just for sanity checks: here is the websockify installation
I have verified that websockify is installed and working on the COMPUTE node:
[host]# websockify --help
Usage:
websockify [options] [source_addr:]source_port [target_addr:target_port]
websockify [options] [source_addr:]source_port â WRAP_COMMAND_LINE
cat /etc/ood/config/clusters.d/mycluster.yml | grep websockify
export WEBSOCKIFY_CMD="//usr/bin/websockify"
As a sanity check, be sure youâre passing the password in the form data. Youâll see it in under the ârequest headersâ portion of your chrome dev tools. I donât think itâs a problem, but just to be sure.
Do you have a similar configuration as us?
I think specifically these items are important in this case.
c.NotebookApp.base_url = '/node/${host}/${port}/'
c.NotebookApp.allow_origin = '*'
c.NotebookApp.disable_check_xsrf = True
The prompt for re-authentication is strange and should not be happening. Is there anything else youâve added to your apache ood-portal.conf?
This problem is so out of left field that at this point Iâm willing to have a webex (or similar meeting technology) with you to help debug.
I have webex, so we could use that. Let me know if have time tomorrow or Friday.