Hi,
I am working on adjusting the Jupyter app to our cluster, the app is currently in Sandbox.
When I click the button “Connect to Jupyter” it opens a new tab with Jupyter asking for password,
if I return and click again it opens Jupyter’s dashboard as it should.
I wonder what is the cause of not having Jupyter dashboard displayed from the first click
Thank you
Hi and welcome. The application isn’t going to be ready until the ~/ondemand/data/sys/dashboard/batch_connect/sys/bc_osc_jupyter/output/$UUID/connection.yml is created (this path may differ becuase we call our app bc_osc_juypter, you may call it something else). Before that it’s in a staring state. This file has your password in it. Or should anyhow.
I also wonder what’s going on.
Can you try this for me?
connection.yml file for that session. If there is no password, check this file again when you return and open the 2nd time.Hello Jeff Ohrstrom,
Thank you for answer and thank you for the wonderful tool Open OnDemand.
I followed your instructions, this is what I have:
{I have to cut a lot f output, you have restrictions to links on this forum, and It seems that is capturing a lot of my output as links}
I ask for a new instance of Jupyter Notebook and I got this job in our Torque/Moab system
And this Session ID from OOD
Session ID: 8ddcda60-91bf-4264-828e-3c6e3e30aff1
I did not click on “Connect to Jupyter”
Now, on the file system, I went to:
~/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/8ddcda60-91bf-4264-828e-3c6e3e30aff1
The Working Directory for the job and consistent with the address you mentioned. The file connection.yml is there and its contents are:
host: tcocs001.hpc.wvu.edu
port: 25339
password: PxMbNrGkj41wPO3h
And the config.py contains
c.NotebookApp.ip = ‘’
c.NotebookApp.port = 25339
c.NotebookApp.port_retries = 0
c.NotebookApp.password = u’sha1:hNRTRQkWHbr8Cdvz:ec67bafc67b49a23472386325b537afcd2c22b7a’
c.NotebookApp.base_url = ‘/node/tcocs001.hpc.wvu.edu/25339/’
c.NotebookApp.open_browser = False
c.NotebookApp.allow_origin = '’
c.NotebookApp.notebook_dir = ‘/users/gufranco’
c.NotebookApp.disable_check_xsrf = True
I have no way to check the password because I am loosing the SALT but I imagine the lines on before.sh are doing their job
SALT="(create_passwd 16)"
password="(create_passwd 16)"
PASSWORD_SHA1="(echo -n "{password}${SALT}" | openssl dgst -sha1 | awk ‘{print $NF}’)"
Now I go back and click on “Connect to Jupyter” I got the page asking for password
I inspect the sources for that page close to Password and contains:
<div class="center-nav">
<form action="/node/tcocs001.hpc.wvu.edu/25339/logiXXX" method="post" class="navbar-form pull-left">
<input type="hidden" name="_xsrf" value="2|ff030bad|024e132d8039e5ec72e01f690bc292fe|1571670441"/>
<label for="password_input"><strong>Password:</strong></label>
<input type="password" name="password" id="password_input" class="form-control">
<button type="submit" class="btn btn-default" id="login_submit">Log in</button>
</form>
</div>
The file connection.yml has not changed after the first click
Now I click a second time and it opens the Jupyter Dashboard.
An inspection to the source code on (I have to replace tt with xx because I am not allowed to post links)
hxxps://ondemand-tf.hpc.wvu.edu/pun/sys/dashboard/batch_connect/sessions
Reveals the following code close to “Connect to Jupyter”
Connect to JupyterI see the password in that div section
And the value matches
connection.yml
host: tcocs001.hpc.wvu.edu
port: 25339
password: PxMbNrGkj41wPO3h
Still a mistery why I have to click twice to get the Jupyter Dashboard
Thank you very much, just to complete the context here are a few details of our system
RedHat 7.6
$ openssl version
OpenSSL 1.1.1a 20 Nov 2018
I am testing this from Firefox and Chrome, no difference from which browser you try to connect,
Thank you
On my previous message part of the HTML got caught by your forum HTML rendering, I am now correctly making it verbatim,
This is the content of the OOD sessions page:
<div id="8ddcda60-91bf-4264-828e-3c6e3e30aff1" class="panel panel-success session-panel" data hash="ce5f9645c16e9644cc9e8d2acfddc1e1"><div class="panel-heading"><h1 class="panel-title">
<a href="/pun/sys/dashboard/batch_connect/dev/jupyter/session_contexts/new"><strong>Jupyter Notebook</strong>
</a> (120375.trcis002.hpc.wvu.edu)<div class="pull-right"><span class="badge">1 node</span> |
<span class="badge">1 core</span> | Running</div><div class="clearfix"></div>
</h1></div><div class="panel-body">
<div><div><a class="btn btn-danger pull-right btn-delete" title="Delete Session" data-confirm="Are you sure?" data-toggle="tooltip" data-placement="bottom" rel="nofollow" data-method="delete" href="/pun/sys/dashboard/batch_connect/sessions/8ddcda60-91bf-4264-828e-3c6e3e30aff1">
<i class="fas fa-trash-alt fa-fw"></i> Delete</a></div>
<p><strong>Host:</strong> <a target="_blank" class="btn btn-primary btn-sm fas fa-terminal" href="/pun/sys/shell/ssh/tcocs001.hpc.wvu.edu">tcocs001.hpc.wvu.edu</a></p><p><strong>Created at:</strong> 2019-10-21 10:51:58 EDT</p><p>
<strong>Time Remaining:</strong> 18 minutes</p><p>
<strong>Session ID:</strong> <a target="_blank" href="/pun/sys/files/fs/users/gufranco/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/8ddcda60-91bf-4264-828e-3c6e3e30aff1">8ddcda60-91bf-4264-828e-3c6e3e30aff1</a></p></div><div>
<hr><div class="ood-appkit markdown"><form action="/node/tcocs001.hpc.wvu.edu/25339/login" method="post" target="_blank">
<input type="hidden" name="password" value="PxMbNrGkj41wPO3h">
<button class="btn btn-primary" type="submit">
<i class="fa fa-eye"></i> Connect to Jupyter
</button>
</form>
</div>
</div></div></div>The second bit of html you’ve sent is good, however I’d imagine it was taken after you successfully connected.
I was able to replicate this issue by simply editing the html and sending the wrong password.
This was more what I was looking for with dev-tools. What password you’re sending that very first time. This API call I was able to trigger a 401 unauthorized. If you’re fast enough you can grab the same bit of html you’ve sent in your second post before it updates. Otherwise you’ll be able to see what you pass in the network tab as I’ve shown here.
In any case, that’s what we need to find out, what do you pass as the password the very first time (if you pass anything at all).
The UI re-reads that connection file every 30 seconds or so. I wonder if it’s just that first read that’s bad? I mean if you just waited a little bit after booting if it would work. Not necessarily the the second time you connect, but on the second read of connection.yml?
Unfortunately turning on debug logs doesn’t really provide anything. I’m able to see in the logs the 401 is returned, but it doesn’t say to whom or why.
I cannot believe it doesn’t read the connection.yml correctly the first time, but then again, that’s the only explanation I have at the moment.
I am doing this all over again, this is the HTML from the page before clicking on “Connect to Jupyter”
<hr><div class="ood-appkit markdown"><form action="/node/tcocs001.hpc.wvu.edu/58216/login" method="post" target="_blank">
<input type="hidden" name="password" value="73yAVrlG96uOIi26">
<button class="btn btn-primary" type="submit">
<i class="fa fa-eye"></i> Connect to Jupyter
</button>
</form>
</div>
I notice the line with the password -> value="73yAVrlG96uOIi26" already on that message.
I am taking time to write this, and I have not clicked yet on “Connect to Jupyter”
The file connection.yml is there::
host: tcocs001.hpc.wvu.edu
port: 58216
password: 73yAVrlG96uOIi26
Now I will click on “Connect to Jupyter” (around 10 min after getting job start running)
and monitoring the network…
Sorry I am not fast enough and I cannot capture when Firefox opens the new tab but it is asking for password again!
I will ask around me here, my knowledge about web development is ground zero.
I will try to capture the message from the web server
Thank you
It’s no problem at all. You have to use Chrome because it has the option to auto load on new tab (at the very bottom of the settings), you’re not going to be able to capture it in FireFox by hand any other way.
Here’s an expansion of the image. The network tab at the top, the login api in the middle and the form data at the bottom (note the scroll bar on the right so it’s the very very bottom of that pane).
Thank you helping me with this.
I will be doing that, but just for now, let me point to an small difference in your case and my case.
In your case you have a message “Invalid Credentials” something that I dont have, I hast have a just the box asking for password.
Another thing to explore is using a different Python and Jupyter version. I am using 3.7.4
And jupyter version:
$ jupyter --version
jupyter core : 4.5.0
jupyter-notebook : 6.0.1
qtconsole : 4.5.5
ipython : 7.8.0
ipykernel : 5.1.2
jupyter client : 5.3.2
jupyter lab : not installed
nbconvert : 5.6.0
ipywidgets : 7.5.1
nbformat : 4.4.0
traitlets : 4.3.2
We will be working from several fronts to figure out what is wrong here.
Thank you
@jeff.ohrstrom I worked with Guillermo on this and was able to capture this in chrome. The password is available in the header. See below image as an example and thanks again for all the help!
Nate
OK forget the authentication bit. It looks like you’re correctly passing auth in the first request.
Let’s now assume it’s a problem with redirection and maybe setting and passing cookies. Sorry it took this long to get here.
Here is what a correct redirection flow is. Notice the status codes received from login and 55808 are 302 Found.
Seems that your 302 redirects are not working the first time?
Here’s what my very first 302 redirect response looks like. See the locaion is relative (yours has your SSO host in it) and it’s from a different Server even. You’re being redirected by your SSO or OOD (Apache 2.4.34 seems like it’s ours) whereas I’m being redirected by Juypter itself (TornadoServer is what Jupyter uses)
I don’t know much about CAS, but maybe you need to tweak your CASScope? I’ve seen in other threads folks need to set that to /.
Thanks @jeff.ohrstrom for the help! The CASScope indeed fixed the issue. Sorry for all the questions lately but we are making good progress thanks to you and your team! I never expected the proxy to be the issue since it worked on the second attempt but I was definitely wrong on that one.
Awesome! I’m glad to get it fixed.
And feel free to ask any questions you like. You just helped the next person that’s may have this same issue.
Hello,
sorry to bother you, but did you set your CASScope to /? I’m having the same problem.
This is my setup:
<LocationMatch "^/node/(?<host>[\w.-]+\.example\.edu)/(?<port>\d+)">
AuthType CAS
Require valid-user
RequestHeader unset Authorization
CASScope /.
# ProxyPassReverse implementation
Header edit Location "^[^/]+//[^/]+" ""
# ProxyPassReverseCookieDomain implemenation
Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""
# ProxyPassReverseCookiePath implementation
Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"
LuaHookFixups node_proxy.lua node_proxy_handler
</LocationMatch>
Thank you!
No bother at all! Hey sorry, the period was just to end the sentence. The CASScope should just be / with nothing else.
Thank you! Not sure what happened, but I didn’t have a period in ood_portal.yml. I double checked everything and restarted apache. The error is gone! Thank you!