# Jupyter notebook Issues: The requested URL was not found on this server + Multiple authentication + Invalid Jupyter password

**URL:** <https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777>\
**Category:** Get Help\
**Tags:** question\
**Created:** [March 12, 2020, 7:31pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777 "2020-03-12T19:31:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 7:31pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/1 "2020-03-12T19:31:55Z")

</div>

I am trying to set up Jupyter notebook with ondemand and I have some issues with URL:  
ondemand routes to  
http:///node/compute-x-x/27006/login  
which is not found

when checking the logs from the running job I see that the correct path should be:  
[http://compute-x-x:27006/node/compute-x-x/27006/login](http://compute-x-x:27006/node/compute-x-x/27006/login)

which routes to the jupyter notebook but asks for a password?

How should I go about fixing the URL routing  
and the password request?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 12, 2020, 7:48pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/2 "2020-03-12T19:48:59Z")

</div>

You probably need to [configure you’re reverse proxy](https://osc.github.io/ood-documentation/master/infrastructure/ood-portal-generator/configuration.html#configure-reverse-proxy). The password business will be taken care of by OOD.

In your `/etc/ood/config/ood_portal.yml` you’ll need these configs.

```auto
# this regex you should probably change to be more specific
host_regex: "[^/]+"
# these should be fine for you
node_uri: "/node"
rnode_uri: "/rnode"

```

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 8:06pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/3 "2020-03-12T20:06:05Z")

</div>

Thanks for the quick response/  
Any suggestions on how the host\_regex should be?  
our hosts are set with compute-x-x and gpu-x-x convention  
we can add the \<my\_center.edu\> part to the hostname if needed.  
Right now it looks like this:  
host\_regex: ‘[\w.-]+.mycenter.edu’  
node\_uri: ‘/node’  
rnode\_uri: ‘/rnode’

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 8:08pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/4 "2020-03-12T20:08:15Z")

</div>

Ok I was able to get that part fixed with adding the mycenter extension to the hostnames.  
What’s remaining is the password prompt which says “invalid”  
First it goes to login authentication page again and then “invalid” password for jupyter.  
Any thoughts?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 12, 2020, 8:27pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/5 "2020-03-12T20:27:13Z")

</div>

First are you running CAS authentication? If so it’s an issue of your CASScope. Even if you aren’t Here’s a very lengthy thread on how to debug this issue.

> [@Jupyter app ask for password first time, works second time](https://discourse.openondemand.org/t/jupyter-app-ask-for-password-first-time-works-second-time/557/3):
>
> Hello Jeff Ohrstrom, Thank you for answer and thank you for the wonderful tool Open OnDemand. I followed your instructions, this is what I have: {I have to cut a lot f output, you have restrictions to links on this forum, and It seems that is capturing a lot of my output as links} I ask for a new instance of Jupyter Notebook and I got this job in our Torque/Moab system Jupyter Notebook And this Session ID from OOD Session ID: 8ddcda60-91bf-4264-828e-3c6e3e30aff1 I did not click on “Conn…

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 8:40pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/6 "2020-03-12T20:40:15Z")

</div>

We don’t use CAS. authentication.  
Will do, thanks…  
It all seems like that the password from jupyter is not correctly passed into the instance through ondemand

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 12, 2020, 8:50pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/7 "2020-03-12T20:50:27Z")

</div>

Bummer, that would have been a quick fix. In any case, that topic has a lot of debugging steps that I would just replicate here. If you do look through it and still can’t find what you’re looking for update this topic with information and we’ll look into it more.

Things we’d want to know are

- what your `view.html.erb` looks like (if it’s different from [ours](https://github.com/OSC/bc_osc_jupyter/blob/master/view.html.erb)).
- if you’re creating a valid `connection.yml` file (the directory for a given session is described in the other topic).
- what you’re passing when you submit the form (what’s included in all those Firefox/Chrome dev tool images in the other topic)

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 9:15pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/8 "2020-03-12T21:15:33Z")

</div>

Looking at that topic, one obvious difference in our case is that,  
when we click the “connect to Jupyter” in the Sandbox environment, it prompts to the ondemand web authentications which users have already done when they logged into the website.  
Then we pass that authentication, and then “jupyter password” webpage appears with a message “invalid authentication”  
For the sake of testing, I tried to copy the password that is generated in  
“connection.yml” file under \<user\_path\>/jupyter-notebook/output/\<jupyter\_instance\>

and when enter that, a blank page appears. It seems the same with entering any other password.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 12, 2020, 9:23pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/9 "2020-03-12T21:23:46Z")

</div>

OK yea something’s very wrong with that. You should not have to authenticate twice. You’re going to need to use Chrome’s web dev tools to ‘open on new tab’ and see how you’re being redirected once you click ‘connect to jupyter’.

I’m not sure why/how you’re being redirected like that. What authentication method do you use. And are you being redirected to a different host? like sandbox.myorg -\> production.myorg?

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 12, 2020, 11:24pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/10 "2020-03-12T23:24:15Z")

</div>

Here is what I see when clicking the “connect to Jupyter”:  
This is the authentication pop-up that appears and I try to cancel –  
The override.css looks like  
\<path\_to\_jupyter\>/static/auth/css/override.css…  
Is that the CAS that you were referring to??

 ![Screen Shot 2020-03-12 at 7.19.03 PM](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/3b5e4b59eec442378e71a4df8ac5488311ac6371.png)

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 13, 2020, 1:49pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/11 "2020-03-13T13:49:03Z")

</div>

CAS is [Central Authentication Service](https://en.wikipedia.org/wiki/Central_Authentication_Service). I don’t know much about it, but it’s a single sign on that some sites configure their apache for user login.

Can you take a second look at that thread I’ve linked. Specifically the 8th and 9th comments.

What does this screen shot look like for you? Specifically, the response headers (they’re a little cut off in this image) of the _very first_ login request.

 ![image](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/668579af7187630d372075c3a2adfdaeede17ca8.png)

My response headers look like this. The big difference here is the ‘Location’. It should be relative (there’s no hostname in the location, it’s only the path) and from Jupyter itself (the ‘server’ is TornadoServer, that’s Jupyter).  
 ![image](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/0832fcda425e36a288f313446c78b4305ae16706.png)

How you’re being redirected seems very off to me, especially if you need to re-login.

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 16, 2020, 3:20pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/12 "2020-03-16T15:20:47Z")

</div>

Hi Jeff,

Thanks for bearing with us as we investigate the issue. We use LDAP for authentication not CAS.  
I will revisit those steps and provide the web screenshot you requested. In the meantime, can you remind me what path should we expect as the output of each jupyter launch?  
For some partitions, I get  
"http://compute-x-x.\<mycenter.edu\>:16863/node/compute-x-x.\<mycenter.edu\>/16863/  
which seems to be wrong?

Any hints on what to check to get that part fixed are appreciated

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 16, 2020, 3:34pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/13 "2020-03-16T15:34:32Z")

</div>

> [@arashnh11](#):
>
> "http://compute-x-x.\<mycenter.edu\>:16863/node/compute-x-x.\<mycenter.edu\>/16863/

That’s the right path, but the wrong host. `https://<OOD HOST>/node/compute-x-x.<mycenter.edu>/16863/` should be the host/path.

If you look at our [view.html.erb](https://github.com/OSC/bc_osc_jupyter/blob/master/view.html.erb) (the box that has the ‘connect to jupyter’ button) the `action` is relative meaning you should still land on the open ondemand host and not be redirected _directly_ to the compute node, we (OOD) should be proxying to the compute node for you.

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 16, 2020, 3:50pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/14 "2020-03-16T15:50:52Z")

</div>

Thanks for the information. Strangely, our nodes still directly use the compute node rather than the OOD HOST.  
. Is there another config file that we should look for to route this correctly?  
"

> ```
> user@host$ cat view.html.erb 
> <form action="/node/<%= host %>/<%= port %>/login" method="post" target="_blank">
> <input type="hidden" name="password" value="<%= password %>">
> <button class="btn btn-primary" type="submit">
> <i class="fa fa-eye"></i> Connect to Jupyter
> </button>
> </form>
> 
> ```

"

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 16, 2020, 3:59pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/15 "2020-03-16T15:59:47Z")

</div>

In your `/opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf` what does this LocationMatch look like?

```xml
  <LocationMatch "^/node/(?<host>[\w.-]+\.osc\.edu)/(?<port>\d+)">
    AuthType openid-connect
    Require valid-user

    # ProxyPassReverse implementation
    Header edit Location "^[^/]+//[^/]+" ""

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>

```

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 16, 2020, 4:06pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/16 "2020-03-16T16:06:56Z")

</div>

user@host$ cat /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf

```
 <LocationMatch "^/node/(?<host>[\w.-]+\.my.center\.edu)/(?<port>\d+)">
    AuthType Basic
    AuthName "private"
    AuthBasicProvider ldap
    AuthLDAPURL "ldaps://ldaps.my.center.edu:portnumber/OU=users,DC=dcname,DC=orgname?name"
    AuthLDAPGroupAttribute cn
    AuthLDAPGroupAttributeIsDN on
    Require valid-user
    AuthLDAPBindDN "cn=cnname,ou=users,dc=dcname,dc=orgname"
    AuthLDAPBindPassword ***

    # ProxyPassReverse implementation
    Header edit Location "^[^/]+//[^/]+" ""

    # ProxyPassReverseCookieDomain implemenation
    Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""

    # ProxyPassReverseCookiePath implementation
    Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
    Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"

    LuaHookFixups node_proxy.lua node_proxy_handler
  </LocationMatch>
```

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 16, 2020, 7:29pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/17 "2020-03-16T19:29:33Z")

</div>

Maybe it could be an issue of rewrites?

In your `/etc/ood/config/ood_portal.yml` do you have this set (or left it to the default)?

```auto
use_rewrites: true

```

Honestly I have no idea why you’re being redirected like that. You’re location - this bit here `Header edit Location "^[^/]+//[^/]+" ""` - doesn’t seem to be working.

That very first request you make to Jupyter should have a `Location` header, can you let me know what it is?

---

<div class="post-metadata">

**Author:** ![arashnh11](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/arashnh11/32/341_2.png) [@arashnh11](https://discourse.openondemand.org/u/arashnh11)\
**Post date:** [March 18, 2020, 1:28am UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/18 "2020-03-18T01:28:01Z")

</div>

the use\_rewrites is set to default.  
See attached.

 ![Screen Shot 2020-03-17 at 9.23.41 PM](https://us1.discourse-cdn.com/flex015/uploads/osc/original/1X/5df8a4ac21491ba67c847221b8f8e32428891db4.jpeg)  
Under the General tag in Headers or Response Headers or Request Headers, the “Location” doesn’t appear.

As mentioned, one difference with the other thread is that it routes to authentication again before getting to jupyter, i.e., after “connect to Jupyter” the authentication prompt window appears again. If I authenticate that, then I am prompted to the jupyter notebook with an “invalid credentials” message. Then, attached is the screenshot you asked for.  
As a side note,

I just see something in the jupyter output file from one of the launches:  
/home/user/ondemand/data/sys/dashboard/batch\_connect/dev/jupyter\_notebook/output/jupyter-id/before.sh: line 38: openssl: command not found which means traffic won’t be encrypted. I don’t think that’s related but just wanted to bring that.

Just for sanity checks: here is the websockify installation

I have verified that websockify is installed and working on the COMPUTE node:

[host]# websockify --help  
Usage:  
websockify [options] [source\_addr:]source\_port [target\_addr:target\_port]  
websockify [options] [source\_addr:]source\_port – WRAP\_COMMAND\_LINE

cat /etc/ood/config/clusters.d/mycluster.yml | grep websockify  
export WEBSOCKIFY\_CMD="//usr/bin/websockify"

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 18, 2020, 1:48pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/19 "2020-03-18T13:48:20Z")

</div>

As a sanity check, be sure you’re passing the password in the form data. You’ll see it in under the ‘request headers’ portion of your chrome dev tools. I don’t think it’s a problem, but just to be sure.

Do you have a similar [configuration as us](https://github.com/OSC/bc_osc_jupyter/blob/master/template/before.sh.erb#L29)?

I think specifically these items are important in this case.

```nohighlight
c.NotebookApp.base_url = '/node/${host}/${port}/'
c.NotebookApp.allow_origin = '*'
c.NotebookApp.disable_check_xsrf = True

```

The prompt for re-authentication is strange and should not be happening. Is there anything else you’ve added to your apache `ood-portal.conf`?

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [March 18, 2020, 9:47pm UTC](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777/20 "2020-03-18T21:47:40Z")

</div>

This problem is so out of left field that at this point I’m willing to have a webex (or similar meeting technology) with you to help debug.

I have webex, so we could use that. Let me know if have time tomorrow or Friday.

[Next page](https://discourse.openondemand.org/t/jupyter-notebook-issues-the-requested-url-was-not-found-on-this-server-multiple-authentication-invalid-jupyter-password/777.md?page=2)