How To Debug Notebook 404

Get Help
1

When I try to hit the URL for Jupyter, I now get a 404. This was working prior and I’m not aware of any particular changes that would have broken this. I’ve verified that the pun capability still work (using the nc + curl test) and I’ve looked through all the logs I can think to check (httpd, messages, slurm output) and I can’t see anything suspicious. The notebook is running at the indicated port/url if I hit it directly. Tried rebooting everything. Not sure where to look next.

2

I’ve found the option c.NotebookApp.base_url = '/node/${host}/${port}/' is important. I believe your 404 is coming from Jupyter itself and we’re just proxying it. I would check to see if this option in the config file is being written correctly if your using /node as the base path (as we do) and not rnode (which may work? we just don’t use it).

If you open your browsers’ dev tools and look at the response of the 404 you may see the response header Server: TornadoServer/4.5.3. This is the server that Jupyter uses.

I’ve seen, in general, we’ll return 502 Bad Gateway if there’s some issue with the way we’re proxying and 404s or similar 300 or 400 level errors come from the application itself, in this case Jupyter.

You URLs directly to jupyter work, what are they? Is it just / or is it /node/<some host>/<some port>

Hope that helps!

3

Thanks Jeff. Looking at my browser history from when things worked, I determined that it was working with FQDN for the target node but now was using a short name and failing. And some of the changes made to the nodes could have impacted that. So I went into submit.yml.erb and wired it to use FQDN:

  set_host: "host=$(hostname -f)"

And now it’s working again! So it seems the short hostname was causing the 404.

Note, I went looking in the global cluster config (per the warning in submit.yml.erb) to try to update the host resolution command in there but didn’t see where it took a command like that for the host (please direct me to the file if possible).

4

Good to hear! This is the structure in the cluster definition you’re looking for.

    v2:
      batch_connect:
        basic:
          set_host: "host=$(hostname -f)"
        vnc:
          set_host: "host=$(hostname -f)"
5

Got it… thanks. Just had to remember to restart the web app :smiley: