Running Jupyter on K8s via OOD

We have a Kubernetes cluster set up on System A. Using the Kubernetes API, we are able to reach it on System B where Open On Demand lives. Currently, I’ve set up a Jupyter app to work with Kubernetes; however, I’m seeing an error: error reading input for Password message in the web app. I’ve not been able to track down where/why this is happening yet. I was wondering if anyone recognizes this problem and how they fixed it.

Additional information:

  • We have an auth type of managed for the Kubernetes cluster configuration file (k8s.yml)
  • We have bootstrapped the k8s cluster but not the OOD web node based on the docs

Is this Jupyter that’s returning the error or OnDemand?

The error is a red box titled “Failed to submit session with the following error:” on the jupyter app’s page on Open On Demand. A juptyer instance is never created or launched from what I can tell.

OK then somehow your ~/.kube/config isn’t correct and/or OnDemand isn’t using it correctly. That’s kubectl asking for your password which means it’s not using the right --context. You c

  • spot check your ~/.kube/config to ensure that it has your OIDC information with the right context & cluster information.
  • spot check your /var/log/ondemand-nginx/$USER/error.log for the exact kubectl command you’re issuing. It should be using the right context (I’m just recalling this from memory, but I believe all we issue is --context, but there could be more flags we’re passing).

Hmm, I think the latter may have clued me in. It seems like there’s an expectation for Kubernetes to be installed on the OOD webnode and its executable to be at /bin/kubectl. If that’s true, I’ll start poking there. Since we’re using microk8s we can set up a link to the proper executable at that path (/bin/kubectl). Otherwise, I think the kube config file and --context look right.