Losing connectivity to JupyterLab session, losing work

Get Help
1

Hello, all.

It’s been a while since I’ve been here, but one of my users is facing an issue that I haven’t been able to figure out. He is running JupyterLab as an interactive app within Open OnDemand (v1.7.0). He is apparently leaving it to run his computations, but when he comes back to it a couple of hours later, his session has been disconnected, and when he attempts to reconnect, he has lost his work and must start over. Is there anything that can be done to permit him to reconnect to JupyterLab and continue where he left off? Or, alternatively, is there a way to extend the amount of time he can remain connected to the app while it completes his calculations?

Thanks,

Richard

2

Hey sorry for the delay in the reply.

I’m not sure what the behavior here is. I’ll have to replicate and get back to you on what potentially we can do.

3

Hello Jeff,

I do see a case, where long running jobs can create issues with Jupyter.
May I know if we had more users complain about this issue?

I see some users, who are asking me if I have a solution for this, where if they run a particular computation that can take more than few hours, between which they can loose network connectivity.
When they come back, they can connect back to the same notebook/session but they cannot see the same running computation from last time.

I tried this myself, and I see that I when I start a long running computation on a GPU. Close the browser, reconnect to open-ondemand interactive sessions and open my notebook - My previous computation would have been still running, but I won’t see any output in the notebook that it is still running. Any more commands that I give to GPU will wait until the previous invisible computation finishes.

This has been a big topic within Jupyter forums:

I do see notes about them working on RTC, which might solve this issue.

Although this is not necessarily directly related to OOD. I wanted to see, if any of users found an innovative solution to disconnects (other than outputting everything to a local log and watch the output from local log on the server)

Thank you,
Lohit

4

Thanks for the background information!

No I haven’t heard much more than what you’re talking about now.

Though this hack comes to mind -

  1. get a desktop
  2. launch jupyter in that desktop
  3. connect to that jupyter from a browser on the same machine

In this way - you never lose connectivity because everything is contained within the desktop. The connection from the browser and jupyter should be stable given the desktop will continue to run for however long. This way the user reconnects to the desktop - which has within it a browser that’s still connected.

5

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.