Hi there!
We just recently pushed our Tensorboard OnDemand app: https://github.com/stanford-rc/sh_ood-apps/tree/master/sh_tensorboard
It’s based on a native installation of Tensorflow, loadable through a module system (we use Lmod). We don’t use containers for that, but that part can easily be customized.
One particularly interesting feature of that Tensorboard OnDemand app (for me at least! :)) is that it implements an authenticating reverse proxy. Because Tensorboard doesn’t provide any kind of authentication mechanism for its web interface, on a shared environment, anybody knowing the hostname and port number of a running Tensorboard instance can connect to it.
To mitigate this, we implemented an authentication mechanism that basically sets a browser cookie in the OnDemand interactive app page (the “Connect to Tensorboard” button does this) which is then checked by the authenticating reverse proxy that controls access to the Tensorboard web interface. Without that cookie, access to the Tensorboard web interface is refused. And if the cookie is ever lost, users can re-create it by visiting the “My Interactive Sessions” page and clicking the “Connect” button again.
It’s been running in production for some time now on our Sherlock cluster and seems to be working fine for us.
If anyone wants to give it a try, please don’t hesitate to let us know how it goes!