Best practice for OOD configuration change out of production

We are running OOD through all our major HPC clusters and we are thinking about adding more features as well some configuration changes for the OOD instances. I would like to learn if there are any best practices for this kind of configuration that would not impact current OOD in production?

Thank you.

Hello and welcome!

Could you provide some more details of what you want to do? I’d say it’s unlikely to affect anything negatively if they are supported changes. But are you changing the UI or something more significant?

Have you already read through the customizations page:
https://osc.github.io/ood-documentation/latest/customizations.html

Do you have a test or dev setup to test on first?

I do want to double check here, are you asking about making changes on your production system and then restarting the OOD service and whether this will affect users? Because it certainly will if you do.

Hi Travis. Thank you for the reply.

The customization page helps a lot for understanding the different configuration options. To be more specific, we are trying to creating more features into our current OOD (e.g. more informative dashboard). So hopefully we could find a way to test the changes before moving them into production. We do have developer mode but that is for specific applications rather than OOD itself.

So if I understand correctly, you are suggesting a dev mode for the OOD instance, right?

What does you current system setup look like? As in, do you have a separate system running ood to do these types of changes on to see the effects before pushing the changes to you production system?

If not, you could set one up on a VM and such.

Or there is another way you could do UI type dev work on OOD within OOD. We do this at OSC when we develop ourselves.

The trick is to setup the Dashboard as an app in your development sandbox, and to then launch it from there. I then use VSCode on the cluster to load that code in, and actually make changes and see them in my dev dashboard rather than the sys dashboard.

Do you have the dev environment enabled already?
https://osc.github.io/ood-documentation/latest/how-tos/app-development/enabling-development-mode.html

The trick to then work on the dashboard app is to make a symlink in your dev and the ondemand dashboard in misc from your home directory. So these locations: ~/ondemand/misc/ondemand/apps/dashboard/ and ondemand/dev/dashboard. OOD will then recognize the dashboard as an app which then you can launch from the developer sandbox apps.

Also, if you do this make sure to go into that symlink directory to build things correctly if you do anything with assets or parts of rails that need reloaded. This can be done with the bin/setup binary in the dashboard app. I know this is a lot, but if you do the dev in ondemand setup and have questions feel free to post and I’ll try to clear up anything as best I can.

1 Like

We only have a OOD in production for each cluster so it’s might not be possible to have a separate system running a test OOD (but this should be a great thing to have).

Learning those tricks, especially using Dashboard as an sandbox app is fresh new to me and it might be useful for now. We have the dev environment enabled so that might be a good method to try first. Let me play with this dashboard app and get back to you if I have further questions. Really appreciate the suggestions!

Sure! I know that post on setting up the dashboard to do dev work on may be a bit jarbled, so please ask if something isn’t making sense and I’m very happy to clear it up.

Hi Travis. I have played with dashboard under developer mode and have a question about how to configure it.

Usually OOD configurations would be made in /etc/ood/config/ and its subfolders. But ~/ondemand/dev/dashboard/ has quite different structures and similar configurations did not work for ./dashboard/.

Could you show me how to do the configurations for this own dashboard app (e.g. Custom layouts in the dashboard as shown in Customizations — Open OnDemand 3.0.3 documentation)? Thanks!

Yep! Ok, so the trick here is to again fake out OOD.

We’ve got this dashboard app now in our sandbox, how do we get the configs to show up there though?

We can do this by going into the root of that app, so cd ~/ondemand/dev/dashboard and from that location create a .env.local file. Now we can start to set environment variables in our dev dashboard! And one clutch environment variable here is the OOD_CONFIG_D_DIRECTORY. This is the key to unlock the magic. Because now we can have our ood dev dashboard set it’s own configuration options and see them take effect.

So, in my own dashboard where i do a majority of dev work on the cluster, mine looks like this:

# .env.local file
# Set  a different color for dev dashboard to prevent confusion with sys dashboard
OOD_BRAND_BG_COLOR='grey'
# Set a different config directory for the dev dashboard to look for configs
OOD_CONFIG_D_DIRECTORY="/users/PZS0714/travert/ondemand/dev/dashboard/config/ondemand.d/"
# you could include any number of other ENV here, like if we wanted 
# to change where message of the days come from so our dev dashboard can display them
MOTD_PATH='/users/PZS0714/travert/ondemand/dev/dashboard/config/ondemand.d/motd'

I then can land all kinds of configs in that new directory for ondemand to pickup and use. Really, aside from interactions with OOD_CORE you could do all dev work in this setup. That’s the norm at OSC for how we do things (I think).

I’m starting to think this should have a page in our docs because…how would anyone piece this together now that I think on it and read this.

I’ve created an issue against our docs to add a section for this so centers and users can be aware how to do this.