We have two OOD environments and both use the same shared home directory. I have noticed that existing interactive jobs were disappeared when I opened both OODs. I guess the problem is both are using the same /home/user/ondemand directory.
Can you be more specific about what these two OOD environments are (staging and development or different sites altogether)
As @azric pointed out you can change them so they’re logically different ‘sites’ if that’s the case. We do this, but the installations are logically different - even though they share the same NFS attachments.
We run dev/staging/prod with the same configurations and it’s fine. If this is your use case, then you only need to have the same cluster.d configs populated everywhere.
Here’s a topic on a similar issue from an LSF user who is running it in multi-cluster mode. If you’re running LSF in the same manner, this may be applicable to you.
@gp4r can you describe the end state functionality you actually want? Do you want the interactive sessions to show up in both OOD instances? Do you want them to act completely independently and not show sessions started from the other?
As an aside, at OSC we currently have 3 physical clusters, 1 running SLURM the other 2 Torque/MOAB, and multiple OOD instances that can connect to all clusters, shared home directories and have interactive sessions show up across all of them.
@gp4r if you’ve enabled ondemand_portal: ood2 then you just need to restart your PUN (the restart web server option at the top right) for it to take affect.
If you want sessions to show up in both OOD instances, then both instances need to be carbon copies of each other. That means they have essentially the same /etc/ood directory (nginx_stage.yml, cluster.d directory, etc).
For them to be completely independent, then specifying a different ondemand_portal is the route you should take.
We do not use HA Proxy because each OOD instance is effectively a different site with no redundancy. For dev/staging/production - they all have the same configurations (notably they all point use the same clusters.d folder). For new logically seperate sites altogether (for instance ondemand.osc.edu vs apps.totalsim.us for our commercial clients) we use differing ondemand_portal configurations to separate them.