Hello,I am using job composer start a slurm job holding a ray cluster,and there is a database recording status of the cluster and job running on it
I wish I can simplely use the stop button stop the cluster,not only use scancel kill the job(I guess),but also give me a chance to running my own code do some prepare such as cancel the task running on ray,do some alert by email etc.
Yes we use scancel to stop the job. I don’t follow the question exactly about stopping the cluster. As an unprivileged user (i.e., not the administrator) most people don’t have access required to stop the entire cluster. Indeed, we wouldn’t want to stop the entire cluster because other people’s jobs would be impacted. For these reasons, we’re not likely to add a button to stop the cluster.
But I could be misunderstanding the question.
In my use case,every user can submit a job,behind that job,it creating a ray cluster,this ray cluster only used by the creator。
OK yea I’m not sure how to accommodate this use case as it seems specific to your workflow. The job is the unit that OnDemand handles mostly. Things/processes within the job we don’t typically interact with.
Thank you for helping
I am using a background monitor to fix the status now,this won’t block my job
But I think if there is some hook for job composer’s action that can be user define will help a lot,for example I can do some notify/statistic logic before a job start/stop to people/admin/sre who might interesting of these event
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.