Systemd adaptor job is not timing out

Get Help
1

I’m trying to use the systemd adaptor to run applications on a Slurm cluster login node.
Below is my cluster definition for such node. I set a timeout of 180s for testing. the documentation says the attribute name is timeout. The ood_core code seems to use a site_timeout attribute ood_core/lib/ood_core/job/adapters/systemd/launcher.rb at master · OSC/ood_core

I tried both, but my systemd job never get cancelled after 3 minutes.
Can you please help me troubleshoot and fix this ?

---
v2:
  metadata:
    title: "Login ccw"
    hidden: true
  job:
    adapter: "systemd"
    submit_host: "ccw-login-1"
    ssh_hosts:
      - ccw-login-1

    timeout: 180 
    strict_host_checking: false
    debug: true

  # Don't show ssh link in batch connect card
  batch_connect:
    ssh_allow: false
    basic:
      script_wrapper: |
        module purge
        %s

Thank you,
Xavier

2

That adapter was a community contribution, so I can only guess. But if you have the debug flag on, it stands to reason that the resulting script_wrapper.erb.sh may exist as a templated sh script. In that script you’ll see how this line was templated. Did it get the correct RuntimeMaxSec? Also I think site_timeout is the absolute maximum you can request, whereas something like bc_num_hours will be the timeout you request for a job and the adapter will choose the smaller of the 2.

3

Thank you jeff.
Unfortunately I don’t see the RuntimeMaxSec in any of files generated. I will try using bc_num_hours.

4

Yea again, not super familiar with this adapter, but it seems like while it’s running you should at least be able to query systemd to see information about the unit?

5

using bc_num_hours solved it.
Thanks Jeff

6

It seems that the cluster level timeout is not working.
I’ve tried to use various attributes like

I’ve updated ood_core/lib/ood_core/job/adapters/systemd/templates/script_wrapper.erb.sh at master · OSC/ood_core to dump the value of script_timeout and it’s always the value of bc_num_hours expressed in seconds.

---
v2:
  metadata:
    title: "Login ccw"
    hidden: true
  job:
    adapter: "systemd"
    submit_host: "ccw-login-1"
    ssh_hosts:
      - ccw-login-1

    site_timeout: 10
    timeout: 10
    max_timeout: 10
    strict_host_checking: false
    debug: true

  # Don't show ssh link in batch connect card
  batch_connect:
    ssh_allow: false
    basic:
      script_wrapper: |
        module purge
        %s

any idea what can be wrong here ?
Thank you
Xavier