# OOD server offload

**URL:** <https://discourse.openondemand.org/t/ood-server-offload/625>\
**Category:** Get Help\
**Created:** [December 3, 2019, 5:54pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625 "2019-12-03T17:54:34Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 3, 2019, 5:54pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/1 "2019-12-03T17:54:34Z")

</div>

My colleague attended the OOD BoF at SC19 where it was mentioned that you have ready a beta of OOD job offload to a standalone Linux server. We would like to try this on our systems. Would you be able to get us set up with that?

Thanks,  
MC

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [December 3, 2019, 9:37pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/2 "2019-12-03T21:37:34Z")

</div>

Martin,

We’re calling the new feature the LinuxHost adapter. We just got a final thumbs up from our head system’s architect today. Docs are in-progress ([Github](https://github.com/OSC/ood-documentation/blob/linuxhostadapter/source/installation/resource-manager/linuxhost.rst)), and we’ll have an RPM available tomorrow.

ps. I just noticed that Github’s RST compiler doesn’t render our code sections properly; the raw file is viewable here: [https://raw.githubusercontent.com/OSC/ood-documentation/linuxhostadapter/source/installation/resource-manager/linuxhost.rst](https://raw.githubusercontent.com/OSC/ood-documentation/linuxhostadapter/source/installation/resource-manager/linuxhost.rst)

---

<div class="post-metadata">

**Author:** ![efranz](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/efranz/32/27_2.png) [@efranz](https://discourse.openondemand.org/u/efranz)\
**Post date:** [December 3, 2019, 9:46pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/3 "2019-12-03T21:46:37Z")

</div>

See [https://osc.github.io/ood-documentation-test/linuxhostadapter/installation/resource-manager/linuxhost.html](https://osc.github.io/ood-documentation-test/linuxhostadapter/installation/resource-manager/linuxhost.html) for a test build of the documentation in progress.

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [December 4, 2019, 8:28pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/4 "2019-12-04T20:28:27Z")

</div>

Martin,

The OnDemand latest repo now has a 1.7.4 build which is the first build where the new adapter is available. You can try the latest by using [https://yum.osc.edu/ondemand/latest/ondemand-release-web-latest-1-6.noarch.rpm](https://yum.osc.edu/ondemand/latest/ondemand-release-web-latest-1-6.noarch.rpm). 1.7.4 has not been tested in prod at OSC, and so we are not yet recommending it for production elsewhere, but if you want to try it out you are welcome. Please, let us know how it goes.

---

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 13, 2019, 4:54pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/5 "2019-12-13T16:54:08Z")

</div>

Hi guys, I did not have the courage to install the new OOD yet (to be safe want to do it on a new VM so need to coordinate with our VM guys). But, we have read the LinuxHost Adapter documentation and have a few questions on it related to how we’d plug it to our existing infrastructure.

First, we are using a home grown tool called Arbiter ([https://dl.acm.org/citation.cfm?id=3333043](https://dl.acm.org/citation.cfm?id=3333043)) to limit abuse of interactive nodes, which uses user cgroups. To avoid conflicts with that tool, we are wondering if it would be possible to tell the LinuxHost Adapter to not enforce any limits, and leave that to the third party tools like Arbiter.

Second, we are wondering if the Adapter keeps any records of jobs running on the interactive node - in order to prevent too many jobs from running on a single node and OOMing it. We can enforce cgroup per user limits through OOD or Arbiter, but what if too many users get offloaded to a single server?

Thanks,  
MC

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [December 13, 2019, 5:18pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/6 "2019-12-13T17:18:41Z")

</div>

Hi Martin,

The LinuxHost adapter does not enforce any resource limits other than a ‘wallclock’ limit which is simply a `timeout` command ([link to source](https://github.com/OSC/ood_core/blob/master/lib/ood_core/job/adapters/linux_host/templates/script_wrapper.erb.sh#L39)). We encourage user limits, but do not require them. OnDemand itself does not know anything about user resource limits.

There is no feature in the adapter to manage the total number of jobs run on the interactive node; indeed this adapter is unusual in that when it lists all jobs it will only ever list `$USER`'s jobs because of the way `tmux` works.

With the disclaimer that I am not a sysadmin or `cgroups` expert I believe that there is a way to limit all resource utilization by the adapter by wrapping the `tmux` or `singularity` executable that the adapter uses and targeting that executable for limits: [https://osc.github.io/ood-documentation/develop/installation/resource-manager/linuxhost.html#approach-2-libcgroup-cgroups](https://osc.github.io/ood-documentation/develop/installation/resource-manager/linuxhost.html#approach-2-libcgroup-cgroups).

ps I should also say that the linked section of documentation **was** written by a sysadmin: @tdockendorf.

---

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 13, 2019, 6:13pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/7 "2019-12-13T18:13:18Z")

</div>

Hi Morgan,

thanks, sounds good. That makes more sense now.

One other question. Is it possible to have more than one submit\_host? And/or to select what particular ssh\_host to use?

The reason for this question is that we have 8 interactive only nodes, named frisco[1-8].chpc.utah.edu, that don’t face any cluster and which we use for interactive work. Each of these machines has different hardware specs, so, users commonly request a specific machine (e.g. frisco8 that has more CPUs/memory if they need them).

Thanks,  
MC

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [December 13, 2019, 6:39pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/8 "2019-12-13T18:39:04Z")

</div>

> [@mcuma](#):
>
> Is it possible to have more than one submit\_host? And/or to select what particular ssh\_host to use?

I didn’t anticipate that as a feature; I have an idea about how I could make the second part work.

Right now you could target nodes individually by giving them each a different cluster config, but that’s not an ideal solution because the Batch Connect framework does not yet\* support changing the target cluster.

\* I am not sure when we will support changing clusters from Batch Connect forms.

---

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 13, 2019, 6:55pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/9 "2019-12-13T18:55:58Z")

</div>

Sounds good, I’ll mess with it once we get that VM set up - which may take a little while since we want to make it a part of a large project.

In either case, we would really love to have the choice of different target clusters in Batch Connect, we’re waiting for that to make the Interactive Apps interface simpler for the 6+ clusters we have. If you could prioritize this that would be great.

Thanks,  
MC

---

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 17, 2019, 7:21pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/10 "2019-12-17T19:21:36Z")

</div>

Hi Morgan,

one more question. How do you handle the authentication on the ssh\_host going from the OOD server? It looks like it’s ssh keys, but, we’re not sure from the docs. And, if so, are they user or host based?

Thanks,  
MC

---

<div class="post-metadata">

**Author:** ![rodgers.355](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/rodgers.355/32/193_2.png) [@rodgers.355](https://discourse.openondemand.org/u/rodgers.355)\
**Post date:** [December 17, 2019, 8:45pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/11 "2019-12-17T20:45:09Z")

</div>

Martin,

The adapter does indeed rely on SSH keys. At OSC we have host based keys set up. I have opened an issue against our docs to clarify that.

I also put together a pull request to add an override for `submit_host` to the adapter. What it does is allow you to set a “native” attribute in a Batch Connect app’s `submit.yml.erb` and that would then send the job to a specific host. The overridden host must be in the `ssh_hosts` list.

If you wanted to demo that a sudoer should create `/etc/ood/config/dashboard/initializers/add_submit_host_override.rb` with the following content:

```ruby
require "ood_core"
require "ood_core/job/adapters/linux_host/launcher"

class OodCore::Job::Adapters::LinuxHost::Launcher
  # @param hostname [#to_s] The hostname to submit the work to
  # @param script [OodCore::Job::Script] The script object defining the work
  def start_remote_session(script)
    cmd = ssh_cmd((script.native && script.native['submit_host_override']) ? script.native['submit_host_override'] : submit_host)

    session_name = unique_session_name
    output = call(*cmd, stdin: wrapped_script(script, session_name))
    hostname = output.strip

    "#{session_name}@#{hostname}"
  end
end

```

Assuming that your form control is named `target_submit_host` your Batch Connect app’s `submit.yml.erb` would then add the native attribute like so:

```auto
batch_connect:
  # ...
script:
  # ...
  native:
    submit_host_override: <%= target_submit_host %>

```

---

<div class="post-metadata">

**Author:** ![mcuma](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/mcuma/32/171_2.png) [@mcuma](https://discourse.openondemand.org/u/mcuma)\
**Post date:** [December 20, 2019, 5:36pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/12 "2019-12-20T17:36:39Z")

</div>

Sounds good, thanks. I am planning to roll a new OOD server with the host adapter after the Holidays so I’ll be in touch once we have that. Happy Holidays.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [April 9, 2020, 2:16pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/13 "2020-04-09T14:16:00Z")

</div>

This feature is a part of the core library now. Just FYI so you don’t need that initializer now.

You can directly do something like this.

```auto
batch_connect:
  # ...
script:
  # ...
  native:
    submit_host_override: <%= target_submit_host %>

```

---

<div class="post-metadata">

**Author:** ![KMuriki](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@KMuriki](https://discourse.openondemand.org/u/KMuriki)\
**Post date:** [November 28, 2020, 8:53am UTC](https://discourse.openondemand.org/t/ood-server-offload/625/14 "2020-11-28T08:53:28Z")

</div>

Jeff,

Should this submit\_host\_override be working in v1.8.12 ?

I’m trying with below in my submit.yml.erb and I’m getting “no implicit conversion of Hash into Array”

```auto
batch_connect:
  # We use the basic web server template for generating the job script
  #
  # @note Do not change this unless you know what you are doing!
  template: "basic"

script:
  native:
    submit_host_override: "n0003.testbed0"

```

Thanks for any tips.  
–Krishna.

---

<div class="post-metadata">

**Author:** ![jeff.ohrstrom](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.openondemand.org/jeff.ohrstrom/32/136_2.png) [@jeff.ohrstrom](https://discourse.openondemand.org/u/jeff.ohrstrom)\
**Post date:** [November 30, 2020, 2:34pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/15 "2020-11-30T14:34:27Z")

</div>

This answer was for the `linux_host` adapter type. What kind of cluster are you trying to submit this with?

---

<div class="post-metadata">

**Author:** ![KMuriki](https://avatars.discourse-cdn.com/v4/letter/k/838e76/32.png) [@KMuriki](https://discourse.openondemand.org/u/KMuriki)\
**Post date:** [December 6, 2020, 8:11am UTC](https://discourse.openondemand.org/t/ood-server-offload/625/16 "2020-12-06T08:11:38Z")

</div>

Yeh I’m using ‘slurm’ adapter in my cluster so nevermind. I incorrectly inferred that you might be saying that submit\_host\_override will work for slurm adapter also. I will go ahead and start playing with linux\_host adapter in my deployment now.

---

<div class="post-metadata">

**Author:** ![westburg.2](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@westburg.2](https://discourse.openondemand.org/u/westburg.2)\
**Post date:** [May 26, 2022, 3:43pm UTC](https://discourse.openondemand.org/t/ood-server-offload/625/17 "2022-05-26T15:43:01Z")

</div>