Regarding host key verification & System Status issue with Ondemand 4.1.4

[harjad84]

Hello,

I experience a lot of problem with ondemand 4.1.4.

1. For example: I experience a following error even though I have shared the host keys in slurm control node but still I am experiencing this same error since day ago. This error comes up with I try to submit a job. I even restarted the service many times but the chnges didn’t take effect. I don understand this and I feel so irritating experience and so tired of working on this issue with this ondemend 4.1.4 version. This is jus on example.
   No ED25519 host key is known for slurmctl.lan.csc.uni-frankfurt.de and you have requested strict checking.
   Host key verification failed.

   Here’s my cluster yml file wher I have added strict_host_key_checking below:

   ---

   v2:
   metadata:
   title: “g_acct_hlr cluster”
   login:
   host: “slurmctl.lan.csc.uni-frankfurt.de”
   job:
   adapter: “slurm”
   cluster: “g_acct_hlr”
   submit_host: “slurmctl.lan.csc.uni-frankfurt.de” # or your cluster login node
   sbatch: “/usr/bin/sbatch”
   squeue: “/usr/bin/squeue”
   scancel: “/usr/bin/scancel”
   sinfo: “/usr/bin/sinfo”
   bin: “/usr/bin”
   conf: “/etc/slurm/slurm.conf”
   bin_overrides:
   sbatch: “/usr/local/bin/sbatch”
   squeue: ““
   scontrol: ““
   scancel: ““
   copy_environment: false
   strict_host_key_checking: false

2. Another example is that, Sytem Status never works and throws below message on web page:
   This page has encountered an unexpected error. Please refresh the page.

Is anyone can help me with this?

Thanks in advance!

[jeff.ohrstrom]

For the second issue - there could be a bug with system-status pages and submit_host configurations. Or your first issue is creating the second.

For the first issue - you have to have the shared key on both the server that you’ve installed OnDemand on and slurmctl.lan.csc.uni-frankfurt.de.

From your comment, I can’t tell if strict_host_key_checking is in the correct indentation. You can use backticks - ` - to format YAML in discourse like so:

```
yaml:
key: value
```

Beyond that - I’ll have to check if strict_host_key_checking is even a config, if I recall correctly it’s just strick host checking, not the key.

[harjad84]

Thanks for your kind response!

Regarding the first issue (inresponse to your answer), i have already shared the key on both the servers (ondemand & slurmctl) but no luck still the same error.

Regarding the second issue, it could be great if you give me some suggestion it. Cause here also I tried my best for past 4 days but no luck.

Thanks again!!!

[statiksof]

just in case you are facing the same issue:

System-status not working after upgrade to 4.1.1

Hi, I just upgraded ondemand to 4.1.1. without errors. system-status (slurm cluster) is not working anymore. it is stuck on `loading`. in the console log: Failed to load resource: the server responded with a status of 500 (Internal Server Error) I also see some errors in error.log: Error during failsafe response: ActionController::UnknownFormat

[jeff.ohrstrom]

@harjad84

Regarding the first issue (inresponse to your answer), i have already shared the key on both the servers (ondemand & slurmctl) but no luck still the same error.

You can try to replicate the issue without OnDemand and just use a shell. If it works from regular shell as a regular user (not root!), then it’ll work with OnDemand, but in any case you can remove OnDemand from the equation to debug. Maybe the keys you have aren’t ED25519 or ssh_config is slightly off on the ondemand machine. In any case, testing the same thing from a shell as a regular user will help reduce the problem down.

The 2nd issue is likely caused by the first.

[harjad84]

@jeff.ohrstrom

You can try to replicate the issue without OnDemand and just use a shell. If it works from regular shell as a regular user (not root!), then it’ll work with OnDemand, but in any case you can remove OnDemand from the equation to debug. Maybe the keys you have aren’t ED25519 or ssh_config is slightly off on the ondemand machine. In any case, testing the same thing from a shell as a regular user will help reduce the problem down.

The 2nd issue is likely caused by the first.

Hi,
Fromm OOD interactive shell and from OOD server shell, it works pretty well but when I try to submit a form using bc_desktop app then I experience the above error. I tried my best to check with this issue but no clue where it got stuck. Keeps on repeating the same error. I have enclosed the screen shot for the same. COuld you please check and let me know?

Thanks in advance!

Best Regards,
Hariharan