Empty job list due to non-utf8 job names

Get Help
1

Apparently the code to provide an Active Jobs list in OoD cannot handle job names (or possibly any field) containing non-utf8 characters. The result is an empty job list (no warning) and an error in the personal logfile. When the characters are in someone else’s job, then only the job list under ‘All Jobs’ is empty.
/var/log/ondemand-nginx/$USER/error.log contains

App 1710930 output: [2024-06-25 11:49:38 +0200 ]  INFO "execve = [{\"SLURM_CONF\"=>\"/etc/slurm/slurm.conf\"}, \"/usr/bin/squeue\", \"--all\", \"--states=all\", \"--noconvert\", \"-o\", \"\\u001E%a\\u001F%N\\u001F%Y\\u001F%j\\u001F%u\\u001F%P\\u001F%M\\u001F%A\\u001F%t\"]"
App 1710930 output: [2024-06-25 11:49:38 +0200 ] ERROR "ArgumentError: invalid byte sequence in UTF-8\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:172:in `split'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:172:in `block (2 levels) in get_jobs'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:157:in `each_line'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:157:in `block in get_jobs'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:153:in `open'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:153:in `get_jobs'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapters/slurm.rb:551:in `info_all'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/job/adapter.rb:93:in `info_all_each'\n/var/www/ood/apps/sys/dashboard/app/models/active_jobs/jobs_json_request_handler.rb:49:in `each'\n/var/www/ood/apps/sys/dashboard/app/models/active_jobs/jobs_json_request_handler.rb:49:in `each_slice'\n/var/www/ood/apps/sys/dashboard/app/models/active_jobs/jobs_json_request_handler.rb:49:in `block in render'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/clusters.rb:123:in `each'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/ood_core-0.25.0/lib/ood_core/clusters.rb:123:in `each'\n/var/www/ood/apps/sys/dashboard/app/models/active_jobs/jobs_json_request_handler.rb:47:in `each_with_index'\n/var/www/ood/apps/sys/dashboard/app/models/active_jobs/jobs_json_request_handler.rb:47:in `render'\n/var/www/ood/apps/sys/dashboard/app/controllers/active_jobs_controller.rb:19:in `block (2 levels) in index'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/mime_responds.rb:214:in `respond_to'\n/var/www/ood/apps/sys/dashboard/app/controllers/active_jobs_controller.rb:10:in `index'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:106:in `run_callbacks'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/live.rb:261:in `block (2 levels) in process'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:42:in `block in running'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/concurrency/share_lock.rb:162:in `sharing'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/activesupport-6.1.7.6/lib/active_support/dependencies/interlock.rb:41:in `running'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/live.rb:253:in `block in process'\n/opt/ood/ondemand/root/usr/share/gems/3.1/ondemand/3.1.4-1/gems/actionpack-6.1.7.6/lib/action_controller/metal/live.rb:303:in `block in new_controller_thread'"

We run OoD-3.1.4, but OoD-3.1.1 has the same issue. A sample slurm jobscript to reproduce the issue has been attached:
non-utf8.txt (90 Bytes)

2

Thanks for the notificaton! I’ve filed this ticket upstream to take care of the same: utf-8 errors in at least Slurm adapter · Issue #838 · OSC/ood_core · GitHub

3

I put a lot of time into trying to replicate this and couldn’t. Then I recalled that the default LANG environment variable is C.

Can you put LANG=en_US.UTF-8 in your /etc/ood/config/apps/dashboard/env file? That should resolve the issue.

4

We tried putting the setting in /etc/ood/config/apps/dashboard/env, but it didn’t fix the issue.
The file is read since the dashboard title has changed.

LANG="en_US.UTF-8"
OOD_DASHBOARD_TITLE="LANG test"
5

:man_facepalming: Actually it should have been env.overload for setting environment variables that are already set.

6

Or you can set it as a part of the nginx_stage.yml configs through pun_custom_env.

https://osc.github.io/ood-documentation/latest/reference/files/nginx-stage-yml.html?highlight=pun_custom_env

7

Thank you for you quick response, but unfortunately setting the variable in /etc/ood/config/apps/dashboard/env.overload

LANG="en_US.UTF-8"
OOD_DASHBOARD_TITLE="LANG=UTF8 test /etc/ood/config/apps/dashboard/env.overload"

or /etc/ood/config/nginx_stage.yml

pun_custom_env:
    OOD_DASHBOARD_TITLE: "LANG=UTF8 test /etc/ood/config/nginx_stage.yml"
    LANG: "en_US.UTF-8"

doesn’t change the behaviour.

8

Did you restart your webserver after the change? You’d have to do to that for the new configs to be picked up.

9

We tried restarting httpd, but the change in the title wasn’t visible. We then killed nginx, so we didn’t have to wait for a timeout/cache refresh/…
Then the title was changed, but the the joblist was still empty.
How should we have to do this?

10

There’s a button in the Help menu to ‘Restart Web Server’. This will cause the app to reload configurations.

I’ll try to figure out why this isn’t working for you and get back with a solution that actually works.

11

Thanx, that’s a useful feature I didn’t know about. :wink:
Still the result is the same.

12

Curious - I was able to correctly set the environment variable with env.overload file. Are you sure this file is 644 permissions (i.e., readable by regular users?)

13

env.overload has 644 permissions.

-rw-r–r-- 1 root root 110 Jun 28 17:06 env.overload

We used the OOD_DASHBOARD_TITLE variable to check if the settings were read and every time the title has changed.

14

Maybe try all 3 in env.overload?

LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_ALL=en_US.UTF-8

As a last resort - may an initializer in /etc/ood/config/apps/dashboard/initializers?

# /etc/ood/config/apps/dashboard/initializers/lang.rb
# set the LANG environment variable.
Rails.application.config.after_initialize do
  ENV['LANG'] = "en_US.UTF-8"
end
15

Both solutions have no effect with the example jobscript I uploaded before:
non-utf8.txt

16

OK there’s something we’re missing here, because I can’t replicate in 3.1.7 in a container. I’m hesitant to try on our production systems because it could obviously break things.

image
image2438×214 55.6 KB

What’s you’re locale -a output look like? I’m guessing it works for me because I have C.utf8 here.

[root@ondemand /]# locale -a
C
C.utf8
POSIX
17 
[root@login04 ~]# locale -a
C
C.utf8
en_AG
en_AU
en_AU.utf8
en_BW
en_BW.utf8
en_CA
en_CA.utf8
en_DK
en_DK.utf8
en_GB
en_GB.iso885915
en_GB.utf8
en_HK
en_HK.utf8
en_IE
en_IE@euro
en_IE.utf8
en_IL
en_IN
en_NG
en_NZ
en_NZ.utf8
en_PH
en_PH.utf8
en_SC.utf8
en_SG
en_SG.utf8
en_US
en_US@ampm
en_US.iso885915
en_US.utf8
en_US.utf8@ampm
en_ZA
en_ZA.utf8
en_ZM
en_ZW
en_ZW.utf8
POSIX
18

Is login4 the machine that’s issuing your squeue command?

19

Yes, and squeue does return a valid result on that query (including the non-utf char).
I just noticed that there is a new 3.1.7 release, I will deploy this one and retry (just to be sure). I will let you know the result.

20

OK - that may be the missing piece - are you using a submit_host to run squeue? it could be ssh that’s throwing this off. You need to update to 3.1.7, but that’s not going to fix this issue I don’t think.