Is there a description of how to configure the module browser?
It seems that we need to add to ondemand.yml
module_file_dir: “/usr/local/lmod/lmod/share/spider-json” # just an example path
And then use the ‘module –redicted spider > ‘meaningful-file-name’
to generate a json record of modules in the module_file_dir.
Can’t tell if this is what’s implemented in the new module browser, or some other approach.
Thanks
~ Em
This is the page displayed.
Hi Em,
The docs have an entry here to cover the use of that new feature:
The key thing there is make sure you are making a file for each cluster that you wish to query from, and just dump the output of the spider command as the example shows. If you have more questions around how to use the new feature let me know though!
Thanks, Travis –
I’ve not before generated the json output. How is that done?
When I try using ‘spider’, i get a real mess compared with what i expect
$ spider -o spider-json $MODULEPATH > cluster.json
Thoughts?
~ Em
ps Here’s the first segment. It seems to be a single-line file….:
{“Blosc2”:{“/usr/local/easybuild_allnodes/modules/all/Blosc2/2.6.1-GCCcore-11.3.0.lua”:{“Version”:“2.6.1-GCCcore-11.3.0”,“pV”:“000000002.000000006.000000001.*gcccore.*zfinal-.000000011.000000003.*zfinal”,“Description”:“Blosc, an extremely fast, multi-threaded, meta-compressor library”,“whatis”:[“Description: Blosc, an extremely fast, multi-threaded, meta-compressor library”,“Homepage: https://www.blosc.org/”,“URL: https://www.blosc.org/”\],“URL”:“https://www.blosc.org/”,“hidden”:false,“fullName”:“Blosc2/2.6.1-GCCcore-11.3.0”,“wV”:“000000002.000000006.000000001.*gcccore.*zfinal-.000000011.000000003.*zfinal”,“help”:“\nDescription\n===========\nBlosc, an extremely fast, multi-threaded, meta-compressor library\n\n\nMore information\n================\n - Homepage: https://www.blosc.org/\n”,“lpathA”:{“/usr/local/easybuild_allnodes/software/Blosc2/2.6.1-GCCcore-11.3.0/lib”:1}}},“HTSlib”:{“/usr/local/easybuild_allnodes/modules/all/HTSlib/1.12-GCC-10.3.0.lua”:{“Version”:“1.12-GCC-10.3.0”,“pathA”:{“/usr/local/easybuild_allnodes/software/HTSlib/1.12-GCC-10.3.0/bin”:1},“Description”:“A C library for reading/writing high-throughput sequencing data.\n This package includes the utilities bgzip and tabix”,“fullName”:“HTSlib/1.12-GCC-10.3.0”,“whatis”:[“Description: A C library for reading/writing high-throughput sequencing data.\n This package includes the utilities bgzip and tabix”,“Homepage: https://www.htslib.org/”,“URL: https://www.htslib.org/”\],“URL”:“https://www.htslib.org/”,“hidden”:false,“wV”:“000000001.000000012.*gcc.*zfinal-.000000010.000000003.*zfinal”,“help”:“\nDescription\n===========\nA C library for reading/writing high-throughput sequencing data.\n This package includes the utilities bgzip and tabix\n\n\nMore information\n================\n - Homepage: https://www.htslib.org/\n”,“lpathA”:{“/usr/local/easybuild_allnodes/software/HTSlib/1.12-GCC-10.3.0/lib”:1},
The json object is what you have there, that output from the spider command you issued. The idea is to just dump that into a file to be read later by the module browser in a location you set with the config field module_file_dir.
So, if you look at the example command in that doc entry it looks like this:
tmp="$(mktemp)"; "$LMOD_DIR/spider" -o spider-json "$MODULEPATH" > "$tmp"; mv "$tmp" /some/directory/my_cluster.json
it’s setting variable for tmp then issuing the spider command from whatever the env var for LMOD_DIR is located, it’s saying we want an o output of spider-json and to place that object in the file located at the tmp location. Then, we mv the file to the location we’ve set for the modules_file_dir.
So, part of this command generates the json object and part of it saves it to the file we want and then part of it moves that file where we’ve set OOD to look.
The core idea is spider will generate this json object for us, we then need to save it and store it in the correct location that we set.
Does this make more sense? I can dig into any area that still seems confusing, just let me know!
Travis – thanks for confirming the format of json appears suitable for use.
Does the instruction need to be in a certain section of ondemand.yml?
I’ve placed it in the 1st section, my line 5
$ head -n 6 /etc/ood/config/ondemand.d/ondemand.yml
xdmod_widget_jobs:
#xdmod_widget_job_efficiency:
module_file_dir: “/usr/local/lmod/lmod/share/spider-json”
What other prereqs are there for the module browser to recognize the json file content?
Any suggestions for testing?
Thanks for the work, and the support!
~ Em
It should just stand as its own stanza anywhere, I don’t think there’s a specific section looking at the doc page.
There shouldn’t be any prereqs aside from making sure OOD can read that json file.
Testing depends on if you’ve setup an environment for testing or not. That is a pretty typical pattern in most technical communities, though HPC does seem to go its own way on that front from what I’ve seen (which is very strange to me, but it is what it is). But yeah, I’d absolutely setup a test environment for stuff like this going forward if you don’t have one so you can test these features out.