Problems getting apps to work under 1.5

Get Help
1

Hi Eric,

I am trying to get OOD 1.5 up on our development server, vm039.bridges.psc.edu. I’m really in a quandry. Any chance I could pick your brain a bit?

Neither Jupyter nor Rstudio is starting up. Both seem to execute properly and I get a blue ‘Connect to Jupyter|Rstudio Server’ button and the output log doesn’t show an error:

JUPYTER output.log:

Script starting...
Generating connection YAML file...
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'python/3.6.2'
[W 13:01:17.269 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 13:01:17.488 NotebookApp] JupyterLab alpha preview extension loaded from /opt/packages/anaconda3.5/lib/python3.6/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 13:01:17.501 NotebookApp] Running the core application with no additional extensions or settings
[I 13:01:17.541 NotebookApp] Serving notebooks from local directory: /home/susan
[I 13:01:17.541 NotebookApp] 0 active kernels 
[I 13:01:17.541 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:43270/node/r752.pvt.bridges.psc.edu/43270/
[I 13:01:17.541 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

RSTUDIO output.log:

Script starting...
Generating connection YAML file...
+ module list
++ /usr/bin/modulecmd bash list
Currently Loaded Modulefiles:
  1) singularity/2.6.0           2) rstudio_singularity/3.4.3
+ eval
Starting up rserver...
+ [[ -v PSC_OOD_CUSTOM_R ]]
+ /opt/packages/R/3.4.3/bin/rserver --www-port 9425 --auth-none 0 --auth-pam-helper-path /home/susan/ondemand/data/sys/dashboard/batch_connect/sys/rstudio_app/output/55e6cf90-d113-48b6-9afa-e56c295a9af1/bin/auth --auth-encrypt-password 0 --rsession-path /home/susan/ondemand/data/sys/dashboard/batch_connect/sys/rstudio_app/output/55e6cf90-d113-48b6-9afa-e56c295a9af1/rsession.sh

But I get this error on both apps:

Not Found
The requested URL /node/r752.pvt.bridges.psc.edu/43270/login was not found on this server.

Now I notice that I have nothing under /var/www/ood/apps/usr. Should there be a link there back to my /home/susan/ondemand/dev dir? Could the problem be that the above directory /home/susan/ondemand/dev exists and a link can’t be created?

Thanks for any help.

Susan

2

More information: I copied a production version of the Jupyter Notebook app from vm028 (our OOD production server) and got it to work although there were some complaints at the end of the log that dirs don’t exist. Here’s the output.log file:

Script starting...
Generating connection YAML file...
ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'python/3.6.2'
[W 13:37:41.270 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 13:37:41.312 NotebookApp] JupyterLab alpha preview extension loaded from /opt/packages/anaconda3.5/lib/python3.6/site-packages/jupyterlab
JupyterLab v0.27.0
Known labextensions:
[I 13:37:41.315 NotebookApp] Running the core application with no additional extensions or settings
[I 13:37:41.320 NotebookApp] Serving notebooks from local directory: /home/susan
[I 13:37:41.321 NotebookApp] 0 active kernels 
[I 13:37:41.321 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:46784/node/r752.opa.bridges.psc.edu/46784/
[I 13:37:41.321 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 13:37:44.801 NotebookApp] 302 POST /node/r752.opa.bridges.psc.edu/46784/login (10.4.128.155) 1.40ms
[I 13:37:44.900 NotebookApp] 302 GET /node/r752.opa.bridges.psc.edu/46784/ (10.4.128.155) 0.52ms
[W 13:37:46.444 NotebookApp] /home/susan/WORK doesn't exist
[W 13:37:46.448 NotebookApp] /home/susan/.#pghsync doesn't exist
[W 13:37:46.450 NotebookApp] /home/susan/SCRATCH doesn't exist
3

Ditto for Rstudio – the production version is working on the development node. So it must have been something in those 2 apps that was causing the problem. This isn’t the latest Rstudio that works for different versions of R, but it’s a start. Here is the output.log file:

Script starting...
Generating connection YAML file...
+ module list
++ /usr/bin/modulecmd bash list
Currently Loaded Modulefiles:
  1) singularity/2.6.0           2) rstudio_singularity/3.4.3
+ eval
Starting up rserver...
+ /opt/packages/R/3.4.3/bin/rserver --www-port 59898 --auth-none 0 --auth-pam-helper-path /home/susan/ondemand/data/sys/dashboard/batch_connect/sys/rstudio_app/output/ee959e1d-bac8-4f76-bd11-8b7891ee853f/bin/auth --auth-encrypt-password 0 --rsession-path /home/susan/ondemand/data/sys/dashboard/batch_connect/sys/rstudio_app/output/ee959e1d-bac8-4f76-bd11-8b7891ee853f/rsession.sh
4

Susan,

I am able to duplicate the error you see if I either disable the node_uri in my testing environment’s ood_portal.yml or if I attempt to connect to a node name that is not whitelisted by the host_regex. There could also be something wrong in batch applications’ view.html.erb.

Would you be willing to share your the host_regex from your ood_portal.yml and view.html.erb?

-Morgan

5

Here are the excerpts you asked for:

host_regex: '(r|gpu|l|xl|or|om|on|ol|dgpu|dr)[0-9\-]+\.(opa|ib)\.bridges\.psc\.edu'

[root@vm039 jupyter_app]# cat view.html.erb
<form action="/node/<%= host %>/<%= port %>/login" method="post" target="_blank">
  <input type="hidden" name="password" value="<%= password %>">
  <button class="btn btn-primary" type="submit">
    <i class="fa fa-eye"></i> Connect to Jupyter
  </button>
  <p></p>
  <p>
  Select the blue <strong>Connect to Jupyter</strong> button above.
  If you see <code>Failed to connect to ...</code>, then wait a few seconds
  before trying the button again. This warning might appear if the Jupyter
  Notebook is still starting up.
</p>
</form>

Thanks for the help. (I’m on PTO yesterday and today but will be checking in now and then.)

-Susan

6

@suzlitz it looks like the host_regex is the problem. In order to be able to use the host r752.pvt.bridges.psc.edu you would need to add pvt to the second capture group: (r|gpu|l|xl|or|om|on|ol|dgpu|dr)[0-9\-]+\.(opa|ib|pvt)\.bridges\.psc\.edu.

7

Adding .pvt to the regex didn’t fix the problem. I think that is done behind the scenes somewhere because the connection part worked previous to .pvt being added.

You suggest that I remove the ‘uri’ line in ood_portal.yml. There are 2 uri lines and I commented both of them out:

ood_portal.yml:node_uri: "/node"
ood_portal.yml:rnode_uri: "/rnode"

No change. I still see 2 Jupyter Notebooks: Form and Manifest.
Will keep poking around to see how to fix this. I’d really like to have this fixed for next Wednesday, April 10th. We have a maintenance period and it would be a great time to swap in the ood-development VM for the very outdated ondemand VM.

Thanks for your help – Susan

8

To be clear did you add .pvt to your host_regex or just pvt? Unless you changed the structure of your host_regex I wouldn’t expect that .pvt would work.

9

The . precedes that regex section:

host_regex: '(r|gpu|l|xl|or|om|on|ol|dgpu|dr)[0-9\-]+\.(opa|ib|pvt)\.bridges\.psc\.edu'
10

Thanks, I just wanted to be sure.

11

Just to document the solution which was determined offline. This problem was caused by the the same state that introduced another problem Interactive Apps drop down menu showing 2 version of Jupyter notebook: Form and Manifest. The issue was that the custom Jupyter notebook interactive app plugin code was installed at /var/www/ood/apps/sys/jupyter_app and then there was another copy of the code at /etc/ood/config/apps/jupyter_app. Because the Dashboard by default looks for an interactive app plugins (batch connect plugins) “subapps” in /etc, the form.yml config files being used for Jupyter in this case were in /etc/ood/config/apps/jupyter_app instead of in /var/www/ood/apps/sys/jupyter_app.

The solution was to remove the duplicate directory at /etc/ood/config/apps/jupyter_app.

The long term solution will be a plugin architecture that doesn’t feature a confusing concept like “sub-apps”.

12

Hi Eric & Crew,

This error crept back into my ondemand server, the new one vm039, which I had previously fixed. The solution you suggest above (make sure no ‘subapps’ appear in /etc/ood/config/apps) is fixed … the only one there was juptyer and it is now named jupyter.bak. I made sure the regex includes pvt and opa:

host_regex: '(r|gpu|l|xl|or|om|on|ol|dgpu|dr)[0-9\-]+\.(opa|ib|pvt)\.bridges\.psc\.edu'

The output.log looks fine:

Script starting…
Generating connection YAML file…
[W 13:58:49.111 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 13:58:49.170 NotebookApp] JupyterLab beta preview extension loaded from /opt/packages/anaconda/anaconda3-5.1.0/lib/python3.6/site-packages/jupyterlab
[I 13:58:49.170 NotebookApp] JupyterLab application directory is /opt/packages/anaconda/anaconda3-5.1.0/share/jupyter/lab
[I 13:58:49.190 NotebookApp] Serving notebooks from local directory: /home/susan
[I 13:58:49.191 NotebookApp] 0 active kernels
[I 13:58:49.191 NotebookApp] The Jupyter Notebook is running at:
[I 13:58:49.191 NotebookApp] http://[all ip addresses on your system]:33186/node/r752.opa.bridges.psc.edu/33186/
[I 13:58:49.191 NotebookApp] Use Control-C to stop this server and shut down all kernels (Preformatted text

Seems I said ‘all is good’ yesterday before I tested whether the apps worked. Any suggestions?

TIA - Susan

13

More info: checking the node, the connection is there just fine:

[root@r744 ~]# ps auxw | grep jup
susan 19259 0.0 0.0 113268 1728 ? S 15:14 0:00 /bin/bash -l /home/susan/ondemand/data/sys/dashboard/batch_connect/sys/jupyter_app/output/8ad09324-67fb-4309-a1d7-9de441a496d3/script.sh
susan 19356 0.9 0.0 329564 60408 ? S 15:14 0:01 /opt/packages/anaconda/anaconda3-5.1.0/bin/python /opt/packages/anaconda/anaconda3-5.1.0/bin/jupyter-notebook --config=/home/susan/ondemand/data/sys/dashboard/batch_connect/sys/jupyter_app/output/8ad09324-67fb-4309-a1d7-9de441a496d3/config.py
root 19679 0.0 0.0 112664 972 pts/2 S+ 15:16 0:00 grep --color=auto jup

14

This error keeps showing up in my /var/log/ondemand-nginx/susan/error.log but I don’t see what the error is:

App 8284 output: [2019-04-15 10:46:20 -0400 ]  INFO "method=GET path=/pun/sys/dashboard/batch_connect/sessions.js format=js controller=BatchConnect::SessionsController action=index status=200 duration=57.83 view=13.74"
App 8284 output: [2019-04-15 10:46:31 -0400 ]  INFO "execve = [{}, \"/opt/packages/slurm/default/bin/squeue\", \"--all\", \"--states=all\", \"--noconvert\", \"-o\", \"%a\\u001F%A\\u001F%B\\u001F%c\\u001F%C\\u001F%d\\u001F%D\\u001F%e\\u001F%E\\u001F%f\\u001F%F\\u001F%g\\u001F%G\\u001F%h\\u001F%H\\u001F%i\\u001F%I\\u001F%j\\u001F%J\\u001F%k\\u001F%K\\u001F%l\\u001F%L\\u001F%m\\u001F%M\\u001F%n\\u001F%N\\u001F%o\\u001F%O\\u001F%q\\u001F%P\\u001F%Q\\u001F%r\\u001F%S\\u001F%t\\u001F%T\\u001F%u\\u001F%U\\u001F%v\\u001F%V\\u001F%w\\u001F%W\\u001F%x\\u001F%X\\u001F%y\\u001F%Y\\u001F%z\\u001F%Z\\u001F%b\", \"-j\", \"5278527\"]"
App 8284 output: [2019-04-15 10:46:31 -0400 ]  INFO "method=GET path=/pun/sys/dashboard/batch_connect/sessions.js format=js controller=BatchConnect::SessionsController action=index status=200 duration=57.80 view=16.94"
App 8284 output: [2019-04-15 10:46:42 -0400 ]  INFO "execve = [{}, \"/opt/packages/slurm/default/bin/squeue\", \"--all\", \"--states=all\", \"--noconvert\", \"-o\", \"%a\\u001F%A\\u001F%B\\u001F%c\\u001F%C\\u001F%d\\u001F%D\\u001F%e\\u001F%E\\u001F%f\\u001F%F\\u001F%g\\u001F%G\\u001F%h\\u001F%H\\u001F%i\\u001F%I\\u001F%j\\u001F%J\\u001F%k\\u001F%K\\u001F%l\\u001F%L\\u001F%m\\u001F%M\\u001F%n\\u001F%N\\u001F%o\\u001F%O\\u001F%q\\u001F%P\\u001F%Q\\u001F%r\\u001F%S\\u001F%t\\u001F%T\\u001F%u\\u001F%U\\u001F%v\\u001F%V\\u001F%w\\u001F%W\\u001F%x\\u001F%X\\u001F%y\\u001F%Y\\u001F%z\\u001F%Z\\u001F%b\", \"-j\", \"5278527\"]"
App 8284 output: [2019-04-15 10:46:42 -0400 ]  INFO "method=GET path=/pun/sys/dashboard/batch_connect/sessions.js format=js controller=BatchConnect::SessionsController action=index status=200 duration=55.56 view=14.59"

In /var/log/ood/ondemand.bridges.psc.edu_error.log, I am seeing these messages:

[Mon Apr 15 10:50:11.498580 2019] [lua:info] [pid 8586] [client 128.182.168.129:46842] req_protocol="HTTP/1.1" req_handler="proxy-server" req_method="GET" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" req_user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" res_content_length="" req_content_type="" res_content_encoding="" req_status="200" req_origin="" time_user_map="72.911" local_user="susan" req_referer="https://ondemand.bridges.psc.edu/pun/sys/dashboard/batch_connect/sessions" res_content_language="" req_port="443" log_time="2019-04-15T14:50:11.498458Z" req_server_name="ondemand.bridges.psc.edu" log_hook="ood" req_accept_charset="" req_hostname="ondemand.bridges.psc.edu" res_content_location="" res_content_disp="" req_is_websocket="false" remote_user="susan" res_location="" req_user_ip="128.182.168.129" req_is_https="true" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1555337702692" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" time_proxy="61.12" res_content_type="text/javascript; charset=utf-8" req_accept_language="en-us,en;q=0.9" req_cache_control="" req_accept_encoding="gzip, deflate, br", referer: https://ondemand.bridges.psc.edu/pun/sys/dashboard/batch_connect/sessions
[Mon Apr 15 10:50:22.503353 2019] [lua:info] [pid 6407] [client 128.182.168.129:46844] req_protocol="HTTP/1.1" req_handler="proxy-server" req_method="GET" req_accept="text/javascript, application/javascript, application/ecmascript, application/x-ecmascript, */*; q=0.01" req_user_agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" res_content_length="" req_content_type="" res_content_encoding="" req_status="200" req_origin="" time_user_map="73.078" local_user="susan" req_referer="https://ondemand.bridges.psc.edu/pun/sys/dashboard/batch_connect/sessions" res_content_language="" req_port="443" log_time="2019-04-15T14:50:22.503244Z" req_server_name="ondemand.bridges.psc.edu" log_hook="ood" req_accept_charset="" req_hostname="ondemand.bridges.psc.edu" res_content_location="" res_content_disp="" req_is_websocket="false" remote_user="susan" res_location="" req_user_ip="128.182.168.129" req_is_https="true" req_filename="proxy:http://localhost/pun/sys/dashboard/batch_connect/sessions.js?_=1555337702693" req_uri="/pun/sys/dashboard/batch_connect/sessions.js" time_proxy="59.398" res_content_type="text/javascript; charset=utf-8" req_accept_language="en-us,en;q=0.9" req_cache_control="" req_accept_encoding="gzip, deflate, br", referer: https://ondemand.bridges.psc.edu/pun/sys/dashboard/batch_connect/sessions
15

Found some info here that might help track down the issue:

cat /var/log/httpd24/error_log

[Sun Apr 14 03:27:01.893609 2019] [http2:warn] [pid 8215] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Sun Apr 14 03:27:01.894306 2019] [lbmethod_heartbeat:notice] [pid 8215] AH02282: No slotmem from mod_heartmonitor
[Sun Apr 14 03:27:01.894507 2019] [passenger:error] [pid 8215] *** Passenger could not be initialized because of this error: The 'PassengerRoot' configuration option is not specified. This option is required, so please specify it. TIP: The correct value for this option was given to you by 'passenger-install-apache2-module'.
[Sun Apr 14 03:27:01.894726 2019] [mpm_prefork:notice] [pid 8215] AH00163: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50 configured -- resuming normal operations
[Sun Apr 14 03:27:01.894732 2019] [core:notice] [pid 8215] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND'
[Mon Apr 15 11:45:49.225967 2019] [mpm_prefork:notice] [pid 8215] AH00170: caught SIGWINCH, shutting down gracefully
[Mon Apr 15 11:53:58.265036 2019] [suexec:notice] [pid 24999] AH01232: suEXEC mechanism enabled (wrapper: /opt/rh/httpd24/root/usr/sbin/suexec)
[Mon Apr 15 11:53:58.265906 2019] [passenger:error] [pid 24999] *** Passenger could not be initialized because of this error: The 'PassengerRoot' configuration option is not specified. This option is required, so please specify it. TIP: The correct value for this option was given to you by 'passenger-install-apache2-module'.
[Mon Apr 15 11:53:58.293080 2019] [http2:warn] [pid 24999] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Mon Apr 15 11:53:58.293627 2019] [lbmethod_heartbeat:notice] [pid 24999] AH02282: No slotmem from mod_heartmonitor
[Mon Apr 15 11:53:58.329059 2019] [passenger:error] [pid 24999] *** Passenger could not be initialized because of this error: The 'PassengerRoot' configuration option is not specified. This option is required, so please specify it. TIP: The correct value for this option was given to you by 'passenger-install-apache2-module'.
[Mon Apr 15 11:53:58.335722 2019] [mpm_prefork:notice] [pid 24999] AH00163: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips mod_auth_kerb/5.4 Phusion_Passenger/4.0.50 configured -- resuming normal operations
[Mon Apr 15 11:53:58.335752 2019] [core:notice] [pid 24999] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND'
16 
vm039:/var/log/http24/error_log
[Mon Apr 15 11:53:58.329059 2019] [passenger:error] [pid 24999] *** Passenger could not be initialized because of this error: The 'PassengerRoot' configuration option is not specified. This option is required, so please specify it. TIP: The correct value for this option was given to you by 'passenger-install-apache2-module'.

This is our ood_config.yml file:

---
#
# Portal configuration
#

# The address and port to listen for connections on
# Example:
#     listen_addr_port: 443
# Default: null (don't add any more listen directives)
# listen_addr_port: 443

# The server name used for name-based Virtual Host
# Example:
#     servername: 'www.example.com'
# Default: null (don't use name-based Virtual Host)
# servername: null
servername: ondemand.bridges.psc.edu

# The port specification for the Virtual Host
# Example:
#     port: 8080
#Default: null (use default port 80 or 443 if SSL enabled)
#port: null

# List of SSL Apache directives
# Example:
#     ssl:
#       - 'SSLCertificateFile "/etc/pki/tls/certs/www.example.com.crt"'
#       - 'SSLCertificateKeyFile "/etc/pki/tls/private/www.example.com.key"'
# Default: null (no SSL support)
#ssl: null
ssl:
#  - 'SSLCertificateFile "/etc/pki/tls/certs/20180320.ondemand.bridges.psc.edu.crt"'
  - 'SSLCertificateFile "/etc/pki/tls/certs/20190409_ondemand_bridges_psc_edu_cert.cer"'
  - 'SSLCertificateKeyFile "/etc/pki/tls/private/ondemand.bridges.psc.edu.key"'
#  - 'SSLCertificateChainFile "/etc/pki/tls/certs/ca-bundle.trust.crt"'

# Root directory of log files (can be relative ServerRoot)
# Example:
#     logroot: '/path/to/my/logs'
# Default: 'logs' (this is relative to ServerRoot)
#logroot: 'logs'
logroot: '/var/log/ood'

# Root directory of the Lua handler code
# Example:
#     lua_root: '/path/to/lua/handlers'
# Default : '/opt/ood/mod_ood_proxy/lib' (default install directory of mod_ood_proxy)
#lua_root: '/opt/ood/mod_ood_proxy/lib'

# Verbosity of the Lua module logging
# (see https://httpd.apache.org/docs/2.4/mod/core.html#loglevel)
# Example:
#     lua_log_level: 'info'
# Default: null (use default log level)
#lua_log_level: null

# System command used to map authenticated-user to system-user
# Example:
#     user_map_cmd: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex --regex=''^(\w+)@example.com$'''
# Default: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex' (this echo's back auth-user)
#user_map_cmd: '/opt/ood/ood_auth_map/bin/ood_auth_map.regex'

# Redirect user to the following URI if fail to map there authenticated-user to
# a system-user
# Example:
#     map_fail_uri: '/register'
# Default: null (don't redirect, just display error message)
#map_fail_uri: null

# System command used to run the `nginx_stage` script with sudo privileges
# Example:
#     pun_stage_cmd: 'sudo /path/to/nginx_stage'
# Default: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage' (don't forget sudo)
#pun_stage_cmd: 'sudo /opt/ood/nginx_stage/sbin/nginx_stage'

# List of Apache authentication directives
# NB: Be sure the appropriate Apache module is installed for this
# Default: (see below, uses basic auth with an htpasswd file)
#auth:
#  - 'AuthType Basic'
#  - 'AuthName "private"'
#  - 'AuthUserFile "/opt/rh/httpd24/root/etc/httpd/.htpasswd"'
#  - 'RequestHeader unset Authorization'
#  - 'Require valid-user'

# Redirect user to the following URI when accessing root URI
# Example:
#     root_uri: '/my_uri'
#     # https://www.example.com/ => https://www.example.com/my_uri
# Default: '/pun/sys/dashboard' (default location of the OOD Dashboard app)
#root_uri: '/pun/sys/dashboard'

auth:
 - 'AuthType Basic'
 - 'AuthName "private"'
 - 'AuthBasicProvider ldap'
 - 'AuthLDAPURL "ldaps://valens.psc.edu:636 dxcldap.psc.edu:636/ou=bridges_users,dc=psc,dc=edu"'
 - 'AuthLDAPBindDN "cn=passwd_reader,ou=admins,dc=psc,dc=edu"'
 - 'AuthLDAPBindPassword "*********"'
 - 'AuthLDAPGroupAttribute memberUid'
 - 'AuthLDAPGroupAttributeIsDN off'
 - 'RequestHeader unset Authorization'
# - 'Require valid-user'
 - '<RequireAll>'
 - '  Require not user chakka aja56 biotech blood bradsol1 chandrau congma glymour gmarcais heewook jvs11 kangas kingsfor langmead midavis milnes mshao omedvede ruffalo smithn6 xiangj xuc13 xwangp zee5'
 - '  Require valid-user'
 - '</RequireAll>'

# Track server-side analytics with a Google Analytics account and property
# (see https://github.com/OSC/mod_ood_proxy/blob/master/lib/analytics.lua for
# information on how to setup the GA property)
# Example:
#     analytics:
#       url: 'http://www.google-analytics.com/collect'
#       id: 'UA-79331310-4'
# Default: null (do not track)
#analytics: null

#
# Publicly available assets
#

# Public sub-uri (available to public with no authentication)
# Example:
#     public_uri: '/assets'
# Default: '/public'
#public_uri: '/public'

# Root directory that serves the public sub-uri (be careful, everything under
# here is open to the public)
# Example:
#     public_root: '/path/to/public/assets'
# Default: '/var/www/ood/public'
#public_root: '/var/www/ood/public'

#
# Logout redirect helper
#

# Logout sub-uri
# Example
#     logout_uri: '/log_me_out'
# NB: If you change this, then modify the Dashboard app with the new sub-uri
# Default: '/logout' (the Dashboard app is by default going to expect this)
#logout_uri: '/logout'

# Redirect user to the following URI when accessing logout URI
# Example:
#     logout_redirect: '/oidc?logout=https%3A%2F%2Fwww.example.com'
# Default: '/pun/sys/dashboard/logout' (the Dashboard app provides a simple
# HTML page explaining logout to the user)
#logout_redirect: '/pun/sys/dashboard/logout'

#
# Reverse proxy to backend nodes
#

# Regular expression used for whitelisting allowed hostnames of nodes
# Example:
#     host_regex: '[\w.-]+\.example\.com'
# Default: '[^/]+' (allow reverse proxying to all hosts, this allows external
# hosts as well)
# host_regex: '[^/]+'
# Allow access to OPA and IB interfaces of bridges, dbmi, olympus compute nodes
# TODO: add access for OpenStack VM networks
host_regex: '(r|gpu|l|xl|or|om|on|ol|dgpu|dr)[0-9\-]+\.(opa|ib|pvt)\.bridges\.psc\.edu'
# host_regex: '*'

# Sub-uri used to reverse proxy to backend web server running on node that
# knows the full URI path
# Example:
#     node_uri: '/node'
# Default: null (disable this feature)
# node_uri: null
node_uri: "/node"

# Sub-uri used to reverse proxy to backend web server running on node that
# ONLY uses *relative* URI paths
# Example:
#     rnode_uri: '/rnode'
# Default: null (disable this feature)
# rnode_uri: null
rnode_uri: "/rnode"

#
# Per-user NGINX Passenger apps
#

# Sub-uri used to control PUN processes
# Example:
#     nginx_uri: '/my_pun_controller'
# Default: '/nginx'
#nginx_uri: '/nginx'

# Sub-uri used to access the PUN processes
# Example:
#     pun_uri: '/my_pun_apps'
# Default: '/pun'
#pun_uri: '/pun'

# Root directory that contains the PUN Unix sockets that the proxy uses to
# connect to
# Example:
#     pun_socket_root: '/path/to/pun/sockets'
# Default: '/var/run/nginx' (default location set in nginx_stage)
#pun_socket_root: '/var/run/nginx'

# Number of times the proxy attempts to connect to the PUN Unix socket before
# giving up and displaying an error to the user
# Example:
#     pun_max_retries: 25
# Default: 5 (only try 5 times)
#pun_max_retries: 5

#
# Support for OpenID Connect
#

# Sub-uri used by mod_auth_openidc for authentication
# Example:
#     oidc_uri: '/oidc'
# Default: null (disable OpenID Connect support)
#oidc_uri: null

# Sub-uri user is redirected to if they are not authenticated. This is used to
# *discover* what ID provider the user will login through.
# Example:
#     oidc_discover_uri: '/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_uri: null

# Root directory on the filesystem that serves the HTML code used to display
# the discovery page
# Example:
#     oidc_discover_root: '/var/www/ood/discover'
# Default: null (disable support for discovering OpenID Connect IdP)
#oidc_discover_root: null

#
# Support for registering unmapped users
#
# (Not necessary if using regular expressions for mapping users)
#

# Sub-uri user is redirected to if unable to map authenticated-user to
# system-user
# Example:
#     register_uri: '/register'
# Default: null (display error to user if mapping fails)
#register_uri: null

# Root directory on the filesystem that serves the HTML code used to register
# an unmapped user
# Example:
#     register_root: '/var/www/ood/register'
# Default: null (display error to user if mapping fails)
#register_root: null
17

The /var/log/ondemand-nginx/susan/error.log file contains the logs from Passenger apps:

Everything that the application writes to the stdout and stderr channels is logged to the log file.

and

Inside this log file you will find informational messages, errors, warnings and debugging messages generated by Passenger, as well as application output

Since its the Passenger convention to write to the NGNIX error.log file it is called error.log. But most are not actually errors.

We are looking into the problem you are experiencing but might not have a response till tomorrow.

18

A co-worker figured out the problem. Somehow when I ran the portal generator / updater, a bunch of lines were dropped from /opt/rh/httpd24/root/etc/httpd/conf.d/ood_portal.conf file. He diff’d the previous ood_portal.conf with the one saved by the generator/updater and found missing lines.

I’ve been changing the file /etc/ood/config_portal.yml so I think when I ran the generator, it overwrote the changes I made to the .yml file.

I think we’re good now. Sorry for the bother.

1 Like
19

I think this is a good example of why the current architecture with generating an ood_portal.conf file from the /etc/ood/config/ood_portal.yml is problematic.

I opened https://github.com/OSC/ondemand/issues/34 to formally track fixing this.