Hello,
I installed OOD using the recipe given here 1. Install Software — Open OnDemand 3.0.0 documentation (osc.github.io) but using this I could only install the Apache modules and not the nginx. I later installed nginx separately. Now I am having the issue of Apache running fine but nginx giving the following error message. Also, the command nginx-stage doesn’t seem to exist for me.
Could anyone please throw some light on this issue and how to proceed ? I am still in the stages of bringing up the ood service.
Thanks in advance.
systemctl status nginx
returns
Apr 13 16:16:14 oodheadnode.hpcc.ttu.edu systemd[1]: Starting nginx - high performance web server...
Apr 13 16:16:14 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Apr 13 16:16:15 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Apr 13 16:16:15 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Apr 13 16:16:16 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Apr 13 16:16:16 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
Apr 13 16:16:17 oodheadnode.hpcc.ttu.edu nginx[3570293]: nginx: [emerg] still could not bind()
Apr 13 16:16:17 oodheadnode.hpcc.ttu.edu systemd[1]: nginx.service: Control process exited, code=exited status=1
Apr 13 16:16:17 oodheadnode.hpcc.ttu.edu systemd[1]: nginx.service: Failed with result 'exit-code'.
Apr 13 16:16:17 oodheadnode.hpcc.ttu.edu systemd[1]: Failed to start nginx - high performance web server.
You don’t need to install a separate nginx binary. We should have installed our own ondemand-nginx in a different location and we setup all the configs for routing.
Remove the nginx that you’ve installed. It won’t boot up because apache’s already using the port 80, so it’s only going to conflict with apache.
Hi Jeff,
Thanks for the insight. I have removed the separate nginx installation. But when I try to access the respective ondemand webpage through a browser, it mentions:
Error -- can't find user for <username>
Run 'nginx-stage -- help' to see a full list of available command line options
However, I don’t seem to have that command available.
So far
I believe nginx_stage’s full path is /opt/ood/nginx_stage/sbin/nginx_stage.
Do not edit /opt/ood/nginx_stage/share/nginx_stage_example.yml. This file doesn’t do anything. If you need to update settings update /etc/ood/config/nginx_stage.yml.
This error is actually quite simple - the user doesn’t exist on the system. How are users provisioned to this system? Are they local users or LDAP users? Seems like you need to either tie into your LDAP or create the local users - whichever scheme you’re trying to go for (LDAP users I hope).
When I say it’s simple, I mean that user doesn’t exist on the machine. It’s not an OnDemand issue. You couldn’t ssh to that machine as that user or if you were root you couldn’t su - <username> or even check their id with id <username>.
Hi Jeff,
Thanks ! I found the nginx stage in /opt/ood/nginx_stage/sbin/nginx_stage location.
You’re right, I actually uncommented certain parts in the /etc/ood/config/nginx_stage.yml and not the */nginx_stage_example.yml file. I just wanted to use the default version of it. So is it necessary to comment out the lines in /etc/ood/config/nginx_stage.yml file at all ?
The user in this case was from a flat file based authentication file I created for testing purpose only. Finally we don’t use the LDAP scheme usually and were thinking of using CILogon type setup with either shibboleth or CAS type of authentication. Would you happen to have any tips on getting a CILogon based system set up ?
We use CILogon with Keycloak with some degree of sucess. So we configure Keycloak to look at our LDAP and you have to federate your user to that local OSC user. As an example I’m ohrstrom.4 at Ohio State University, but my Ohio SuperComputer Center user (what’s in our LDAP) is johrstrom. Keycloak handles that mapping for us when I login through OSU.
I’m not sure how to setup CILogon with shibboleth or CAS, though we have some ACCESS (previously EXSEDE) documentation for the same.
Hi Jeff,
Thanks a lot for your response.
I am still figuring out the proper user authentication method for our university.
Meanwhile, I tried running the ood nginx command which returns a strange error line at the very end. I wonder what might be causing it.
[root@oodheadnode ~]# /opt/ood/nginx_stage/sbin/nginx_stage --help
Traceback (most recent call last):
14: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
13: from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
12: from /opt/ood/nginx_stage/lib/nginx_stage.rb:25:in `<top (required)>'
11: from /opt/ood/nginx_stage/lib/nginx_stage.rb:38:in `<module:NginxStage>'
10: from /opt/ood/nginx_stage/lib/nginx_stage.rb:38:in `extend'
9: from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:411:in `extended'
8: from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:494:in `set_default_configuration'
7: from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:501:in `read_configuration'
6: from /usr/share/ruby/psych.rb:577:in `load_file'
5: from /usr/share/ruby/psych.rb:577:in `open'
4: from /usr/share/ruby/psych.rb:578:in `block in load_file'
3: from /usr/share/ruby/psych.rb:277:in `load'
2: from /usr/share/ruby/psych.rb:390:in `parse'
1: from /usr/share/ruby/psych.rb:456:in `parse_stream'
/usr/share/ruby/psych.rb:456:in `parse': (/etc/ood/config/nginx_stage.yml): did not find expected key while parsing a block mapping at line 19 column 1 (Psych::SyntaxError)
Additionally, at this point the nginx must be able to show some output on the screen if I try to open it using a web browser right ? But the browser shows this, which are the same lines as the previous command line options.
Error -- /usr/share/ruby/psych.rb:456:in `parse': (/etc/ood/config/nginx_stage.yml): did not find expected key while parsing a block mapping at line 19 column 1 (Psych::SyntaxError)
from /usr/share/ruby/psych.rb:456:in `parse_stream'
from /usr/share/ruby/psych.rb:390:in `parse'
from /usr/share/ruby/psych.rb:277:in `load'
from /usr/share/ruby/psych.rb:578:in `block in load_file'
from /usr/share/ruby/psych.rb:577:in `open'
from /usr/share/ruby/psych.rb:577:in `load_file'
from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:501:in `read_configuration'
from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:494:in `set_default_configuration'
from /opt/ood/nginx_stage/lib/nginx_stage/configuration.rb:411:in `extended'
from /opt/ood/nginx_stage/lib/nginx_stage.rb:38:in `extend'
from /opt/ood/nginx_stage/lib/nginx_stage.rb:38:in `<module:NginxStage>'
from /opt/ood/nginx_stage/lib/nginx_stage.rb:25:in `<top (required)>'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
from /usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
I am trying to do some online searches and explore these but would you happen to know what might be off here ?
Thank you.
After fixing that issue the same nginx_stage command returns the desired output I feel, as follows:
[root@oodheadnode ~]# /opt/ood/nginx_stage/sbin/nginx_stage --help
Usage: nginx_stage COMMAND [OPTIONS]
Commands:
pun # Generate a new per-user nginx config and process
app # Generate a new nginx app config and reload process
app_reset # Reset all staged app configs with the current template
app_list # List all staged app configs
app_clean # Clean up any staged app configs that point to deleted apps
nginx # Generate/control a per-user nginx process
nginx_show # Show the details for a given per-user nginx process
nginx_list # List all user running PUNs
nginx_clean # Clean all user running PUNs with no active connections
General options:
-h, --help # Show this help message
-v, --version # Show version
All commands can be run with -h (or --help) for more information.
However, I think at this point I am missing something in the /etc/ood/config/ood_portal.yml I believe as by running the following command I run into an error message still:
[root@oodheadnode ~]# /opt/ood/nginx_stage/sbin/nginx_stage pun --user 'flat_file_username' --app-init-url 'https://ondemand.hpcc.ttu.edu/nginx/init?redir=$http_x_forwarded_escaped_uri'
can't find user for flat_file_username
Run 'nginx_stage --help' to see a full list of available command line options.
What are you trying to accomplish running nginx_stage? Apache will run that command for you to boot up the PUN. What’s the issue you’re trying to diagnose?
Hi Jeff,
I am trying to basically trying to connect a given user’s (in this case my flat file with the username and password, we will later add the university-wide authentication) account to take me to the OOD’s default page via nginx. When I try to open the server’s website then it just returns:
Error -- can't find user for flat_file_username
Run 'nginx_stage --help' to see a full list of available command line options.
So I was wondering if the nginx_stage.yml file needed some un-commenting and addition of relevant parameters. The Apache module doesn’t know of the flat_file’s user perhaps. Or rather which part of configuration process pertain to making OOD aware of a user either from a flat file or any authentication step I might use ? That is what I am trying to understand.
Thanks.
It seems like you need to setup user mapping. This doc page may help. Once you authenticate (however you’ve got that work) you’ll be assigned a REMOTE_USER. This, you need to map to a local user.
In the examples I’m mapping Annie.Oakley@osc.edu to the local user annie.oakley.
I’m not sure how you’re currently getting the value flat_file_username, but the system is trying to start a process tree as that user. I.e., it’s trying to su - flat_file_username which isn’t working because that’s not a real user.
Whoever you’re becoming (if flat_file_username is a pseudonym for the actual username you’re trying to become) has to be an actual user on the system.
Hi Jeff,
Thanks for your response.
Yes I have the university-wide authentication to work on, and I am presently going through the link you forwarded me. It has a few options.
I was pursuing the flat_file based authentication just to get to an OOD defualt webpage. I did that adding the following to the /etc/ood/config/ood_portal.yml file:
and putting in a relevant file in that location and I used scl enable ondemand – htpasswd -c /etc/httpd/.htpasswd $username command to direct the service to search for a user there. It does go through the “authentication” part, it prompts me and I put in the details.
Having said that, depending on what university-wide authentication system we end up using I might have to do the mapping accordingly. Do you have any suggestion in what mapping you use for the CILogon with Keycloak method in your case ?
Thank you.
Keycloak should take care of it all for you by pulling the preferred_username from LDAP. All your users should be legitimate users in LDAP and when Keycloack connects to it and reads attributes from it it can return the correct user.
To your current issue - and I always have to state that basic auth is extremely insecure, so be aware of that - let’s say you created the user annie.oakley through
htpasswd -c/etc/httpd/.htpasswd annie.oakley
That’s fine - apache knows about this user and can authenticate them. However, the user does not exist on the system. I.e., you can’t su - annie.oakley or id annie.oakley.
So you’d need to create the user manually though a useradd command.
sudo useradd annie.oakley
I doubt very much you’d want to use AuthUserFile with an actual LDAP user, but that’s an option too. Likely a bad option, but still.
Hi Jeff,
Alright ! That’s comforting to hear that Keycloack based systems automatically pull user details from LDAP. Our university IT support suggested Shibboleth. I am installing and configuring the corresponding modules.
Thanks a lot for the insight regarding the flat file authentication. I see now, the user has to be created in the local system. That is very helpful for my understanding. I was beginning to doubt that issue but now I got it fully.
Thanks a lot.
Regards.
Dear Jeff,
I installed Shibboleth modules from RPM using the process described here - RPMInstall - Service Provider 3 - Confluence (atlassian.net).
I then restarted the Apache service and it fails to restart.
I was thinking of upgrading to OnDemand version 3 (latest one) from version 2 on which I was working.
Do you think that is a wise decision or would you recommend checking something in addition to that before going for the upgrade ?
Also, what would you say is the safe/correct way to perform this upgrade ?
Thank you.
Your issues aren’t likely due to the version. You can upgrade, and that’s fine, but I don’t believe that it’ll resolve this issue specifically.
What the issue you’re having - I can’t currently tell, but logs in /var/log/httpd will indicate the issue.
The 3.0 release notes should specify what you need to do to upgrade. And/or use automation which should take care of everything. We have support for puppet and ansible.
Hi Jeff,
The specific error line is the following:
httpd: Syntax error on line 357 of /etc/httpd/conf/httpd.conf: Syntax error on line 13 of /etc/httpd/conf.d/shib.conf: Cannot load /usr/lib64/shibboleth/mod_s>
When I checked the mentioned lines in the respective files, they just were these:
line 357 of /etc/httpd/conf/httpd.conf
IncludeOptional conf.d/*.conf
and
line 13 of /etc/httpd/conf.d/shib.conf
LoadModule mod_shib /usr/lib64/shibboleth/mod_shib_24.so
I was therefore thinking of a version mismatch between the shibboleth and httpd.
Thank you.