A number of users are reporting the following error when trying to get into the ondemand portal web service. Any ideas on how to fix this?
[emerg] bind() to unix:/var/run/ondemand-nginx/username/passenger.sock failed (98: Address already in use)
Can you check the states of the process’ attached to that file?
Something like this.
[jeff@3f2643b2ad92 /]$ lsof | grep 'jeff/passenger.sock'
nginx 159 jeff 6u unix 0x000000001133461b 0t0 543168 /var/run/ondemand-nginx/jeff/passenger.sock
[jeff@3f2643b2ad92 /]$ ps 159
PID TTY STAT TIME COMMAND
159 ? S 0:00 nginx: worker process
For me 159 is in a group of process’ like this. Seems to me nginx is in some bad state on your system?
4 S jeff 141 1 0 80 0 - 91556 - 15:08 ? 00:00:00 Passenger watchdog
0 S jeff 144 141 0 80 0 - 372620 - 15:08 ? 00:00:00 Passenger core
5 S root 154 1 0 80 0 - 25556 - 15:08 ? 00:00:00 nginx: master process (jeff) -c /var/lib/ondemand-nginx/config/puns/jeff.conf
5 S jeff 159 154 0 80 0 - 25557 - 15:08 ? 00:00:00 nginx: worker process
I’ve found that if I just remove that file an new set of nginx process’ will start so you’ll be left some extra processes hanging about, which we don’t want.
First I’d say see if sudo /opt/ood/nginx_stage/sbin/nginx_stage nginx_clean will clean any of these up. We may have to start killing off processes, but I don’t want to start until we know more. I don’t need to get output for all of them, just a single one that seems to be indicative of all the rest.
Thanks for your quick reply as always.
The first one doesn’t show any process associated with the reported users.
lsof | grep ‘username/passenger.sock’ returns nothing.
If I run that with my own username which actually works, I can see the processes associated with nginx on the ports – nothing for the reported user.
nginx 24115 root 6u unix 0xffff93b527fea400 0t0 398385922 /var/run/ondemand-nginx/username/passenger.sock
nginx 24117 username 6u unix 0xffff93b527fea400 0t0 398385922 /var/run/ondemand-nginx/username/passenger.sock
- Arash
Sorry, you’ll have to sudo lsof | grep otheruser/passegner.sock to see other folks’ open files.
I just ran it as root so. should be the same… sudo shows similar outcome.
Does the user have any processes running on that machine? I’d triple check the output of lsof, maybe by piping into a file and searching for the username in that file.
Checked. Piped the whole lsof to a file. Strangely enough, found nothing under those users… No processes running under reported users. Can see processes for users that are actually running on the webserver without issues.
This behavior doesn’t happen for all users – At least for multiple users I checked right now their access is OK.
Are there some end-user web server settings that may cause this?
I’m not sure how this could happen. If you’re sure the user has nothing running on that server, you can delete that passenger.sock file.
That file should just be recreated with no issue. Again, the worry here is hanging processes, so if there’s nothing running by those users, just removing that file should work just fine.
I deleted the whole user passenger folder for the reported user. The user could get to the web using other web-browsers like opera and Brave.
So the issue I think is related to the settings of the users’ web browser. So the error didn’t actually indicate a port-in-use on the machine side.
Thanks for your quick help as always!
Hey guys,
We had or have the same issue. FYI, Our IDAM path for OOD is convoluted, ie. AD creds with IPA, sssd, pwath and a module in Apache (cannot remember it’s name) and some users were created in AD by central IT with spaces in their name. 
In our case (a few years ago) to fix the usernames up the unix team had to remap usernames in IPA to short names so the user now has a long username from AD and a short name and unix UID for Linux.
They can SSH into hosts with either of the usernames so long as the username has no spaces in it, not so with OOD as we get the same error you reported if they use the long name.
Have not had time to figure it out as I am very new to OOD but I would be looking in code for usage of “getent” and “Id” calls intermixed against the users username in OOD code in the first instance. I think these commands return diff results in some cases like ours.
From our end we get around it by telling users to login with their short Linux username and it works
I.e jdoyle rather than “doyle john” or “cpeters” rather than “peterschris” . How are you validating user auth when users login to OOD.
Writing from the train on my mobile, apologies if this sounds a bit confusing.
Chris
@christopherwelsh thanks for chiming in! Yea we have this utility to map remote users to local ones. In the simplest case it’s 1:1 and we try to start a PUN as the $REMOTE_USER (whoever you tried to login as).
But you may be on to something here with user ids mapping to users.
I am not quite on the ball here but It smells like if we login with, what I term as the long username, then passenger trys to open a socket, fails and trys again? I.e run two sockets. The case for Short username which we map in IPA (long to short), it is fine. So somewhere the code logic is not managing this case perhaps?
PS. Love the doco this project writes up. Nice work.