Nginx_stage hangs on nginx_show

Greetings –

We’re having reports that running sessions and new sessions had interrupted access in the past hour. When attempting to using ‘nginx_stage nginx_show --user=xxx123’ the process hangs. Any thoughts? This is the steady state – all attempts for multiple ‘users’ have hung since the issue was reported. Thanks

[root@wmaster ondemand-nginx]# /opt/ood/nginx_stage/sbin/nginx_stage nginx_show --user=wxy215
User: wxy215
Instance: 6274
^CTraceback (most recent call last):
10: from -e:1:in `’

     9: from /opt/ood/nginx_stage/lib/nginx_stage/application.rb:41:in `start'
     8: from /opt/ood/nginx_stage/lib/nginx_stage/generator.rb:93:in `invoke'
     7: from /opt/ood/nginx_stage/lib/nginx_stage/generator.rb:93:in `each'
     6: from /opt/ood/nginx_stage/lib/nginx_stage/generator.rb:93:in `block in invoke'
     5: from /opt/ood/nginx_stage/lib/nginx_stage/generator.rb:93:in `instance_eval'
     4: from /opt/ood/nginx_stage/lib/nginx_stage/generators/nginx_show_generator.rb:32:in `block in <class:NginxShowGenerator>'
     3: from /opt/ood/nginx_stage/lib/nginx_stage/generators/nginx_show_generator.rb:32:in `new'
     2: from /opt/ood/nginx_stage/lib/nginx_stage/socket_file.rb:13:in `initialize'
     1: from /opt/ood/nginx_stage/lib/nginx_stage/socket_file.rb:48:in `get_processes'

/opt/ood/nginx_stage/lib/nginx_stage/socket_file.rb:48:in ``': Interrupt

The situation did resolve and regular functionality is observed. Still open to thoughts. While nginx_stage was hanging, I was able to login to a dashboard session, and start a shell app, and experience other standard functionality.

Ultimately, I took no action, so will look into the logs to see if there is a record of how and what occurred.

Cheers

Hi Emily.

Thank for the post. I am glad that it ultimately resolved. I would love to see the Ondemand Server logs during the time that this was happening. Almost sounds like the OnDemand Server (apache) had an unusual load, but just guessing of course.

Thanks,
-gerald

Hi, Gerald – Thanks for the response. What log is of particular interest? The access_log includes an interest list of connections, of which here is an example:
83.212.126.59 - - [20/Mar/2023:14:53:45 -0400] “GET http://129.22.106.12:80/phpmy-admin/scripts/setup.php HTTP/1.0” 301 267 “-” “-”

The IP resolves to $ nslookup 83.212.126.59
59.126.212.83.in-addr.arpa name = snf-60081.vm.okeanos-global.grnet.gr

This is date stamped later than reports of problems began. We think now that the problems with access were timeouts due to a mounted file system becoming unavailable. This set of queries from, seemingly, greece, is a bit puzzling. That IP does not match any of our end-user logs. Here’s the full deets:
83.212.126.59 - - [20/Mar/2023:14:53:45 -0400] “GET http://129.22.106.12:80/phpmy-admin/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:45 -0400] “GET http://129.22.106.12:80/mysqlmanager/scripts/setup.php HTTP/1.0” 301 268 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:45 -0400] “GET http://129.22.106.12:80/pma2005/scripts/setup.php HTTP/1.0” 301 263 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/PMA2005/scripts/setup.php HTTP/1.0” 301 263 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/db/scripts/setup.php HTTP/1.0” 301 258 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/php-myadmin/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/phpmanager/scripts/setup.php HTTP/1.0” 301 266 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/mysql-admin/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/p/m/a/scripts/setup.php HTTP/1.0” 301 261 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/sqlmanager/scripts/setup.php HTTP/1.0” 301 266 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/admin/phpmyadmin/scripts/setup.txt HTTP/1.0” 301 272 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/phpMyAdmin2/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/dbadmin/scripts/setup.php HTTP/1.0” 301 263 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/_phpMyAdmin/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/webdb/scripts/setup.php HTTP/1.0” 301 261 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/pma/scripts/setup.php HTTP/1.0” 301 259 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.5.7-pl1/scripts/setup.php HTTP/1.0” 301 276 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/php/scripts/setup.php HTTP/1.0” 301 259 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/admin/scripts/setup.php HTTP/1.0” 301 261 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/webadmin/scripts/setup.php HTTP/1.0” 301 264 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/secret/phpmyadmin/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/secret123/phpmyadmin/scripts/setup.php HTTP/1.0” 301 276 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.3/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/websql/scripts/setup.php HTTP/1.0” 301 262 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:46 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.10.3/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpma/scripts/setup.php HTTP/1.0” 301 261 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/myadmin/scripts/setup.php HTTP/1.0” 301 263 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2/scripts/setup.php HTTP/1.0” 301 268 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/web/phpMyAdmin/scripts/setup.php HTTP/1.0” 301 270 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.7/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/mysql/scripts/setup.php HTTP/1.0” 301 261 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/php-my-admin/scripts/setup.php HTTP/1.0” 301 268 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/typo3/phpmyadmin/scripts/setup.php HTTP/1.0” 301 272 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.1.2/scripts/setup.php HTTP/1.0” 301 275 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/mysqladmin/scripts/setup.php HTTP/1.0” 301 266 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.5.5-pl1/scripts/setup.php HTTP/1.0” 301 276 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpmyadmin/scripts/setup.php HTTP/1.0” 301 266 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/admin/pma/scripts/setup.php HTTP/1.0” 301 265 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.5.5/scripts/setup.php HTTP/1.0” 301 272 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/PHPMYADMIN/scripts/setup.php HTTP/1.0” 301 266 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/SQL/scripts/setup.php HTTP/1.0” 301 259 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin3/scripts/setup.php HTTP/1.0” 301 267 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/MyAdmin/scripts/setup.php HTTP/1.0” 301 263 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.5.4/scripts/setup.php HTTP/1.0” 301 272 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/sqlweb/scripts/setup.php HTTP/1.0” 301 262 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.10.2/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.9.2/scripts/setup.php HTTP/1.0” 301 275 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.0/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.8.0.2/scripts/setup.php HTTP/1.0” 301 274 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.10.0.2/scripts/setup.php HTTP/1.0” 301 275 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin-2.11.4/scripts/setup.php HTTP/1.0” 301 273 “-” “-”
83.212.126.59 - - [20/Mar/2023:14:53:47 -0400] “GET http://129.22.106.12:80/phpMyAdmin/scripts/setup.php HTTP/1.0” 301 266 “-” “-”

After thinking about it, let’s let it ride for now. If this happens again, we can do some troubleshooting while it’s happening.

Thanks,
-gerald

I agree, Gerald – there’s nothing actionable without more information.
Thanks for your attention
Best wishes,
~ Em
(and apologies for my error in salutation yesterday)

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.