In the file browser I can see my files but when I try to edit a file the file editor page is blank. Is there a way to turn on additional debugging?
Thanks!
In /var/log/ondemand-nginx/mknox/error.log I see:
App 1133 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
If you’re running ood 1.5, I’d suggest
ls -l /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb
and
rpm -qV ondemand-passenger
If the file is missing, and rpm complains about it, a yum reinstall ondemand-passenger might be the solution (for RedHat/CentOS systems anyway).
The file in question is part of the ondemand-passenger rpm (ondemand-passenger-5.3.7-2.el7.x86_64) in ood 1.5, but is not present in ood 1.3; I don’t have an ood 1.4 installation to check against, so no ideas there. Also, if you have SELinux
turned on, you might want to check your SEL logs to see if SEL prevented that file from being accessed.
Cheers,
Ric
Thanks for the tips.
We’re running ood version: 1.5.5
I was thinking that error message is coming from the app itself, as if it can’t access the file I’m trying to edit. I can use the “New File” button in the file manager to create files, but I can’t use the editor to then edit those newly created files.
The file is there:
[root@3675f683dd83 mknox]# ls -l /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb
-rw-r–r-- 1 root root 5172 Feb 26 19:25 /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb
[root@3675f683dd83 mknox]# rpm -qa | grep ondemand-passenger
ondemand-passenger-5.3.7-2.el7.x86_64
Does
[root@ ~]# sestatus
SELinux status: disabled
If not, I’d try configuring SELinux as “disabled” before doing anything else.
Also, if you run
rpm -qV ondemand-passenger
it will check protections, groups, owner, etc. of all the package members and report discrepancies; no output means no discrepancies…
Outside of that, I don’t have any useful ideas,
Ric
Okay, thanks for the help! I’ll keep poking at it.
Is there any way to turn on additional logging?
[root@3675f683dd83 /]# sestatus
SELinux status: disabled
[root@3675f683dd83 /]# rpm -qV ondemand-passenger
[root@3675f683dd83 /]#
@mknox there is no additional logging available for the File Editor. Can you please post the full stack trace that you are seeing in the Nginx error log?
Below is what I see in /var/log/ondemand-nginx/mknox/error.log after trying to use the file editor a couple times.
I also tried running strace against the nginix process and saw the following:
[{EPOLLIN, {u32=1146325872, u64=94194074094448}}], 512, -1) = 1
accept4(6, {sa_family=AF_LOCAL, NULL}, [2], SOCK_NONBLOCK) = 3
epoll_ctl(9, EPOLL_CTL_ADD, 3, {EPOLLIN|EPOLLRDHUP|EPOLLET, {u32=1146326801, u64=94194074095377}}) = 0
epoll_wait(9, [{EPOLLIN, {u32=1146326801, u64=94194074095377}}], 512, 60000) = 1
recvfrom(3, “GET /pun/sys/file-editor/edit/ho”…, 1024, 0, NULL, NULL) = 1024
recvfrom(3, “X-Forwarded-For: 10.2.163.159\r\nX”…, 8192, 0, NULL, NULL) = 118
stat(“/var/www/ood/apps/sys/file-editor/public/edit/home/mknox/afilefromood”, 0x55ab4446b410) = -1 ENOENT (No such file or directory)
stat(“/var/www/ood/apps/sys/file-editor/public/edit/home/mknox/afilefromood.html”, 0x55ab4458b5c0) = -1 ENOENT (No such file or directory)
stat(“/var/www/ood/apps/sys/file-editor/passenger_wsgi.py”, 0x55ab4446bda0) = -1 ENOENT (No such file or directory)
stat(“/var/www/ood/apps/sys/file-editor/config.ru”, {st_mode=S_IFREG|0644, st_size=154, …}) = 0
epoll_ctl(9, EPOLL_CTL_MOD, 3, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1146326801, u64=94194074095377}}) = 0
socket(AF_LOCAL, SOCK_STREAM, 0) = 8
ioctl(8, FIONBIO, [1]) = 0
epoll_ctl(9, EPOLL_CTL_ADD, 8, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1146327033, u64=94194074095609}}) = 0
connect(8, {sa_family=AF_LOCAL, sun_path=“/tmp/passenger.n4Nt4os/agents.s/core”}, 110) = 0
getsockopt(8, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
writev(8, [{“GET /pun/sys/file-editor/edit/ho”…, 2365}], 1) = 2365
epoll_wait(9, [{EPOLLOUT, {u32=1146326801, u64=94194074095377}}, {EPOLLOUT, {u32=1146327033, u64=94194074095609}}], 512, 12000000) = 2
epoll_wait(9, [{EPOLLOUT, {u32=1146327033, u64=94194074095609}}], 512, 12000000) = 1
epoll_wait(9, [{EPOLLIN|EPOLLOUT, {u32=1146327033, u64=94194074095609}}], 512, 12000000) = 1
recvfrom(8, “HTTP/1.1 404 Not Found\r\nStatus: “…, 16384, 0, NULL, NULL) = 15601
writev(3, [{“HTTP/1.1 404 Not Found\r\nContent-”…, 807}, {“3a16\r\n”, 6}, {”\n\n\n “…, 14870}, {”\r\n”, 2}], 4) = 15685
epoll_wait(9, [{EPOLLIN|EPOLLOUT|EPOLLHUP|EPOLLRDHUP, {u32=1146327033, u64=94194074095609}}], 512, 11999996) = 1
recvfrom(8, “”, 16384, 0, NULL, NULL) = 0
close(8) = 0
writev(3, [{“0\r\n\r\n”, 5}], 1) = 5
write(5, “unix: - mknox [25/Apr/2019:05:02”…, 308) = 308
close(3) = 0
epoll_wait(9,
[ N 2019-04-25 04:52:21.4122 15411/T1 age/Cor/CoreMain.cpp:1310 ]: Starting Passenger core…
[ N 2019-04-25 04:52:21.4123 15411/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2019-04-25 04:52:21.4870 15411/T1 age/Cor/CoreMain.cpp:985 ]: Passenger core online, PID 15411
App 15466 output: Rails Error: Unable to access log file. Please ensure that /var/www/ood/apps/sys/dashboard/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /var/www/ood/apps/sys/dashboard/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
App 15466 output: [2019-04-25 04:52:23 +0000 ] WARN “Error parsing announcement file ‘/etc/ood/config/announcement.md’: No such file or directory @ rb_sysopen - /etc/ood/config/announcement.md”
App 15466 output: [2019-04-25 04:52:23 +0000 ] WARN “Error parsing announcement file ‘/etc/ood/config/announcement.yml’: No such file or directory @ rb_sysopen - /etc/ood/config/announcement.yml”
App 15466 output: [2019-04-25 04:52:23 +0000 ] INFO “method=GET path=/pun/sys/dashboard/ format=html controller=DashboardController action=index status=200 duration=64.91 view=36.37”
[ N 2019-04-25 04:52:24.3672 15411/T5 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
App 15566 output: Rails Error: Unable to access log file. Please ensure that /var/www/ood/apps/sys/file-editor/log/production.log exists and is writable (ie, make it writable for user and group: chmod 0664 /var/www/ood/apps/sys/file-editor/log/production.log). The log level has been raised to WARN and the output directed to STDERR until the problem is fixed.
App 15566 output: [2019-04-25 04:52:35 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit format=html controller=PagesController action=index status=200 duration=11.61 view=11.09”
App 15566 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
App 15566 output: [2019-04-25 04:53:40 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/home/mknox/fieldiag.sh format=html controller=PagesController action=index status=404 duration=3.77 view=1.36”
App 15566 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
App 15566 output: [2019-04-25 04:54:16 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/home/mknox/fieldiag.sh format=html controller=PagesController action=index status=404 duration=2.23 view=1.31”
App 15566 output: [2019-04-25 04:55:12 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/tmp/yum.log format=html controller=PagesController action=index status=404 duration=1.49 view=1.30”
App 15566 output: [2019-04-25 04:55:56 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/tmp/yum.log format=html controller=PagesController action=index status=404 duration=1.52 view=1.34”
App 15566 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
App 15566 output: [2019-04-25 04:56:35 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/tmp/yum.log format=html controller=PagesController action=index status=404 duration=2.43 view=1.47”
[ N 2019-04-25 04:57:23.0794 15411/T4 age/Cor/CoreMain.cpp:1117 ]: Checking whether to disconnect long-running connections for process 15466, application /var/www/ood/apps/sys/dashboard (production)
App 15566 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
App 15566 output: [2019-04-25 04:58:12 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/home/mknox/afilefromood format=html controller=PagesController action=index status=404 duration=3.02 view=1.34”
App 15566 output: /opt/ood/ondemand/root/usr/share/passenger/helper-scripts/rack-loader.rb: No such file or directory - file
App 15566 output: [2019-04-25 04:58:43 +0000 sys/file-editor] INFO “method=GET path=/pun/sys/file-editor/edit/home/mknox/afilefromood format=html controller=PagesController action=index status=404 duration=3.00 view=1.24”
We tracked the issue down to this:
fileinfo = %x[ file -b --mime-type #{@pathname.to_s.shellescape} ]
in /var/www/ood/apps/sys/file-editor/app/controllers/pages_controller.rb
If the “file” utility is not installed the editor fails to load. The “file” application should probably be listed as a dependency when installing the open ondemand rpm.
@mknox good find. I’m curious what distro you are running on that doesn’t have file by default? According to yum whatprovides file it is part of CentOS base.
On a related note we had something similar happen where the dependency on netcat wasn’t well publicized; I’m going to add a project to track down all the places we are shelling out to another program so that we have a definitive list.
The host is Ubuntu 18.04 but the portal is running inside a docker container which is built “FROM centos:7”.
Mike
Ah, that makes sense then. I’d be very curious to know how the rest of your installation goes. As you may know we have recently had some interest from sites running Ubuntu, and I was thinking that the fastest way to offer OnDemand to them would be through containerization.
So far everything else (slurm jobs, file browser, interactive apps, authentication) has worked perfectly. Once we get it cleaned up I’ll share the Dockerfile we’re using to build the ood container.
Mike
@mknox any update on the containerization effort?
Is there an OOD git repo where we could send our dockerfile example?
@mknox https://github.com/OSC/ood-images is the logical place for it, and in particular I’d love to see how it compares with: https://github.com/OSC/ood-images/blob/master/docker/Dockerfile.
