Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
I have a simplex ZUMspot on a Pi Zero W board (Current rev is Pi-Star:3.4.17/Dashboard:20191220/ZUMspot v1.4.16 and updates are current) and the unit appears to work normally except for one problem, and that is after some period of time (30 min to several hours, appears to be random) the HTTP interface becomes unreachable, sometimes with a '504 Gateway Time-out' error and sometimes no response at all. HTTP operation is completely normal immediately after initial boot or a reboot, but the problem will reoccur predictably some period of time thereafter. This does not seem to be a configuration error since the unit always works normally (for a while) after boot. There does not seem to be any IP connectivity issues since when the problem occurs connectivity to Brandmeister is still up, calls through the hotspot work normally, and I can access the unit locally via ssh. I have even tried restarting nginx when the problem occurs but that action has no effect. WiFi power saving has been turned but off with no improvement. Also I have checked available RAM and disk space when the problem occurs but everything looks normal.
When the problem occurs the unit appears to be working normally in all ways except for the lack of connectivity on the HTTP connection. This might suggest an external problem but I have tried multiple PC and browsers, even a different router, but the problem remains unaffected so I can only assume that the fault is in fact with the hotspot.
Been pulling my hair out trying to troubleshoot this, not sure where else to look?
Thanks
When the problem occurs the unit appears to be working normally in all ways except for the lack of connectivity on the HTTP connection. This might suggest an external problem but I have tried multiple PC and browsers, even a different router, but the problem remains unaffected so I can only assume that the fault is in fact with the hotspot.
Been pulling my hair out trying to troubleshoot this, not sure where else to look?
Thanks
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
Unclear:
Are you accessing the web page from /outside/ the LAN? Since SSH from /inside/ is working, I'd tend to think HTTP from /inside/ might also work.
The answer will isolate where to look... If HTTP /inside/ the LAN is failing, then I'd be investigating the server and any common components (router). If only from outside the LAN, you may need to work with your ISP or others
https://developer.mozilla.org/en-US/doc ... Status/504
... implicates proxy configurations, so if some external server handles connection to your node, the problem may lie between it and your node.
Are you accessing the web page from /outside/ the LAN? Since SSH from /inside/ is working, I'd tend to think HTTP from /inside/ might also work.
The answer will isolate where to look... If HTTP /inside/ the LAN is failing, then I'd be investigating the server and any common components (router). If only from outside the LAN, you may need to work with your ISP or others
https://developer.mozilla.org/en-US/doc ... Status/504
... implicates proxy configurations, so if some external server handles connection to your node, the problem may lie between it and your node.
--
AF6VN
Dennis L Bieber
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
I am attempting to access from inside the LAN. And that's what's so confusing since there really are no network elements between me and the hotspot other than the router, but I have tried a second router with the same results. Add to that different PCs and browsers with no change and it would seem to pretty positively isolate the problem to the hotspot itself, and since I still have IP connectivity (evidenced by ability to ping, access via putty, etc.) when HTTP access is down it would appear that some Pi-Star process critical to HTTP access must be failing. As above I've tried restarting nginx when the problem occurs but that doesn't bring the interface back up, nothing seems to short of a complete reboot.
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
504 Gateway timeout is an HTTP error code. Coming from NGINX web server on Pi-Star. The "gateway" its talking about is that the webserver is trying to talk to the PHP data engine that renders the page. For some reason the php-fpm data engine is timing out.
If you ssh into the ZUMspot does it appear to be under high cpu load for some reason? Only time I saw this happen is when we had a pistar setup at an event and too many people were connecting to the dashboard to monitor who was talking and it overloaded the nginx webserver. I doubt that is your problem specifically but got to be some kind of resource exhaustion going on.
When its having the problem If you ssh into it and run 'top' at command line what do the top numbers look like. Should be something like this.
Also what are the two top processes in the list and their %CPU and %MEM.
If you ssh into the ZUMspot does it appear to be under high cpu load for some reason? Only time I saw this happen is when we had a pistar setup at an event and too many people were connecting to the dashboard to monitor who was talking and it overloaded the nginx webserver. I doubt that is your problem specifically but got to be some kind of resource exhaustion going on.
When its having the problem If you ssh into it and run 'top' at command line what do the top numbers look like. Should be something like this.
Code: Select all
top - 20:40:45 up 8 min, 1 user, load average: 1.02, 0.91, 0.56
Tasks: 89 total, 1 running, 88 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.7 us, 5.4 sy, 0.0 ni, 89.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 479.8 total, 279.0 free, 129.0 used, 71.8 buff/cache
MiB Swap: 512.0 total, 491.5 free, 20.5 used. 290.4 avail Mem
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
Before the problem arises I see a few (from one to four at any given time) php scripts running, and then after the problem arises I see none so yes, the scripts (or whatever process generates them) crashing is apparently the problem. I didn't notice any unusually high CPU utilization associated with the failure but it's also possible there was some event that I missed.
I have had a few static TGs up in the past and as an experiment I terminated all of them and the problem seemed to go away, or at least I still had HTTP access the following morning which is longer than it's ever gone. I never saw any unusually CPU utilization while the static links were up so I didn't associate it with the web access issue but it appears they may be related... or were related because as an experiment I reestablished the static TGs to see if I could replicate the failure and thus far have been unable. I will let things run a little longer but so far no more failures, even with the static TGs up (again, not sure if there ever was any relationship to this and the php issue.) I will continue to monitor and see if the problem will resurface. Odd because it was 100% reproducible in the past.
I have had a few static TGs up in the past and as an experiment I terminated all of them and the problem seemed to go away, or at least I still had HTTP access the following morning which is longer than it's ever gone. I never saw any unusually CPU utilization while the static links were up so I didn't associate it with the web access issue but it appears they may be related... or were related because as an experiment I reestablished the static TGs to see if I could replicate the failure and thus far have been unable. I will let things run a little longer but so far no more failures, even with the static TGs up (again, not sure if there ever was any relationship to this and the php issue.) I will continue to monitor and see if the problem will resurface. Odd because it was 100% reproducible in the past.
Code: Select all
top - 00:24:53 up 3 min, 1 user, load average: 1.01, 0.69, 0.30
Tasks: 106 total, 1 running, 104 sleeping, 0 stopped, 1 zombie
%Cpu(s): 20.5 us, 32.9 sy, 0.0 ni, 45.0 id, 0.0 wa, 0.0 hi, 1.6 si, 0.0 st
KiB Mem: 493252 total, 147172 used, 346080 free, 7620 buffers
KiB Swap: 0 total, 0 used, 0 free. 89140 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2156 www-data 20 0 124164 9512 7016 S 6.1 1.9 0:00.19 php5-fpm
1256 www-data 20 0 124616 11672 9068 S 5.1 2.4 0:02.09 php5-fpm
1110 www-data 20 0 124880 12564 9776 S 3.8 2.5 0:03.55 php5-fpm
997 root 10 -10 21036 10940 2600 S 3.5 2.2 0:07.14 MMDVMHost
1131 www-data 20 0 124616 11872 9180 S 1.6 2.4 0:02.25 php5-fpm
1089 pi-star 20 0 5012 2384 2004 R 1.3 0.5 0:01.50 top
217 root 20 0 0 0 0 S 1.0 0.0 0:00.62 kworker/u2:2
3 root 20 0 0 0 0 S 0.6 0.0 0:00.69 ksoftirqd/0
65 root -51 0 0 0 0 S 0.3 0.0 0:00.22 irq/86-mmc1
945 www-data 20 0 15668 4120 3072 S 0.3 0.8 0:00.06 nginx
1067 root 20 0 8816 4996 4368 S 0.3 1.0 0:00.84 sshd
1 root 20 0 5964 4124 2712 S 0.0 0.8 0:05.94 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
4 root 20 0 0 0 0 S 0.0 0.0 0:00.13 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.25 kworker/u2:0
7 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
8 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kdevtmpfs
9 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 oom_reaper
12 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kcompactd0
14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 watchdogd
18 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kworker/0:1
19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 rpciod
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xprtiod
21 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 nfsiod
32 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
Interesting... Your list shows "php5-fpm"... My system shows "php-fpm7.0"
How nice... /usr/bin/php routes to /etc/alternatives/php which routes back to /usr/bin/php7.0
This is on an R-Pi 3B running Pi-Star 4.1.0-rc7 / 20191220
Code: Select all
top - 12:51:27 up 24 days, 1:00, 1 user, load average: 0.08, 0.14, 0.05
Tasks: 121 total, 1 running, 120 sleeping, 0 stopped, 0 zombie
%Cpu(s): 4.0 us, 2.3 sy, 0.0 ni, 93.4 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
MiB Mem : 975.1 total, 522.3 free, 130.7 used, 322.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 750.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18480 root 15 -5 99252 19368 5508 S 2.3 1.9 8:54.81 ircddbgatewayd
19881 www-data 20 0 127480 13980 11092 S 2.3 1.4 0:06.46 php-fpm7.0
19958 www-data 20 0 127416 13476 10836 S 2.0 1.3 0:06.38 php-fpm7.0
4667 root 20 0 0 0 0 I 1.7 0.0 0:01.76 kworker/u8:1-brcmf_wq/mmc+
18643 root 10 -10 20256 9696 2736 S 1.7 1.0 8:09.49 MMDVMHost
68 root -51 0 0 0 0 S 1.0 0.0 120:53.15 irq/86-mmc1
5128 pi-star 20 0 10452 3020 2540 R 1.0 0.3 0:01.40 top
19106 www-data 20 0 127480 13984 11092 S 1.0 1.4 0:07.71 php-fpm7.0
1 root 20 0 33852 8268 6436 S 0.0 0.8 100:07.46 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:03.32 kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00 rcu_gp
Code: Select all
pi-star@pi-star-3b(ro):~$ ls -l /usr/bin/php*
lrwxrwxrwx 1 root root 21 Aug 2 2016 /usr/bin/php -> /etc/alternatives/php
-rwxr-xr-x 1 root root 3115388 Mar 8 2019 /usr/bin/php7.0
pi-star@pi-star-3b(ro):~$ ls -l /etc/alternatives/php*
lrwxrwxrwx 1 root root 15 Dec 21 2018 /etc/alternatives/php -> /usr/bin/php7.0
lrwxrwxrwx 1 root root 31 Dec 21 2018 /etc/alternatives/php.1.gz -> /usr/share/man/man1/php7.0.1.gz
This is on an R-Pi 3B running Pi-Star 4.1.0-rc7 / 20191220
--
AF6VN
Dennis L Bieber
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
Note that I'm still on v3.14, not really interested in 4.x until it is out of beta.
I gather that what I'm experiencing is unusual. Would running the modem hat on a more powerful processor (i.e. Pi3 vs Pi Zero) result in more reliable operation under load? From what I can tell by looking at stats the Zero seems to be quite adequate for a single simplex user though.
I gather that what I'm experiencing is unusual. Would running the modem hat on a more powerful processor (i.e. Pi3 vs Pi Zero) result in more reliable operation under load? From what I can tell by looking at stats the Zero seems to be quite adequate for a single simplex user though.
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
The pizero is capable of running pistar under normal situations. Its memory is the primary limitation I have run into over the years. For example if I try to do "apt-get upgrade" while the nginx web server is running the pi-zero usually runs out of memory and kernel starts to force prune processes from memory. Randomly killing everything from SSH daemon to syslog server.
So I make sure to shut everything unnecessary down or put disk in RW and create a temporary file based swap area to give the pi-zero some extra breathing room. Right now I tend to run a custom image with a swap partiition on sdcard and have modified. Things so swap is turne don every time I do rpi-rw and off when I do the rpi-ro.
My guess its some kind of resource exhaustion going on. Of which candidates are CPU, Memory and tmp storage. Temp storage is a ram disk in memory and odd things happens when some things cause logging to go wild and temp files up.
Top gives you an idea of memory usage and cpu. If it happens again also might want to take at disk usage. Run 'df" and look at lines that have a FileSystem type of tmpfs and see if any are at 95% or higher on the Use%.
But guess will have to wait to see if it happens again and if you can get some info to try to narrow down to a root cause.
So I make sure to shut everything unnecessary down or put disk in RW and create a temporary file based swap area to give the pi-zero some extra breathing room. Right now I tend to run a custom image with a swap partiition on sdcard and have modified. Things so swap is turne don every time I do rpi-rw and off when I do the rpi-ro.
My guess its some kind of resource exhaustion going on. Of which candidates are CPU, Memory and tmp storage. Temp storage is a ram disk in memory and odd things happens when some things cause logging to go wild and temp files up.
Top gives you an idea of memory usage and cpu. If it happens again also might want to take at disk usage. Run 'df" and look at lines that have a FileSystem type of tmpfs and see if any are at 95% or higher on the Use%.
But guess will have to wait to see if it happens again and if you can get some info to try to narrow down to a root cause.
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
3.4 releases use php5.
Php7 was introduced in the 4.0 release (now expired) and is current with 4.1 releases.
Pi zero is still capable of handling pistar.
The higher spec Pi's are quicker with boot up, serving webpages, processing config changes etc. but don't give anything more to the general basic operation of a pistar install.
3.4. releases are still supported but production time, resources, new features, system advances etc. have moved to 4.1
Sent from my SM-G935F using Tapatalk
Php7 was introduced in the 4.0 release (now expired) and is current with 4.1 releases.
Pi zero is still capable of handling pistar.
The higher spec Pi's are quicker with boot up, serving webpages, processing config changes etc. but don't give anything more to the general basic operation of a pistar install.
3.4. releases are still supported but production time, resources, new features, system advances etc. have moved to 4.1
Sent from my SM-G935F using Tapatalk
Andrew M1DNS.
Pi-star Admin Team.
Pi-star Admin Team.
Re: Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'
> Pi zero is still capable of handling pistar.
Is the Pi Zero capable (in terms of memory and processing power) of supporting 4.1x or would it be advisable to stay with 3.14?
Is the Pi Zero capable (in terms of memory and processing power) of supporting 4.1x or would it be advisable to stay with 3.14?