Pi-Star HTTP interface becomes unreachable with '504 Gateway Time-out'

[N6PL]

I have a simplex ZUMspot on a Pi Zero W board (Current rev is Pi-Star:3.4.17/Dashboard:20191220/ZUMspot v1.4.16 and updates are current) and the unit appears to work normally except for one problem, and that is after some period of time (30 min to several hours, appears to be random) the HTTP interface becomes unreachable, sometimes with a '504 Gateway Time-out' error and sometimes no response at all. HTTP operation is completely normal immediately after initial boot or a reboot, but the problem will reoccur predictably some period of time thereafter. This does not seem to be a configuration error since the unit always works normally (for a while) after boot. There does not seem to be any IP connectivity issues since when the problem occurs connectivity to Brandmeister is still up, calls through the hotspot work normally, and I can access the unit locally via ssh. I have even tried restarting nginx when the problem occurs but that action has no effect. WiFi power saving has been turned but off with no improvement. Also I have checked available RAM and disk space when the problem occurs but everything looks normal.

When the problem occurs the unit appears to be working normally in all ways except for the lack of connectivity on the HTTP connection. This might suggest an external problem but I have tried multiple PC and browsers, even a different router, but the problem remains unaffected so I can only assume that the fault is in fact with the hotspot.

Been pulling my hair out trying to troubleshoot this, not sure where else to look?

Thanks

[AF6VN]

Unclear:

Are you accessing the web page from /outside/ the LAN? Since SSH from /inside/ is working, I'd tend to think HTTP from /inside/ might also work.

The answer will isolate where to look... If HTTP /inside/ the LAN is failing, then I'd be investigating the server and any common components (router). If only from outside the LAN, you may need to work with your ISP or others

https://developer.mozilla.org/en-US/doc ... Status/504

... implicates proxy configurations, so if some external server handles connection to your node, the problem may lie between it and your node.

[N6PL]

I am attempting to access from inside the LAN. And that's what's so confusing since there really are no network elements between me and the hotspot other than the router, but I have tried a second router with the same results. Add to that different PCs and browsers with no change and it would seem to pretty positively isolate the problem to the hotspot itself, and since I still have IP connectivity (evidenced by ability to ping, access via putty, etc.) when HTTP access is down it would appear that some Pi-Star process critical to HTTP access must be failing. As above I've tried restarting nginx when the problem occurs but that doesn't bring the interface back up, nothing seems to short of a complete reboot.

[kc7ngc]

504 Gateway timeout is an HTTP error code. Coming from NGINX web server on Pi-Star. The "gateway" its talking about is that the webserver is trying to talk to the PHP data engine that renders the page. For some reason the php-fpm data engine is timing out.

If you ssh into the ZUMspot does it appear to be under high cpu load for some reason? Only time I saw this happen is when we had a pistar setup at an event and too many people were connecting to the dashboard to monitor who was talking and it overloaded the nginx webserver. I doubt that is your problem specifically but got to be some kind of resource exhaustion going on.

When its having the problem If you ssh into it and run 'top' at command line what do the top numbers look like. Should be something like this.

Code: Select all

top - 20:40:45 up 8 min,  1 user,  load average: 1.02, 0.91, 0.56
Tasks:  89 total,   1 running,  88 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.7 us,  5.4 sy,  0.0 ni, 89.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    479.8 total,    279.0 free,    129.0 used,     71.8 buff/cache
MiB Swap:    512.0 total,    491.5 free,     20.5 used.    290.4 avail Mem

Also what are the two top processes in the list and their %CPU and %MEM.

[N6PL]

Before the problem arises I see a few (from one to four at any given time) php scripts running, and then after the problem arises I see none so yes, the scripts (or whatever process generates them) crashing is apparently the problem. I didn't notice any unusually high CPU utilization associated with the failure but it's also possible there was some event that I missed.

I have had a few static TGs up in the past and as an experiment I terminated all of them and the problem seemed to go away, or at least I still had HTTP access the following morning which is longer than it's ever gone. I never saw any unusually CPU utilization while the static links were up so I didn't associate it with the web access issue but it appears they may be related... or were related because as an experiment I reestablished the static TGs to see if I could replicate the failure and thus far have been unable. I will let things run a little longer but so far no more failures, even with the static TGs up (again, not sure if there ever was any relationship to this and the php issue.) I will continue to monitor and see if the problem will resurface. Odd because it was 100% reproducible in the past.

Code: Select all

top - 00:24:53 up 3 min,  1 user,  load average: 1.01, 0.69, 0.30
Tasks: 106 total,   1 running, 104 sleeping,   0 stopped,   1 zombie
%Cpu(s): 20.5 us, 32.9 sy,  0.0 ni, 45.0 id,  0.0 wa,  0.0 hi,  1.6 si,  0.0 st
KiB Mem:    493252 total,   147172 used,   346080 free,     7620 buffers
KiB Swap:        0 total,        0 used,        0 free.    89140 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 2156 www-data  20   0  124164   9512   7016 S  6.1  1.9   0:00.19 php5-fpm
 1256 www-data  20   0  124616  11672   9068 S  5.1  2.4   0:02.09 php5-fpm
 1110 www-data  20   0  124880  12564   9776 S  3.8  2.5   0:03.55 php5-fpm
  997 root      10 -10   21036  10940   2600 S  3.5  2.2   0:07.14 MMDVMHost
 1131 www-data  20   0  124616  11872   9180 S  1.6  2.4   0:02.25 php5-fpm
 1089 pi-star   20   0    5012   2384   2004 R  1.3  0.5   0:01.50 top
  217 root      20   0       0      0      0 S  1.0  0.0   0:00.62 kworker/u2:2
    3 root      20   0       0      0      0 S  0.6  0.0   0:00.69 ksoftirqd/0
   65 root     -51   0       0      0      0 S  0.3  0.0   0:00.22 irq/86-mmc1
  945 www-data  20   0   15668   4120   3072 S  0.3  0.8   0:00.06 nginx
 1067 root      20   0    8816   4996   4368 S  0.3  1.0   0:00.84 sshd
    1 root      20   0    5964   4124   2712 S  0.0  0.8   0:05.94 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kthreadd
    4 root      20   0       0      0      0 S  0.0  0.0   0:00.13 kworker/0:0
    5 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0       0      0      0 S  0.0  0.0   0:00.25 kworker/u2:0
    7 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-drain
    8 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kdevtmpfs
    9 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
   10 root      20   0       0      0      0 S  0.0  0.0   0:00.00 khungtaskd
   11 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
   12 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
   13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
   14 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
   15 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
   16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
   17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 watchdogd
   18 root      20   0       0      0      0 S  0.0  0.0   0:00.09 kworker/0:1
   19 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 rpciod
   20 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xprtiod
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kswapd0
   22 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 nfsiod
   32 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
   33 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
   34 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset

[AF6VN]

Interesting... Your list shows "php5-fpm"... My system shows "php-fpm7.0"

Code: Select all

top - 12:51:27 up 24 days,  1:00,  1 user,  load average: 0.08, 0.14, 0.05
Tasks: 121 total,   1 running, 120 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.0 us,  2.3 sy,  0.0 ni, 93.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
MiB Mem :    975.1 total,    522.3 free,    130.7 used,    322.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.    750.9 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
18480 root      15  -5   99252  19368   5508 S   2.3   1.9   8:54.81 ircddbgatewayd
19881 www-data  20   0  127480  13980  11092 S   2.3   1.4   0:06.46 php-fpm7.0
19958 www-data  20   0  127416  13476  10836 S   2.0   1.3   0:06.38 php-fpm7.0
 4667 root      20   0       0      0      0 I   1.7   0.0   0:01.76 kworker/u8:1-brcmf_wq/mmc+
18643 root      10 -10   20256   9696   2736 S   1.7   1.0   8:09.49 MMDVMHost
   68 root     -51   0       0      0      0 S   1.0   0.0 120:53.15 irq/86-mmc1
 5128 pi-star   20   0   10452   3020   2540 R   1.0   0.3   0:01.40 top
19106 www-data  20   0  127480  13984  11092 S   1.0   1.4   0:07.71 php-fpm7.0
    1 root      20   0   33852   8268   6436 S   0.0   0.8 100:07.46 systemd
    2 root      20   0       0      0      0 S   0.0   0.0   0:03.32 kthreadd
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp

Code: Select all

pi-star@pi-star-3b(ro):~$ ls -l /usr/bin/php*
lrwxrwxrwx 1 root root      21 Aug  2  2016 /usr/bin/php -> /etc/alternatives/php
-rwxr-xr-x 1 root root 3115388 Mar  8  2019 /usr/bin/php7.0
pi-star@pi-star-3b(ro):~$ ls -l /etc/alternatives/php*
lrwxrwxrwx 1 root root 15 Dec 21  2018 /etc/alternatives/php -> /usr/bin/php7.0
lrwxrwxrwx 1 root root 31 Dec 21  2018 /etc/alternatives/php.1.gz -> /usr/share/man/man1/php7.0.1.gz

How nice... /usr/bin/php routes to /etc/alternatives/php which routes back to /usr/bin/php7.0

This is on an R-Pi 3B running Pi-Star 4.1.0-rc7 / 20191220

[N6PL]

Note that I'm still on v3.14, not really interested in 4.x until it is out of beta.

I gather that what I'm experiencing is unusual. Would running the modem hat on a more powerful processor (i.e. Pi3 vs Pi Zero) result in more reliable operation under load? From what I can tell by looking at stats the Zero seems to be quite adequate for a single simplex user though.

[kc7ngc]

The pizero is capable of running pistar under normal situations. Its memory is the primary limitation I have run into over the years. For example if I try to do "apt-get upgrade" while the nginx web server is running the pi-zero usually runs out of memory and kernel starts to force prune processes from memory. Randomly killing everything from SSH daemon to syslog server.

So I make sure to shut everything unnecessary down or put disk in RW and create a temporary file based swap area to give the pi-zero some extra breathing room. Right now I tend to run a custom image with a swap partiition on sdcard and have modified. Things so swap is turne don every time I do rpi-rw and off when I do the rpi-ro.

My guess its some kind of resource exhaustion going on. Of which candidates are CPU, Memory and tmp storage. Temp storage is a ram disk in memory and odd things happens when some things cause logging to go wild and temp files up.

Top gives you an idea of memory usage and cpu. If it happens again also might want to take at disk usage. Run 'df" and look at lines that have a FileSystem type of tmpfs and see if any are at 95% or higher on the Use%.

But guess will have to wait to see if it happens again and if you can get some info to try to narrow down to a root cause.

[M1DNS]

3.4 releases use php5.

Php7 was introduced in the 4.0 release (now expired) and is current with 4.1 releases.

Pi zero is still capable of handling pistar.
The higher spec Pi's are quicker with boot up, serving webpages, processing config changes etc. but don't give anything more to the general basic operation of a pistar install.

3.4. releases are still supported but production time, resources, new features, system advances etc. have moved to 4.1

Sent from my SM-G935F using Tapatalk

[N6PL]

> Pi zero is still capable of handling pistar.

Is the Pi Zero capable (in terms of memory and processing power) of supporting 4.1x or would it be advisable to stay with 3.14?