DMRIds.dat updates not consistent across multiple hotspots

[KN2TOD]

Across six hotspots, the DMRIds.dat file is not being updated consistently as expected.

Prowled through the logs to see if something amiss shows up but not sure I'm looking at the right logs or what specifically I should be searching for in those logs.

All systems running 4.1.2-20200706. Systems failing are Pi-Zero's; manual updates required to correct.

=============================================================================
Linux pi-star-1 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l
-rw-r--r-- 1 root bin 3549545 Jul 22 03:27 /usr/local/etc/DMRIds.dat
-rw-r--r-- 1 root bin 3547101 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================
Linux pi-star-2 5.4.51+ #1326 Fri Jul 17 10:39:45 BST 2020 armv6l
-rw-r--r-- 1 root bin 3545310 Jul 20 13:23 /usr/local/etc/DMRIds.dat *
-rw-r--r-- 1 root bin 3545310 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================
Linux pi-star-3 4.19.97+ #1294 Thu Jan 30 13:10:54 GMT 2020 armv6l
-rw-r--r-- 1 root bin 3545310 Jul 20 12:29 /usr/local/etc/DMRIds.dat *
-rw-r--r-- 1 root bin 3545310 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================
Linux pi-star-4 4.19.97+ #1294 Thu Jan 30 13:10:54 GMT 2020 armv6l
-rw-r--r-- 1 root bin 3547101 Jul 21 09:00 /usr/local/etc/DMRIds.dat *
-rw-r--r-- 1 root bin 3547101 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================
Linux pi-star-5 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l
-rw-r--r-- 1 root bin 3549545 Jul 22 03:27 /usr/local/etc/DMRIds.dat
-rw-r--r-- 1 root bin 3547101 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================
Linux pi-star-6 4.19.66+ #1253 Thu Aug 15 11:37:30 BST 2019 armv6l
-rw-r--r-- 1 root bin 3545310 Jul 20 13:06 /usr/local/etc/DMRIds.dat *
-rw-r--r-- 1 root bin 3545310 Jul 22 03:25 /usr/local/etc/DMRIds.dat.20200722
=============================================================================

[G8SEZ]

Interesting, same with my 3 Pi Zero Ws, one on a July 7th update, one on July 19th and one on July 20th.

Can't see why this should be down to anything Pi Zero specific, it's not like the low CPU grunt can affect the IDs it downloads.

[KN2TOD]

Thanks for confirming! Was thinking it might be just me and the particular configuration I have set up here, so it's good to know others are seeing the same thing(s). The more eyes we have on the problem...

Agree the Pi-Zero angle is suspicious but doesn't make much sense. If they were a factor, you'd think they would be hit-or-miss phenom and not consistently failing. Also, I suspect this has been going on for a while, but I just hadn't noticed until a couple of days ago. Now I plan on being more consistent in studying it but need some hints on what to look for behind the scenes to get at the root cause.

[G8SEZ]

I ran sudo pistar-update from a terminal so the apt-get update and apt-get upgrade process ran before the services were shut down and the poster git pull and host file updates ran.

I will have a look at the pistar-update script and see if I can see whether it could be failing randomly or something like that.

[G8SEZ]

The retrieval is done by a simple curl --fail from a file on the pistar.uk site.

I suppose that when a lot of these things simultaneously request the same file, the server might struggle a bit in some cases.

Not sure how to cure this without improved infrastructure or possibly some randomisation of the time that PiStars update themselves.

[KN2TOD]

So, modified the pistar-update file on one of my failing systems to see if I can trap some messages overnight:

Code: Select all

	echo "Updating Hostfiles..."
	/usr/local/sbin/HostFilesUpdate.sh > /tmp/pistar-hostfile-update
#	/usr/local/sbin/HostFilesUpdate.sh > /dev/null 2>&1
	echo "Done"

It's only the DMRIds.dat file that's failing, all the others seem to be updating just fine.

Still not clear where it's logging the processing of the overall update procedure.

[KN2TOD]

This problem is occurring on the Pi 3's as well as the Pi-zeros, just not as often. Seems like the Pi 3's may fail one out of 10 days, while (conversely), the Pi-zeros may succeed one out of 10 days, roughly. This explains why I've seen callsigns show up on two different hotspots simultaneously but one as numerics and other as alphabetic, as well as the fact that the phenom seems to come and go across the lot of them.

The Pi 3's are hardwired (ethernet) but the Pi-zero's are all wireless. But processor and connectivity speeds don't appear to correlate cleanly with the problem. What does correlate is the timing: they all seem to fail during the nightly processes and not during any manually induced updates at other times. It may be the shear volume of data being downloaded at that time: six hotspots simultaneously updating ALL the various host files involved. (I'm not seeing any lags in the other host files, but they are all way smaller - the DMRIds.dat file is by far the largest.)

Coincidentally, and apropos to this, I occasionally have errors accessing particular websites such as <forum.pistar.uk>:

-----

I have been able to get the nightly process to cough up some information, specifically this error:

Code: Select all

<html>
<head><title>500 Internal Server Error</title></head>
<body bgcolor="white">
<center><h1>500 Internal Server Error</h1></center>
<hr><center>nginx</center>
</body>
</html>

(This corresponds to the errors I see in a browser from time to time)

If the process runs normally, it logs this instead:

Code: Select all

HTTP/1.1 200 OK
Date: Fri, 24 Jul 2020 10:11:53 GMT
Content-Type: application/octet-stream
Content-Length: 3553885
Connection: keep-alive
Set-Cookie: __cfduid=d51d4a6d45438125f0e66b6376bac3f9e1595585513; expires=Sun, 23-Aug-20 10:11:53 GMT; path=/; domain=.pistar.uk; HttpOnly; SameSite=Lax
Last-Modified: Fri, 24 Jul 2020 00:17:42 GMT
ETag: "5f1a28a6-363a5d"
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
CF-Cache-Status: DYNAMIC
cf-request-id: 0421e7df340000e3ca76a1b200000001
Server: cloudflare
CF-RAY: 5b7cdc11ef1ae3ca-ATL

The key here is that I'm not accessing <pistar.uk> directly, but rather being routed through a local (regional/national) gateway: Cloudflare. This gateway seemingly hiccups from time to time, for reasons unknown, although I think I can safely say its load related.

(One can verify if their own queries are also being diverted: issue a "ping www.pistar.uk" and copy down the ip address, enter that ip address in a browser window and see what you get.)

So...

While there are ways to get around the redirection of Cloudflare, it seems to just add more complexity to the situation.

Instead, I'm investigating adding some options to the curl command for the file in question: --retry and --retry-delay.

This change has worked on one failing hotspot (I've left the others as is as a baseline) for one cycle but it will take a few more days to see if this holds. The number of retries and/or the delay between the retries may need to be adjusted.

...to be continued.

[KN2TOD]

Not sure I want to tinker with the scheduling, as I'm not readily seeing any directions on how this might be accomplished or whether it's advisable. If, as you note, Cloudflare is proving problematic, shifting a batch process to another time frame may not be any more effective than my moving my browsing habits to a different time of day. Perhaps trying an automated way of pounding on the site until it responds would function just as well as my continually hitting the refresh key in my browser when such situations arise. After all, this is what Cloudflare advises when it chokes on a query.

-----
If past contributions are any indications of future contributions, I don't see us making any headway in these matters.