ZumSpot Rebooting Occasionally on its own

[KC6N]

Let me try a couple things first before doing a remote in thing. So the one without the OLED (#1) isn't doing it as far as I can tell. It shows one case at about 2:26AM (My time) so that is the normal early AM reset -- then everything else is normal.

I burned a couple uSD cards with 3.4.17 to see how that does. I can also try turning off the OLED's. Whatever this thing is, it has droped me in the middle of a QSO a couple times but then the HotSpot comes back up.

[KC6N]

Time for an update – I have been running test, truing to gather data on this issue for about a week and a half. Here is a summary of everything I know at this point.

Description of issue: It appears that after a certain amount of time (currently undetermined) the MMDVM application enters a mode where it periodically re-starts as signified by this bit of log data here:

Code: Select all

M: 2020-05-28 01:31:17.631 Closing the MMDVM
I: 2020-05-28 01:31:18.024 Stopped the DMR Id lookup reload thread
M: 2020-05-28 01:31:18.025 Closing D-Star network connection
M: 2020-05-28 01:31:18.025 DMR, Closing DMR Network
I: 2020-05-28 01:31:18.026 MMDVMHost-20200503_Pi-Star_v4 exited on receipt of SIGTERM
I: 2020-05-28 01:31:18.463 This software is for use on amateur radio networks only,
I: 2020-05-28 01:31:18.463 it is to be used for educational purposes only. Its use on
I: 2020-05-28 01:31:18.463 commercial networks is strictly prohibited.
I: 2020-05-28 01:31:18.463 Copyright(C) 2015-2020 by Jonathan Naylor, G4KLX and others
M: 2020-05-28 01:31:18.464 MMDVMHost-20200503_Pi-Star_v4 is starting
M: 2020-05-28 01:31:18.464 Built 22:38:53 May 5 2020 (GitID #7718676)
I: 2020-05-28 01:31:18.464 General Parameters
I: 2020-05-28 01:31:18.464 Callsign: KC6N
….
I: 2020-05-28 01:31:29.166 OVCM: off
I: 2020-05-28 01:31:29.166 DMR Roaming Beacons Type: off
I: 2020-05-28 01:31:29.166 Started the DMR Id lookup reload thread
I: 2020-05-28 01:31:29.311 Interfaces Info
I: 2020-05-28 01:31:29.312 IPv4: lo:127.0.0.1
I: 2020-05-28 01:31:29.312 IPv4: wlan0:192.168.1.136
I: 2020-05-28 01:31:29.312 Default interface is : wlan0
I: 2020-05-28 01:31:29.312 IP to show: wlan0:192.168.1.136
M: 2020-05-28 01:31:29.312 MMDVMHost-20200503_Pi-Star_v4 is running
M: 2020-05-28 01:31:39.290 DMR, Logged into the master successfully
This repeats approximately every 8 minutes – It takes a while to start but once it starts it continues forever.
I have three separate HotSpots ZS#1, ZS#2 and ZS#3 as below:
ZS#1 A Zum Radio v 0.4 hat (the older one without the display) on a Pi ZeroW board running Pi-Star 4.1.2
ZS#2 A Zum Radio v 0.6 hat (with the display) on a Pi ZeroW board running Pi-Star 4.1.2
ZS#3 A Zum Radio v 0.6 hat (with the display) on a Pi ZeroW board running Pi-Star 4.1.2
ZumSpots 2 and 3 are identical and exhibit the issue, ZumSpot 1 does not.
ZS1 is generally configured for DSTAR only and sits on XRF012A
ZS2 is generally configured for DSTAR (REF012A) and Fusion (Alabama Link)
ZS3 is generally configured for DSTAR (REF012A) And DMR (Brandmeister)

So Far:

• 1. I have rebuilt all three of them from the ground up including manual entry of the configuration parameters (so as not to inherit anything bad from a possibly corrupt config backup) – Still resetting
  2. I replaced the common power supply with another on – Still resetting
  3. I replaced the PiZeroW with a Pi 3b on #3 and still had the issue
  4. I tried running only #2 – Still saw the problem
  5. I tried running only #3 – Still saw the problem
  6. Tried #2 on Fusion only – Same issue
  7. Tried #2 on DSTAR only – Same Issue
  8. Tried #3 on DMR only – Same issue
  9. Tried #3 on DSTAR only – Same Issue
  10. Checked Throttling on all of them: sudo vcgencmd get_throttled (returned 0x0)
  11. Checked dmesg | grep -I volt (returned nothing)
  12. Built a new pair of µSD cards with Pi Star 3.4.17. I ran this FW for 2 days – Did NOT see the issue in 48 hours of use
  13. I removed the µSD cards e/w 3.4.17 and put the cards e/w 4.1.2 back in to verify that the problem was reproducible and, sure enough, in the AM after what looked like a successful overnight run on both the repetitive 8 minute resets returned on #2 and #3.
  14. I tried an update on each (#2 and #3) and while the unit did close and re-start there was nothing on the update display (i.e. the data that usually scrolls by didn’t scroll by), so I do not know whether it updated or not. (NOTE: I tried this while both devices were in the failed state). So in the failed state – update does not run properly.
  15. An interesting observation is that while I have DSTAR (REF012A) set up on both #2 and #3 they do not seem to be working on DSTAR. At least when I key up on #2, I do not see myself back on #3 and visa versa. This is not the case on 3.4.17 or on 4.1.2 (when it is working). DMR does get through on #3 (I can see myself on BM last heard). So it looks like DMR keeps working.
  16. Turning off OLED the on #3 and #2. This does NOT solve the problem. #2 & #3 resetting on an 8-minute cycle. I thought this might be a possibility since ZS#1 (no display) runs fine on 4.2.1 and has run fine All week.
  17. Ran an update on 4.1.2 after a re-boot on both #2 and #3. This time the update ran and didn’t just hang which it appears to do in “broken mode”, however, unfortunately the issue (8 minute periodic MMDVM resets) reappeared on #2 and #3 at sometime during the night – it takes a while for this to happen – and persisted until I reset them in the AM.
  18. At this point, I re-flashed both ZUM radio hats. Both hats were re-flashed Zumspot FW v1.4.17 (this is what they have had all along – but worth a try anyway). This made no difference. In the morning, the MMDVM application on both #2 and #3 was resetting every 8 minutes continuously. FWIW: this seems to be the result of something happening after 10:00pm local time i.e. in the night. If I reset them in the morning (say 10:00AM) they will run flawlessly all day but the next morning, they will have gone in to the problematic mode.
  19. I will try running ZS#2 from wired internet (I only have one e-net to ISB adapter). After an overnight run ZS#2 was found to be resetting on an 8-minute interval when I checked in the AM. This issue seems to occur at night for some reason. It might have something to do with the nightly update. One strange thing, though, is that ZS#3 ran fine and was still running fine in the AM.

At this point I do not know what to do. I wrote this all down in the hope that it will give someone some ideas as to what might be going on or possibly some more things that can be tried.

[KC6N]

You are running two hotspots on your network connected to the same reflector?

I would suggest only running one hotspot and completely shutting the other two down and seeing what that does.

Yes ZS2 and ZS3 are both carrying REF012A. I have tried running them separately and still saw the issue. I'll repeat that to verify.

What model router are you using? Are you using DHCP, if so how long is the DHCP lease? Are you using any hardware address assignment for DHCP?

The router is a Linksys model 6350 which is used as a router only, it's WiFi is off. Wifi is supported by a pair of Ubiquiti Pucks (one upstairs and the other downstairs). All three ZumSpots are connected to the upstairs one which is the closest. DHCP lease time is 1440 minutes (24 hours). I don't know about any HW address management -- it is straight DHCP but i can reserve the IP addresses or change the lease time if need be.

Do you have different host names assigned for each hotspot?

Yes. Pi-Star1, Pi-Star2 and Pi-star3

[KC6N]

Addresses are not changing. I disconnected all but ZS#2 (one of the ones with the display that was running Fusion and DSTAR). So, ZS#2 has been running for a couple days without issue. I am going to let it run for a while and see if the problem is gone, then I'll fire up one of the others and see what brings it back -- maybe there'll be a clue there.

I am a bit confused because i am pretty sure i have run this one by itself in this configuration before but ZS#1 (w/o display DSTAR only connected to XRF012A).

So ill let ZS#2 run by itself for a few days and see what i see. So far, so good. The only reset I see is the one at 10:26 GMT which is the early morning re-set. What gets updated in that process?

[KC6N]

Hmmm....., this is interesting. No, I had not seen these. I skimmed through these but will need to take a bit of time to read them carefully. It looks like running DSTAR on two HS on the same reflector is not a good idea. What's comforting is that apparently, I am not the first one to make the mistake. Thank you for dredging these up, apparently I didn't know what to search for. I tried to cover a lot of these issues when i set them up but it looks like running DSTAR simultaneously on the same reflector from two hotspots is a problem.

FWIW: all three spots have their own frequency, on DSTAR each has it's own module ID:

pi-star1 (438.050 MHz): DMR ID=310656401, DSTAR Callsigns are KC6N^^^A, KC6N^^^G Running DSTAR (only) XRF012A
pi-star2 (439.025 MHz): DMR ID=310656402, DSTAR Callsigns are KC6N^^^B, KC6N^^^G Running DSTAR on REF012A and Fusion on Alabama Link
pi-star3 (439.075 MHz): DMR ID=310656403, DSTAR Callsigns are KC6N^^^C, KC6N^^^G Running DSTAR on REF012A and DMR (Brandmeister 3103)

Right now only pi-star2 is running and it has been running well for a 1.5 days. I am going to run it for another couple days and see how it does. If it is solid, then i can try to make it break again by lighting up the other DSTAR.

Thanks for the help.

[KC6N]

Further testing: I spent a lot of time on the Pi-Star Forums and got some suggestions from KE7FNS (Jason, I believe). His thinking was that perhaps this was a consequence of having both Pi-Star2 and Pi-Star3 tied to REF012A. I decided to do a bit more testing to see if I could nail that down.

Once again, my standard setup has been:
pi-star1 (438.050 MHz): DMR ID=310656401, DSTAR Callsigns are KC6N^^^A, KC6N^^^G Running DSTAR (only) XRF012A
pi-star2 (439.025 MHz): DMR ID=310656402, DSTAR Callsigns are KC6N^^^B, KC6N^^^G Running DSTAR on REF012A and Fusion on Alabama Link
pi-star3 (439.075 MHz): DMR ID=310656403, DSTAR Callsigns are KC6N^^^C, KC6N^^^G Running DSTAR on REF012A and DMR (Brandmeister 3103

To test this out:

1. I first fired up just Pi-Star2 only and ran that for about 5 days w/o issue. Only Pi-Star2 connected to REF012A.

2. I next added Pi-Star1. So now, Pi-Star1 and Pi-Star2 are both on configured as described above. This setup ran for three days without any issue on Pi-Star1 or Pi-Star2. Only Pi-Star2 connected to REF012A. Pi-Star1 is always on XRF012A

3. Next, I activated Pi-Star3 in DMR mode only (w/o the DSTAR REF012A connection activated). This fine for at least 48 hours. In this setup Pi-Star3 had DMR only and Pi-Star2 had DSTAR and Fusion. Only Pi-Star2 connected to REF012A.

4. Next, I activated Pi-Star3 in DMR w/ the DSTAR REF012A connection activated. So, it has DSTAR and DMR (as it used to have). Pi-Star2 has Fusion only w/ DSTAR turned off. Pi-Star1 is running DSTAR on XRF012A as always. No resets on any of the three units after two days (48 hours). Only Pi-Star2 connected to REF012A.

5. Next, I have set Pi-Star2 for DSTAR (REF012A) and Fusion and Pi-Star3 for DSTAR (REF012A) and DMR (Brandmeister) with Pi-Star1 connected to DSTAR XRF012A only. Thus, the three units are as they originally were when I started noticing the 8-minute reset problem for the MMDVM application on Pi-Star2 and Pi-Strar3. In the AM I found that Pi-Star2 was still working on DSTAR (REF012A) and Fusion but Pi-Star3 was in the 8-minute MMDVM re-start mode. Both Pi-Star2 and Pi-Star3 connected to REF012A.

6. Next, I have configured Pi-Star2 and Pi-Star3 identically, single mode, DSTAR only and both on REF012A. Shure enough, in the morning both of the hotspots were doing the 8 minute MMDVM reset dance. (Pi-Star1 was still pointed at XRF012A) Both Pi-Star2 and Pi-Star3 connected to REF012A.

Based on this testing I am going going to agree with KE7FNS and conclude that this has something to do with DSTAR and having two HotSpots both tied to REF012A (or probably any REFxxxX).

I noticed that I do not have KC6N^^^A, KC6N^^^B or KC6N^^^C registered on the gateway which might have something to do with it. I will give that a try as soon as i can gain access to the gateway to update the registration.

I may put up another thread in the DSTAR section to see if I can sort exactly what is happening. It may turn out that the Reflector sees two instances of KC6N and boots one off? But at this point it appears that the fix is to not run two simultaneous connections to the same REFxxx reflector. There is no reason that I need to do that anyway. Thanks to KE7FNS for all your help. I should have done a more thorough search at the outset but I didn't really know what to search for.