pi-star freezing/locking up

[KN2TOD]

(I didn't know about Linux much either but that's part of fun of this: it's a learning experience!)

Start with some simple steps first, to see if we're on the same page: log into SSH (directly or via config>expert>ssh access) and key in the following:

Code: Select all

pi-star@pi-star-1(ro):~$ df -kTh

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4       15G  1.8G   12G  13% /
devtmpfs       devtmpfs  454M     0  454M   0% /dev
tmpfs          tmpfs     487M     0  487M   0% /dev/shm
tmpfs          tmpfs      32M  7.6M   25M  24% /run
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs      32M     0   32M   0% /sys/fs/cgroup
tmpfs          tmpfs      64M   12K   64M   1% /tmp
tmpfs          tmpfs      16K     0   16K   0% /var/lib/sudo
tmpfs          tmpfs      64M   45M   20M  70% /var/log
tmpfs          tmpfs     1.0M     0  1.0M   0% /var/lib/nginx/body
tmpfs          tmpfs     4.0M  2.2M  1.9M  54% /var/lib/samba/private
tmpfs          tmpfs     4.0M   16K  4.0M   1% /var/lib/vnstat
tmpfs          tmpfs      64K     0   64K   0% /var/lib/php/sessions
tmpfs          tmpfs     1.0M   72K  952K   8% /var/cache/samba
tmpfs          tmpfs      32K   16K   16K  50% /var/lib/dhcpcd5
tmpfs          tmpfs      16K  4.0K   12K  25% /var/lib/logrotate
/dev/mmcblk0p1 vfat       63M   50M   14M  80% /boot
tmpfs          tmpfs      98M     0   98M   0% /run/user/1000
tmpfs          tmpfs      16K     0   16K   0% /var/lib/misc

and check the usage for the /var/log directory.

If that directory looks like mine (high), issue the following:

Code: Select all

pi-star@pi-star-1(ro):~$ cd /var/log
pi-star@pi-star-1(ro):log$ ls -la

total 45016
drwxr-xr-x  6 root root       540 Feb 14 00:00 .
drwxr-xr-x 12 root root      4096 Mar 26  2020 ..
-rw-r-----  1 root adm     152368 Feb 14 12:50 auth.log
-rw-r-----  1 root adm     202880 Feb 14 00:00 auth.log.1
-rw-------  1 root root     11275 Feb 13 10:22 boot.log
-rw-rw----  1 root utmp         0 Feb 14 00:00 btmp
-rw-rw----  1 root utmp        20 Feb 13 10:22 btmp.1.gz
-rw-r-----  1 root adm    6333534 Feb 14 12:50 daemon.log
-rw-r-----  1 root adm    7889300 Feb 14 00:00 daemon.log.1
-rw-r-----  1 root adm     211520 Feb 14 10:59 debug
-rw-r-----  1 root adm     262449 Feb 14 00:00 debug.1
-rw-r-----  1 root adm    3699580 Feb 14 12:50 kern.log
-rw-r-----  1 root adm    4613918 Feb 14 00:00 kern.log.1
-rw-rw-r--  1 root utmp    292292 Feb 13 10:23 lastlog
-rw-r-----  1 root adm      19386 Feb 14 12:50 messages
-rw-r-----  1 root adm      49427 Feb 14 00:00 messages.1
drwxr-xr-x  2 root root        60 Feb 13 10:22 nginx
-rw-------  1 root root       185 Feb 13 10:22 php7.0-fpm.log
drwxrwxr-x  2 root mmdvm       80 Feb 14 04:38 pi-star
drwx------  2 root root        40 Feb 13 10:22 private
-rw-r-----  1 root adm   10057170 Feb 14 12:50 syslog
-rw-r-----  1 root adm   12531221 Feb 14 00:00 syslog.1
drwxr-xr-x  2 root root        40 Feb 13 10:22 unattended-upgrades
-rw-r-----  1 root adm        900 Feb 14 12:17 user.log
-rw-r-----  1 root adm       3301 Feb 14 00:00 user.log.1
-rw-rw-r--  1 root utmp         0 Feb 14 00:00 wtmp
-rw-rw-r--  1 root utmp       193 Feb 13 10:23 wtmp.1.gz

and note the size of the syslog files (also high here in this example).

From there you can issue commands like "tail -n 100 syslog" (lists the last - most current - 100 lines of log; adjust the value if you want to look further back). Obviously, the syslog continuously changes, so reissuing the command from time to time will yield varying results.

Sample:

Code: Select all

pi-star@pi-star-1(ro):log$ tail -n 10 syslog
Feb 14 13:09:01 pi-star-1 CRON[4963]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Feb 14 13:09:01 pi-star-1 systemd[1]: Starting Clean php session files...
Feb 14 13:09:01 pi-star-1 systemd[1]: phpsessionclean.service: Succeeded.
Feb 14 13:09:01 pi-star-1 systemd[1]: Started Clean php session files.
Feb 14 13:10:01 pi-star-1 CRON[7375]: (root) CMD (/usr/local/sbin/pistar-upnp.service start > /dev/null 2>&1 &)
Feb 14 13:10:01 pi-star-1 systemd[1]: Started Backup MMDVM log files.
Feb 14 13:10:01 pi-star-1 kernel: [96466.447276] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Feb 14 13:10:01 pi-star-1 pistar-mmdvm-log-backups[7376]: MMDVM-2022-02-14.log (13:07:48) backed up
Feb 14 13:10:02 pi-star-1 kernel: [96467.166176] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
Feb 14 13:10:02 pi-star-1 systemd[1]: mmdvm-log-backup.service: Succeeded.

If you find an interesting term or phrase or keyword, or have something in mind to search for, you can issue 'grep -i "...." syslog | tail -n 100' (piping the results from grep to tail is important here if you happen to hit on a phrase, the phrase that is indicative of the runaway condition):
Sampe:

Code: Select all

pi-star@pi-star-1(ro):log$ grep -i "validation" syslog | tail -n 10
Feb 14 10:59:32 pi-star-1 kernel: [88637.796093] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:32 pi-star-1 kernel: [88637.901949] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:32 pi-star-1 kernel: [88638.007565] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.112810] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.218097] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.323155] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.428230] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.533756] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.639396] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16
Feb 14 10:59:33 pi-star-1 kernel: [88638.744557] ieee80211 phy0: brcmf_cfg80211_change_iface: iface validation failed: err=-16

[Ki6anr]

Awesome! That’s a great start! Since you stated these logs are not stored in the sd card itself is it safe to assume they are deleted with a reboot?

in your first example where you cd into the var/log directory, is the nezt command to “delete” it?

with that in mind, if the syslog is also large (like yours) do you delete that one as well? or are you just saying to search for possible errors causing a freeze in that one?

I appreciate your patience and insight. Good stuff!

[KN2TOD]

NO! The next command after the "cd" is "ls" (for "list") with the options "-la" (to list in column form with attributes). If you're in doubt or are curious, issue a command with "--help" as the only arg (e.g. "ls --help" or "df --help") to get the full details.

For now, we're just exploring/diagnosing, collecting information to find out what the root problem is and what can be done about it, one step at a time.

[Ki6anr]

Gotcha. I will check it out tonight. Thanks again. Also, Andy hasn’t got back to me about another question above, perhaps you know the answer. It is my understanding pi-star does an “update check” every night at about 3am, does that typically reboot pi-star? Like I mentioned, my hotspot just so happened to survive the night last night eithout freezing however in the morning I had found it disconnected to the reflector I had it linked to. It was unlinked this morning (I have it default to boot and not connect to a reflector temporarily) which tells me it must have rebooted?

If the nightly “update” does in fact reboot the pi-star shouldnt it also purge the logs? I read a post from Andy somewhere else on this forum that pi-star is supposed to automatically clean up the logs so I am curious as to 1) why your logs are backing up and 2) if the nightly update reboots should this not happen anyway.

[KN2TOD]

The nightly updates do not reboot. Unlinking is a different and somewhat involved problem, best discussed and followed in other threads. (In general, various problems can occur anywhere in your home network when your ISP decides to reset a block of IP address connections during the night.)

Your hotspot may have survived last night simply because the daily log rotation got to act before things got out of hand - it's all of matter of timing, and not necessarily repeatable night after night. (Log rotations are on a different schedule than the nightly updates as I recall, so ....)

[KN2TOD]

To clarify:

Most of the logs/files are kept in temp storage (i.e. memory), so a reboot naturally clears all such logs. Handy when you generally need to reset a failed/failing system and start over, but problematic when you need to do a post-mortem analysis.

"Rotation" is kind of a generic term referring to several different strategies: compress a log and save and start a new log (age off the oldest saved log); rename current log and start a new log (again, age off the oldest renamed log); or trim a log (cut it down by a certain percentage of lines or size, for instance). Temp files (web pages or print files for instance) can be also be handled in a similar fashion.

Some logs/files are rotated daily (like 3 am, as part of the nightly process), some hourly, and some intermittently based on size or depth (number of lines).

Nothing complicated here, but difficult to explain all the in's and out's of a particular installation's setup.

The file we're interested in here, syslog, is rotated at midnight (via strategy #2) but whose growth is otherwise unconstrained over a 24 hour period.

But this particular file may not be what's causing your system failure, but it's the file we need to look in (for a start) to find what's going on in your system that's causing it to fail.

[M1DNS]

Whilst the overnight updates don't reboot the pi, they do restart services. So if you don't have a startup reflector set. It will come back up unlinked.

Sent via smoke signals from my SM-G935F M1DNS (Admin)