RPI_V4.1.0-RC7 - Disk full

General support for the Pi-Star System
Post Reply
N4ZWR
Posts: 2
Joined: Tue Nov 20, 2018 10:36 pm

RPI_V4.1.0-RC7 - Disk full

Post by N4ZWR » Fri Jan 03, 2020 8:31 pm

Hello,
I am having an interesting problem that I have no idea where to correct. I loaded 4.1.0-RC7 on a PI3B+ and the hotspot stops running after several hours. I found that the looking at DMESG that it failed to open the runtime journal as noted from the log entries below. doing a DF on the cmdline it looks like /run is at 100% full, and I have seen also that /var/log is 100%. I thought that I might have a bad image or something, but I reloaded with new SD card and it seems to still be happening. apart from the log messages below, and a DF that shows /run and sometimes /var/log full I am at a loss as to where to look for further details about what is going on. I have searched the forum with not a lot of information found. Is this a bug in RC7 or is this something that on my side is causing the problem. as a side note, I was previously running 4.0 version and didn't see this problem, hotspot ran for weeks with no trouble.

anyone else seeing this type behavior?

Thanks,
Sam N4ZWR

**DF **
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 15213412 1528340 13031388 11% /
devtmpfs 367684 0 367684 0% /dev
tmpfs 499268 0 499268 0% /dev/shm
tmpfs 32768 32768 0 100% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 32768 0 32768 0% /sys/fs/cgroup
tmpfs 16 0 16 0% /var/lib/logrotate
tmpfs 16 0 16 0% /var/lib/sudo
tmpfs 32 16 16 50% /var/lib/dhcpcd5
tmpfs 1024 72 952 8% /var/cache/samba
tmpfs 65536 11952 53584 19% /var/log
tmpfs 64 0 64 0% /var/lib/php/sessions
tmpfs 65536 4 65532 1% /tmp
tmpfs 1024 0 1024 0% /var/lib/nginx/body
tmpfs 4096 2176 1920 54% /var/lib/samba/private
/dev/mmcblk0p1 64456 54416 10040 85% /boot


**DMESG Excert**
[Fri Jan 3 13:50:13 2020] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[Fri Jan 3 14:17:02 2020] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[Fri Jan 3 14:17:02 2020] systemd-journald[102]: Received request to rotate journal from PID 1
[Fri Jan 3 14:17:02 2020] systemd-journald[102]: Vacuuming done, freed 0B of archived journals from /run/log/journal/18634e1404184594bc66ddacfba0bf43.
[Fri Jan 3 14:17:02 2020] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to create new runtime journal: No space left on device
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to open user journal directory '/var/log/journal/18634e1404184594bc66ddacfba0bf43/': No such file or directory
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: File /run/log/journal/18634e1404184594bc66ddacfba0bf43/system.journal corrupted or uncleanly shut down, renaming and replacing.
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: File /run/log/journal/18634e1404184594bc66ddacfba0bf43/system.journal corrupted or uncleanly shut down, renaming and replacing.
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: File /run/log/journal/18634e1404184594bc66ddacfba0bf43/system.journal corrupted or uncleanly shut down, renaming and replacing.
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: File /run/log/journal/18634e1404184594bc66ddacfba0bf43/system.journal corrupted or uncleanly shut down, renaming and replacing.
[Fri Jan 3 15:11:17 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device
[Fri Jan 3 15:11:22 2020] systemd-journald[102]: Failed to open runtime journal: No space left on device

kc7ngc
Posts: 48
Joined: Fri Sep 21, 2018 2:47 am

Re: RPI_V4.1.0-RC7 - Disk full

Post by kc7ngc » Sat Jan 04, 2020 1:22 am

My first guess is something major is going wrong with the system that is generating a large amount of errors in the systemd journal as that is the only thing that I can think of that would fill up both /var/run and /var/log. Large quantities of sdcard read errors due to bad sdcards are one of many potential causes.

So step #1 is to find out what is filling up /var/run. Run the following on the system to identify the large files under var/run.

Code: Select all

du /var/run/ | sort -g
Largest files should be at the bottom. From previous experience I am guessing the largest file is going to be under /var/run/log/journal which is the systemd journal file.

If that is the case need to run journalctl and look at thee log to try to find out what is filling up the log:

Code: Select all

journalctl | less 
That should show you the contents of the journal and allow you to page through it.

It's going to be normal to see things like watchdog services in that and a few pi-star services saying they failed to start every 5 minutes or so. what you are looking for is things that are happening many times a minute every minute. That might indicate what is filling up the log.

If the largest file from the du command turns out to be something other than the journal then post the output and will see what we can do to figure out what is going on.

User avatar
VK7HSE
Posts: 192
Joined: Sun Apr 15, 2018 6:35 pm
Contact:

Re: RPI_V4.1.0-RC7 - Disk full

Post by VK7HSE » Sat Jan 04, 2020 11:18 am

I've found that most of the time it's the pi-star logs that are not getting purged during the nightly updates. So after a few days if you have external screens (nextion) they become unresponsive and the dashboard will also appear to be frozen.

Also check what log level you are using as it's handy to have a verbose log when you setting up your hotspot. But will lead to problems similar to what you are currently experiencing!

Sent from my Pixel 4 XL using Tapatalk


User avatar
W1KMC
Posts: 111
Joined: Wed May 02, 2018 8:34 pm
Location: Bangor, ME USA
Contact:

Re: RPI_V4.1.0-RC7 - Disk full

Post by W1KMC » Sat Jan 04, 2020 12:00 pm

Disk full could be as simple as the card you are using is too small (4GB works but 8,16,32 would be better (I use 16GB cards [class 10 or U1] in mine) )

I don't think the image does it automatically so also try running

sudo pistar-expand

on any new images you make as a first command in SSH (through dashboard expert mode or a terminal program (PuTTY on windows or something similar on a MAC, terminal in linux)

once you run this reboot Pistar and continue as usual.

what this does is expands the card's file system to use the remaining space left for files.

Try this and let us know.
I am no developer, guru, expert, nor do I claim to be or want to be. All advice I give is purely from experience and my efforts to help others.

W1KMC - Kevin M Carman
Bangor, ME
Jumbospot on a Pi0W
DMR 3123142 & 3123143
HHUS Trunk Ext 4329

N4ZWR
Posts: 2
Joined: Tue Nov 20, 2018 10:36 pm

Re: RPI_V4.1.0-RC7 - Disk full

Post by N4ZWR » Sat Jan 04, 2020 12:15 pm

Looks like I have found my issue. running du /var/run | sort -g from the cmdline, I found that
/var/run/log/journal/18634e1404184594bc66ddacfba0bf43 was the largest file. Doing Journalctl | less, I found lots of NextionDriver[719] messages occurring every second, like updating clock on the nextion display.

After looking at the loglevel in expert/MMDVMHOST for the NextionDriver, it looks like the log level was set to 4, which is verbose. I guess that when they updated the install for the nextion driver, they set the default to 4 which explains all of the nextion driver messages. I lowered it to 2 and have been running for 12+ hours and all looks much better now.

The only other message that I see that is repeating every minute or so multiple times a second is for

minissdpd[718]: peer 169.254.27.210:59713 is not from a LAN

Not sure what this one is, but my major contributor for locking things up was the nextion driver.

Thanks for the info and what to look for, I greatly appreciate it.

Thanks,
Sam N4ZWR

kc7ngc
Posts: 48
Joined: Fri Sep 21, 2018 2:47 am

Re: RPI_V4.1.0-RC7 - Disk full

Post by kc7ngc » Sat Jan 04, 2020 4:38 pm

N4ZWR wrote:
Sat Jan 04, 2020 12:15 pm
Looks like I have found my issue
Glad you figured it out. /var/log and /var/run are just small temp file systems that only exist in ram and not on the sdcard when running under the pistar image. So under rare occasions such as error conditions or just verbose logging it can fill up before nightly prune operation can trim things. Glad you were able to fairly easily remedy the situation

Does look like that in early November the author of the NextionDriver changed the config for the NextionDriverInstall script to a loglevel of 4:
https://github.com/on7lds/NextionDriver ... Driver.cfg
Which is going to cause problems for anyone running pistar when the NextionDriver is logging at that verbosity.
N4ZWR wrote:
Sat Jan 04, 2020 12:15 pm
The only other message that I see that is repeating every minute or so multiple times a second is for

minissdpd[718]: peer 169.254.27.210:59713 is not from a LAN
The 169.254.0.0/16 IP space is used by devices when they fail to get a DHCP address on a network. Most common is a Windows box that comes up, can't get an ip address, so it self selects an ip in that range. Since the minisspdpd is a upnp daemon on the raspberry pi that listens to broadcast traffic I am guessing that somewhere on the network is a windows system or device that came up and didn't get a valid IP address. The service is hearing it and deciding it doesn't know what to do with the traffic from that ip. So you might want to check your network and see if you have a unhappy computer someplace that isn't properly configured on the network.

M1DNS
Posts: 719
Joined: Thu Apr 05, 2018 5:30 am

Re: RPI_V4.1.0-RC7 - Disk full

Post by M1DNS » Sat Jan 04, 2020 6:44 pm

W1KMC wrote:Disk full could be as simple as the card you are using is too small (4GB works but 8,16,32 would be better (I use 16GB cards [class 10 or U1] in mine) )

I don't think the image does it automatically so also try running

sudo pistar-expand.
Pistar 4.1 runs auto expand on the very first initial boot up.

Sent from my SM-G935F using Tapatalk


Andrew M1DNS.
Pi-star Admin Team.

User avatar
W1KMC
Posts: 111
Joined: Wed May 02, 2018 8:34 pm
Location: Bangor, ME USA
Contact:

Re: RPI_V4.1.0-RC7 - Disk full

Post by W1KMC » Sun Jan 05, 2020 1:23 am

Thanks Andy, I was not aware it did. On my 3.4 I think I had to do it manually.But if 4.1 does this for you, I stand corrected and appologize for my mis-statement.
I am no developer, guru, expert, nor do I claim to be or want to be. All advice I give is purely from experience and my efforts to help others.

W1KMC - Kevin M Carman
Bangor, ME
Jumbospot on a Pi0W
DMR 3123142 & 3123143
HHUS Trunk Ext 4329

M1DNS
Posts: 719
Joined: Thu Apr 05, 2018 5:30 am

Re: RPI_V4.1.0-RC7 - Disk full

Post by M1DNS » Sun Jan 05, 2020 8:14 am

W1KMC wrote:Thanks Andy, I was not aware it did. On my 3.4 I think I had to do it manually.But if 4.1 does this for you, I stand corrected and appologize for my mis-statement.
No need for apologies kev, no harm done. ImageImage

Sent from my SM-G935F using Tapatalk

Andrew M1DNS.
Pi-star Admin Team.

Post Reply