Unable to Switch Back to 'rpi-ro'

General support for the Pi-Star System
KD8DVR
Posts: 106
Joined: Tue Aug 14, 2018 6:23 pm
Location: Columbus, OH USA
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by KD8DVR »

KE0FHS wrote: Sat Oct 19, 2019 2:10 pm
KD8DVR wrote: Sat Oct 19, 2019 1:10 am Yeah. It goes back to ro after an update.
This bug has always been a bit random for me, going to rw sometimes, but not always; so while it's great to see some people having success seeing it go back to ro after an update, I'll be waiting a week or two before I start thinking it's really solved.
Yeah. I can understand being cautious.
AntiSquid Disclaimer: All messages posted are personal opinion only and may not constitute fact.
AF6VN
Posts: 821
Joined: Fri Jul 20, 2018 1:15 am

Re: Unable to Switch Back to 'rpi-ro'

Post by AF6VN »

Well - to add to the situation... R-Pi 3B Pi-Star 3.4.17

After an uptime of over 16 days, I just ran the sequence over in the WiFi category explaining how to set the locale (it had been using en-GB, I switched it to en-US -- and I should state that just using dpkg-reconfigure was not sufficient; had to use raspi-config to ensure the environment was set)... And my unit is now stuck in the RW stage, requiring a reboot to clear.

I did run an lsof, but I'm not sure what I should be looking for in that long listing.

--
AF6VN
Dennis L Bieber
W4JEW
Posts: 59
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW »

Well, this is still an issue. I appreciate everyone's comments as to what they did that resolved the issue for them, but I continue to see very inconsistent behavior.

I can't tell you how many times I speak with others running Pi-Star where they didn't realize how important it was to shut down their hotspots gracefully. They simply disconnect the power from the device which makes me cringe! Of course, the first thing I check is to see if their SD card became corrupted from an improper shutdown. More often than not, that's exactly what happened.

At this time, I have three Pi-Star based hotspots - 2x running 4.1.0 RC8 and 1x running 3.4.17. Both of the hotspots running 4.1.0 RC8 are perpetually stuck in read-write mode while the one running 3.4.17 is happily running in read-only mode.

Andy - what can we do to help get to the bottom of this issue?
Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA
User avatar
KE0FHS
Posts: 1122
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by KE0FHS »

W4JEW wrote: Tue Mar 03, 2020 8:17 am Well, this is still an issue.
My personal troubleshooting procedure for this issue has been to always update manually via SSH so that I get any Raspbian OS updates along with the Pi-Star dashboard updates. My process is to log into Pi-Star via SSH, run sudo pistar-update, then reboot the hotspot, then repeat until the update doesn't result in any changes and ends up correctly in read-only mode. Then I run sudo pistar-upgrade.

Since I did this in the latter half of Feb 2020 and ended up on Pi-Star 4.1.0-RC8 20200221, I haven't again experienced the issue of Pi-Star getting stuck in read-write mode. I don't know if this is due to changes in Pi-Star, which have been minimal, or changes to the OS, which were significant, and it's too early to say for sure that this issue has been fixed, but so far, so good.

Again, I think the key to my experience is to run the update manually via SSH in order to get the Raspbian OS updates as well.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

KE0FHS wrote: Tue Mar 03, 2020 3:46 pm
W4JEW wrote: Tue Mar 03, 2020 8:17 am Well, this is still an issue.
My personal troubleshooting procedure for this issue has been to always update manually via SSH so that I get any Raspbian OS updates along with the Pi-Star dashboard updates. My process is to log into Pi-Star via SSH, run sudo pistar-update, then reboot the hotspot, then repeat until the update doesn't result in any changes and ends up correctly in read-only mode. Then I run sudo pistar-upgrade.

Since I did this in the latter half of Feb 2020 and ended up on Pi-Star 4.1.0-RC8 20200221, I haven't again experienced the issue of Pi-Star getting stuck in read-write mode. I don't know if this is due to changes in Pi-Star, which have been minimal, or changes to the OS, which were significant, and it's too early to say for sure that this issue has been fixed, but so far, so good.

Again, I think the key to my experience is to run the update manually via SSH in order to get the Raspbian OS updates as well.
The RC8 upgrade added in the rng-tools package, that is the thing that appears to have made the difference. I have not seen any problems with rw mode sticking since this change. It may be that people are not aware of what happened and why but the github history should show it for reference.
--

Brian G8SEZ
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

KE7FNS wrote: Thu Mar 05, 2020 4:05 am
G8SEZ wrote: Wed Mar 04, 2020 7:27 pm The RC8 upgrade added in the rng-tools package, that is the thing that appears to have made the difference.
I manually run daily apt updates and apt upgrades on my systems which have been on RC8 since release and they still get stuck in RW. Adding rngtools didn't solve the issue, so I just reboot it when needed.
Quite odd then, I don't have any pearls of wisdom to offer, like you I run manual updates when necessary but usually I find that sudo pistar-update and -upgrade are just fine when run from an ssh session. Occasionally it is necessary to use apt-get update && apt-get dist-upgrade to get new packages that Raspbian have added to the distro base.

I should add that if you don't use the pistar-* scripts then the apt-get process leaves the SD card in rw mode, so you need sudo rpi-ro afterwards to fix that.
--

Brian G8SEZ
W4JEW
Posts: 59
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW »

I wish that rebooting fixed the issue - even temporarily. It does not. Any time run Pistar-update/upgrade the final message beside exiting the script is “mount / is busy”

When the script exits, the filesystem is (rpi-rw).

There’s something that’s holding files open that doesn’t want to release them long enough to allow Pi-Star to flip back to read only.

Running lsof shows tons of files open so it’s difficult to tell which it is.
Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA
AF4FA
Posts: 47
Joined: Wed Jan 23, 2019 6:20 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by AF4FA »

I do not know what he uses to check the system but I use the following.
sudo dmesg and then sudo fsck.
W4JEW
Posts: 59
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW »

@KE7FNS

First - let me clear one thing up... I know this is not a Pi-Star issue. It's an issue that plagues the Raspberry Pi platform and is a direct result from the Pi's reliance on SD cards as their primary method of storage. Thankfully the 3B+ and 4 allow you to use alternative storage (SSD/HD or even USB sticks) which are far more robust than SD cards. I have yet to experiment with other devices such as the NanoPi and Odroid SBCs that allow you to use alternative storage mediums. I hope to be able to do that some day in the not so distant future.

I bring this issue up because I appreciate the fact that Pi-Star makes every effort possible to keep the filesystem in a read-only format to alleviate the possibility of corruption due to an abrupt shutdown. I am hopeful that there's even more that can be done and look to the developers to determine if there's more that can be done.

The last time I ran into an issue, I noticed the green activity light flashed a couple of times then nothing after I powered on the Pi. At that point, I suspected a filesystem issue.

I connected it to an external display and also connected a keyboard, then powered it back on. The OS started to boot then shortly after displayed an error and prompted me to run fsck. I removed the SD card, inserted it into a USB SD card reader, then connected it to another Raspberry Pi and ran fsck from there - the repair was successful.

I had another incident several months ago where I didn't have the ability to connect the RPi to an external display/keyboard, but I was able to connect to it with a USB UART adapter to get directly into the console.

I've become very disciplined about creating regular backups of my hotspots and shutting it down gracefully. I've also been using pistar-remote religiously when I'm unable to shut it down via traditional methods. This is a PHENOMENAL feature in Pi-Star and, IMHO, it's not leveraged anywhere near as much as it should be.

I help a lot of people with their hotspots who are not as computer savvy as others. They do not understand the importance of gracefully shutting the system down. It is not readily apparent to a lot of people that they should treat their Raspberry Pi-based hotspots differently than they do an OpenSpot which is designed to be highly resilient to power-loss.

I don't always have the ability to physically examine their hotspots, so when they fail to boot, my recommendation is to re-image and restore from backup. That process always gets them back to a working state. I also strongly encourage them to have multiple Micro SD Cards and to carry a USB/SD-Card reader with them in the event they ever need to re-image in the field.

In the end, my goal is to raise awareness to this issue and to help in any way I can to determine the root cause of the inability to fall back to read-only mode. I sincerely hope there's a way to incorporate changes that will continue to improve the resiliency of the platform. My concern is that this issue is directly tied to the reliance on devices that require SD cards as their storage medium (by default) and there's not much else that can be done.

If that ends up being the case - then so be it!
Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA
OH3NFC
Posts: 1
Joined: Fri Dec 25, 2020 11:14 am

Re: Unable to Switch Back to 'rpi-ro'

Post by OH3NFC »

Hello, I'm new on this forum.

Still this is an old thread, I'm writing about my experience on this week.

I have been fighting about two years now in a situation where the root filesystem remains rw-state after a boot.

I found this link very useful in finding out what process is hanging and preventing the read-only state, see https://unix.stackexchange.com/question ... -read-only

I used a command:

Code: Select all

sudo lsof +f -- / | awk '$4 == "FD" || $4 ~ /[0-9][wu]$/'
which revealed this:

Code: Select all

COMMAND    PID        USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
dhclient   832        root    4w   REG  179,2      961   1719 /var/lib/dhcp/dhclient.eth1.leases
A DHCP client kept the direcory /var/lib busy. This happened every time when I put a 4G USB modem in the USB-port. With the RJ45 ethernet connection everything was Okay.

In my case the solution was uncommenting two lines in /etc/network/interfaces and replace 'em with a line.

Code: Select all

#allow-hotplug eth1
#iface eth1 inet dhcp
iface eth1 inet manual
I hope this can help others too.

73 de Veijo OH4VA/OH3NFC
Post Reply