Unable to Switch Back to 'rpi-ro'

General support for the Pi-Star System
W4JEW
Posts: 58
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW » Tue Mar 03, 2020 8:17 am

Well, this is still an issue. I appreciate everyone's comments as to what they did that resolved the issue for them, but I continue to see very inconsistent behavior.

I can't tell you how many times I speak with others running Pi-Star where they didn't realize how important it was to shut down their hotspots gracefully. They simply disconnect the power from the device which makes me cringe! Of course, the first thing I check is to see if their SD card became corrupted from an improper shutdown. More often than not, that's exactly what happened.

At this time, I have three Pi-Star based hotspots - 2x running 4.1.0 RC8 and 1x running 3.4.17. Both of the hotspots running 4.1.0 RC8 are perpetually stuck in read-write mode while the one running 3.4.17 is happily running in read-only mode.

Andy - what can we do to help get to the bottom of this issue?
Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA

User avatar
KE0FHS
Posts: 1110
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by KE0FHS » Tue Mar 03, 2020 3:46 pm

W4JEW wrote:
Tue Mar 03, 2020 8:17 am
Well, this is still an issue.
My personal troubleshooting procedure for this issue has been to always update manually via SSH so that I get any Raspbian OS updates along with the Pi-Star dashboard updates. My process is to log into Pi-Star via SSH, run sudo pistar-update, then reboot the hotspot, then repeat until the update doesn't result in any changes and ends up correctly in read-only mode. Then I run sudo pistar-upgrade.

Since I did this in the latter half of Feb 2020 and ended up on Pi-Star 4.1.0-RC8 20200221, I haven't again experienced the issue of Pi-Star getting stuck in read-write mode. I don't know if this is due to changes in Pi-Star, which have been minimal, or changes to the OS, which were significant, and it's too early to say for sure that this issue has been fixed, but so far, so good.

Again, I think the key to my experience is to run the update manually via SSH in order to get the Raspbian OS updates as well.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

KE7FNS
Posts: 1035
Joined: Wed Apr 17, 2019 11:11 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KE7FNS » Tue Mar 03, 2020 9:59 pm

W4JEW wrote:
Tue Mar 03, 2020 8:17 am
Of course, the first thing I check is to see if their SD card became corrupted from an improper shutdown. More often than not, that's exactly what happened.
What files are being corrupted when you check their SD card and how are you checking it to determine that is has been corrupted?
W4JEW wrote:
Tue Mar 03, 2020 8:17 am
At this time, I have three Pi-Star based hotspots - 2x running 4.1.0 RC8 and 1x running 3.4.17. Both of the hotspots running 4.1.0 RC8 are perpetually stuck in read-write mode while the one running 3.4.17 is happily running in read-only mode.
The simplest and easiest solution to your problem is to just reboot every once in a while and not dwell on it.


During my own testing I found a way I can easily force the system back into RO mode at any time but I'm unsure of the ramifications of doing it that way, so I'd rather only discuss it in private with someone at least as knowledgeable with linux as myself. So far nobody has bothered to reply.
If someones previous actions are any indication of their future actions, then I predict the deletion and removal of access will happen at any moment. 7-11-2020.

"07/13/20 This Website Has Been Taken Down" ... again :lol:

User avatar
G8SEZ
Posts: 236
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ » Wed Mar 04, 2020 7:27 pm

KE0FHS wrote:
Tue Mar 03, 2020 3:46 pm
W4JEW wrote:
Tue Mar 03, 2020 8:17 am
Well, this is still an issue.
My personal troubleshooting procedure for this issue has been to always update manually via SSH so that I get any Raspbian OS updates along with the Pi-Star dashboard updates. My process is to log into Pi-Star via SSH, run sudo pistar-update, then reboot the hotspot, then repeat until the update doesn't result in any changes and ends up correctly in read-only mode. Then I run sudo pistar-upgrade.

Since I did this in the latter half of Feb 2020 and ended up on Pi-Star 4.1.0-RC8 20200221, I haven't again experienced the issue of Pi-Star getting stuck in read-write mode. I don't know if this is due to changes in Pi-Star, which have been minimal, or changes to the OS, which were significant, and it's too early to say for sure that this issue has been fixed, but so far, so good.

Again, I think the key to my experience is to run the update manually via SSH in order to get the Raspbian OS updates as well.
The RC8 upgrade added in the rng-tools package, that is the thing that appears to have made the difference. I have not seen any problems with rw mode sticking since this change. It may be that people are not aware of what happened and why but the github history should show it for reference.
--

Brian G8SEZ

KE7FNS
Posts: 1035
Joined: Wed Apr 17, 2019 11:11 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KE7FNS » Thu Mar 05, 2020 4:05 am

G8SEZ wrote:
Wed Mar 04, 2020 7:27 pm
The RC8 upgrade added in the rng-tools package, that is the thing that appears to have made the difference.
I manually run daily apt updates and apt upgrades on my systems which have been on RC8 since release and they still get stuck in RW. Adding rngtools didn't solve the issue, so I just reboot it when needed.
If someones previous actions are any indication of their future actions, then I predict the deletion and removal of access will happen at any moment. 7-11-2020.

"07/13/20 This Website Has Been Taken Down" ... again :lol:

User avatar
G8SEZ
Posts: 236
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ » Thu Mar 05, 2020 3:58 pm

KE7FNS wrote:
Thu Mar 05, 2020 4:05 am
G8SEZ wrote:
Wed Mar 04, 2020 7:27 pm
The RC8 upgrade added in the rng-tools package, that is the thing that appears to have made the difference.
I manually run daily apt updates and apt upgrades on my systems which have been on RC8 since release and they still get stuck in RW. Adding rngtools didn't solve the issue, so I just reboot it when needed.
Quite odd then, I don't have any pearls of wisdom to offer, like you I run manual updates when necessary but usually I find that sudo pistar-update and -upgrade are just fine when run from an ssh session. Occasionally it is necessary to use apt-get update && apt-get dist-upgrade to get new packages that Raspbian have added to the distro base.

I should add that if you don't use the pistar-* scripts then the apt-get process leaves the SD card in rw mode, so you need sudo rpi-ro afterwards to fix that.
--

Brian G8SEZ

KE7FNS
Posts: 1035
Joined: Wed Apr 17, 2019 11:11 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KE7FNS » Thu Mar 05, 2020 6:54 pm

G8SEZ wrote:
Thu Mar 05, 2020 3:58 pm
I should add that if you don't use the pistar-* scripts then the apt-get process leaves the SD card in rw mode, so you need sudo rpi-ro afterwards to fix that.
Yes, I am fully aware of that. I have setup my own script so that when I run apt (note: NOT apt-get) it sets the filesystems to RW, runs apt, and then returns the system to RO similar to how the pistar-upgrade and pistar-update scripts would do.
If someones previous actions are any indication of their future actions, then I predict the deletion and removal of access will happen at any moment. 7-11-2020.

"07/13/20 This Website Has Been Taken Down" ... again :lol:

W4JEW
Posts: 58
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW » Fri Mar 06, 2020 11:30 pm

I wish that rebooting fixed the issue - even temporarily. It does not. Any time run Pistar-update/upgrade the final message beside exiting the script is “mount / is busy”

When the script exits, the filesystem is (rpi-rw).

There’s something that’s holding files open that doesn’t want to release them long enough to allow Pi-Star to flip back to read only.

Running lsof shows tons of files open so it’s difficult to tell which it is.
Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA

KE7FNS
Posts: 1035
Joined: Wed Apr 17, 2019 11:11 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KE7FNS » Sat Mar 07, 2020 12:35 am

W4JEW wrote:
Fri Mar 06, 2020 11:30 pm
I wish that rebooting fixed the issue - even temporarily. It does not.
That is not a true statement at all. Clearly when the RPi is booted you can look at the terminal and see that it is in RO mode. In fact when the RPi is rebooted, the filesystems in question are loaded from scratch as RO. (cat /etc/fstab if you want proof, it is mounting them as RO) The RW problem arises after certain services and scripts have been executed. Some of those get executed automatically right after the system is brought up and stable.
W4JEW wrote:
Fri Mar 06, 2020 11:30 pm
There’s something that’s holding files open that doesn’t want to release them long enough to allow Pi-Star to flip back to read only.

Running lsof shows tons of files open so it’s difficult to tell which it is.
Correct. Files that were used during the apt-get upgrade process were opened, and then deleted, the filesystem still has them marked as open.

It is not one single file holding up the remount, its tens, hundreds or even thousands of files that are completely different depending on which packages were touched when it was installed/upgraded.

There isn't a simple way to fix this short of two options:

1. quit updating raspbian which is highly not recommended, or
2. reboot.
KE7FNS wrote:
Tue Mar 03, 2020 9:59 pm
W4JEW wrote:
Tue Mar 03, 2020 8:17 am
Of course, the first thing I check is to see if their SD card became corrupted from an improper shutdown. More often than not, that's exactly what happened.
What files are being corrupted when you check their SD card and how are you checking it to determine that is has been corrupted?
You also forgot to answer the questions I asked of how you are determining that the SD card is corrupted, so I'll try again...
If someones previous actions are any indication of their future actions, then I predict the deletion and removal of access will happen at any moment. 7-11-2020.

"07/13/20 This Website Has Been Taken Down" ... again :lol:

AF4FA
Posts: 42
Joined: Wed Jan 23, 2019 6:20 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by AF4FA » Sun Mar 08, 2020 2:57 pm

I do not know what he uses to check the system but I use the following.
sudo dmesg and then sudo fsck.

Post Reply