Unable to Switch Back to 'rpi-ro'

General support for the Pi-Star System
KN2TOD
Posts: 264
Joined: Sun Nov 11, 2018 6:36 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KN2TOD »

I too have been experiencing the same phenom recently, albeit with a something different scenario:

For a good while now, my various hotspots have been stable, with only occasional "lockups" in RW state which I had typically attributed to rare events like power sags or interrupted updates of various sorts.

But around the first week in December, one by one my hotspots started stalling in RW states. Reboots restored them to RO status except 24 hours later they would be back in RW states. They would not flip all a once, but rather one here, two the next day, and so on. The change from RO to RW seemed to be happened at night (during the nightly updates?) but nothing in the logs seemed to be amiss or otherwise indicate exactly when the statuses flipped. So I created a logging script and dropped it into the cron.hourly folder (running at 17 after each hour) to see if I could narrow down the time frame.

But, while puzzling over this turn of events, I happened to notice that certain system updates (apt update/upgrade sequences) would also lock the systems in RW state. Annoying but not entirely unexpected. And then - coincidentally - a thread about "unattended updates" appeared in the forums.

Checking the "UU" directory strongly suggested that this is/was the source of the RW state: the UU updates continued for another day or two and then stopped - and so did the mysterious locked RW states. (Obviously, my experience here is based on the particular way I run/maintain my hotspots, so YMMV.)

Your general observations fit with mine: given what I see in typical system updates - the creation and deletion of numerous backup and temp files - it is not surprising that there are/may be some loose (open) files left around at the end of such machinations, especially if they are done "unattended".

More research is needed, obviously.

---

My logging routine remains in place, as will my review of the UU files:

Code: Select all

logger -t "[$$]" "Pi-Star --> Disk Status: $(grep 'dev/root' /proc/mounts | sed -n 's%.*\(r[ow]\),.*%\1%p' | tr 'a-z' 'A-Z' | sed 's%RW%RW*%g')"
(the results can be grepped via "-->" in the system logs)

And the unattended update task continues on, but nothing happening since 12/12-13:

Code: Select all

pi-star@pi-star-1(ro):~$ lsll /var/log/unattended-upgrades
total 12
-rw-r--r-- 1 root root    0 Dec 13 06:39 unattended-upgrades-dpkg.log
-rw-r--r-- 1 root root 8265 Dec 27 06:33 unattended-upgrades.log
-rw-r--r-- 1 root root    0 Dec 12 21:22 unattended-upgrades-shutdown.log
pi-star@pi-star-1(ro):~$ 
(Not clear who decides what is included in such updates or when they should be applied. Can any Linux gurus shed light on this?)
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

There is an unattended-upgrades package, it is designed to update security related packages on the RPi OS system, so it doesn't do everything.

There are other ways of doing that, for instance using sudo pistar-update at an ssh terminal prompt, but if you do that then sometimes there will be packages that are held back so you also can do the following:

sudo rpi-rw
sudo apt-get update
sudo apt-get dist-upgrade
sudo rpi-ro

If that last doesn't get you ro in the prompt then you will need to restart the system.

HTH
--

Brian G8SEZ
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

KE7FNS wrote: Sun Dec 27, 2020 8:28 pm
G8SEZ wrote: Sun Dec 27, 2020 8:19 pm sudo rpi-rw
sudo apt-get update
sudo apt-get dist-upgrade
sudo rpi-ro
Just FYI, you don't need to rpi-rw before you execute apt-get, the apt-get command gets intercepted and a script puts the system in RW, executes apt-get and then puts the system back in ro when done.

Also, the new preferred method seems to have shifted from apt-get to apt, and dist-upgrade has shifted to full-upgrade. Andy doesn't have the scripts setup for apt to do the auto RW/RO stuff either, but it is easily copied from how he did the apt-get one.
Stuff is always changing, I will confess that I do know about (and use) the full-upgrade change but fingers were typing from muscle memory.

rpi-rw is a sort of can't-do-any-harm command. But I get your point.
--

Brian G8SEZ
KN2TOD
Posts: 264
Joined: Sun Nov 11, 2018 6:36 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KN2TOD »

Surprise, surprise! Got a hit today: unattended update early this morning and now all my HS's that show an update are locked in a RW state:

Code: Select all

Dec 28 00:17:04 pi-star-3 [21565]: Pi-Star --> Disk Status: RO
Dec 28 01:17:04 pi-star-3 [25266]: Pi-Star --> Disk Status: RO
Dec 28 02:17:03 pi-star-3 [1901]: Pi-Star --> Disk Status: RO
Dec 28 03:17:05 pi-star-3 [15393]: Pi-Star --> Disk Status: RO
Dec 28 04:17:04 pi-star-3 [22207]: Pi-Star --> Disk Status: RO
Dec 28 05:17:04 pi-star-3 [2890]: Pi-Star --> Disk Status: RO
Dec 28 06:17:04 pi-star-3 [13112]: Pi-Star --> Disk Status: RW*
Dec 28 07:17:02 pi-star-3 [24773]: Pi-Star --> Disk Status: RW*
Dec 28 08:17:03 pi-star-3 [29345]: Pi-Star --> Disk Status: RW*
Dec 28 09:17:03 pi-star-3 [5352]: Pi-Star --> Disk Status: RW*
Dec 28 10:17:02 pi-star-3 [16965]: Pi-Star --> Disk Status: RW*

-rw-r--r-- 1 root adm  1680 Dec 28 06:16 unattended-upgrades-dpkg.log
-rw-r--r-- 1 root root 8958 Dec 28 06:16 unattended-upgrades.log
-rw-r--r-- 1 root root    0 Dec 12 21:14 unattended-upgrades-shutdown.log
Those of my HS's that show no unattended updates are still in RO status, but I'm assuming/betting they will roll over in the next 24 hours.
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

Looks like the python3-apt package updated and unlike the tzdata update a few days ago it requires a reboot.

3 out of my 4 PiStars have done this, must be down to the randomised timing of the updates so one of them missed it.
--

Brian G8SEZ
KN2TOD
Posts: 264
Joined: Sun Nov 11, 2018 6:36 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KN2TOD »

OH3NFC wrote: Sun Dec 27, 2020 12:43 pm I used a command:

Code: Select all

sudo lsof +f -- / | awk '$4 == "FD" || $4 ~ /[0-9][wu]$/'
which revealed this:

Code: Select all

COMMAND    PID        USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
dhclient   832        root    4w   REG  179,2      961   1719 /var/lib/dhcp/dhclient.eth1.leases
73 de Veijo OH4VA/OH3NFC
The LSOF-AWK combo command above did not yield any results, but a brute-force GREP did. Does this/can this combo command be tweaked? And, does anything jump out here relative to the last unattended update reported here?

Code: Select all

pi-star@pi-star-3(rw):~$ lsof +f -- / 2>/dev/null | awk '$4 == "FD" || $4 ~ /[0-9][wu]$/'
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME

pi-star@pi-star-3(rw):~$ lsof | grep -ie "[0-9][wu]"
lsof        837                     pi-star    0u      CHR      136,0      0t0          3 /dev/pts/0
lsof        837                     pi-star    1w     FIFO       0,12      0t0   43824345 pipe
lsof        837                     pi-star    2u      CHR      136,0      0t0          3 /dev/pts/0
lsof        837                     pi-star    5w     FIFO       0,12      0t0   43824353 pipe
grep        838                     pi-star    1u      CHR      136,0      0t0          3 /dev/pts/0
grep        838                     pi-star    2u      CHR      136,0      0t0          3 /dev/pts/0
lsof        839                     pi-star    7w     FIFO       0,12      0t0   43824354 pipe
systemd   23235                     pi-star    1u     unix 0x68e56714      0t0   16004618 type=STREAM
systemd   23235                     pi-star    2u     unix 0x68e56714      0t0   16004618 type=STREAM
systemd   23235                     pi-star    3u     unix 0x2ed65ad0      0t0   16004643 type=DGRAM
systemd   23235                     pi-star    4u  a_inode       0,13        0       5288 [eventpoll]
systemd   23235                     pi-star    5u  a_inode       0,13        0       5288 [signalfd]
systemd   23235                     pi-star    8u  a_inode       0,13        0       5288 [timerfd]
systemd   23235                     pi-star    9u  netlink                 0t0   16004679 KOBJECT_UEVENT
systemd   23235                     pi-star   10u  a_inode       0,13        0       5288 [eventpoll]
systemd   23235                     pi-star   12u     unix 0xaf752074      0t0   16004697 /run/user/1000/gnupg/S.gpg-agent.browser type=STREAM
systemd   23235                     pi-star   16u     unix 0x6f1ccbad      0t0   16004682 /run/user/1000/systemd/notify type=DGRAM
systemd   23235                     pi-star   17u     unix 0xcf0a1a1c      0t0   16004684 type=DGRAM
systemd   23235                     pi-star   18u     unix 0x905e075a      0t0   16004685 type=DGRAM
systemd   23235                     pi-star   19u     unix 0x107667bb      0t0   16004686 /run/user/1000/systemd/private type=STREAM
systemd   23235                     pi-star   20u     unix 0x823554f8      0t0   16004688 type=STREAM
systemd   23235                     pi-star   21u  a_inode       0,13        0       5288 [timerfd]
systemd   23235                     pi-star   26u     unix 0xadd67207      0t0   16004700 /run/user/1000/bus type=STREAM
systemd   23235                     pi-star   27u     unix 0xb9a86bce      0t0   16004736 /run/user/1000/gnupg/S.gpg-agent.extra type=STREAM
systemd   23235                     pi-star   28u     unix 0x4282f810      0t0   16004740 /run/user/1000/gnupg/S.dirmngr type=STREAM
systemd   23235                     pi-star   29u     unix 0xa28f0221      0t0   16004742 /run/user/1000/gnupg/S.gpg-agent.ssh type=STREAM
systemd   23235                     pi-star   30u     unix 0xa02f46b6      0t0   16004744 /run/user/1000/gnupg/S.gpg-agent type=STREAM
bash      30664                     pi-star    0u      CHR      136,0      0t0          3 /dev/pts/0
bash      30664                     pi-star    1u      CHR      136,0      0t0          3 /dev/pts/0
bash      30664                     pi-star    2u      CHR      136,0      0t0          3 /dev/pts/0
bash      30664                     pi-star  255u      CHR      136,0      0t0          3 /dev/pts/0
W4JEW
Posts: 59
Joined: Sun Aug 12, 2018 12:53 am
Location: Atlanta, GA, United States
Contact:

Re: Unable to Switch Back to 'rpi-ro'

Post by W4JEW »

W4JEW wrote: Tue Aug 14, 2018 9:06 pm Yes - because rebooting is always a good fix. I ran 'lsof' to see if I could identify what was causing the issue but there were so many files held open.

I was hoping to avoid the reboot...

Check out GeorgiaDMR.net - https://www.georgiadmr.net
And on Groups.io - https://groups.io/g/GeorgiaDMR

Jeff Hochberg
W4JEW
Atlanta, GA
User avatar
G8SEZ
Posts: 553
Joined: Fri Apr 13, 2018 8:26 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by G8SEZ »

Seeing this happen quite regularly now, all 4 Pi-Stars are doing it, 2 Pi Zero W, 1 Pi Zero 2W and 1 Pi 2B.

Hard to identify the culprit.
--

Brian G8SEZ
AF6VN
Posts: 821
Joined: Fri Jul 20, 2018 1:15 am

Re: Unable to Switch Back to 'rpi-ro'

Post by AF6VN »

G8SEZ wrote: Wed Jun 01, 2022 6:21 pm Seeing this happen quite regularly now, all 4 Pi-Stars are doing it, 2 Pi Zero W, 1 Pi Zero 2W and 1 Pi 2B.

Hard to identify the culprit.
Likely some OS file (stating the obvious) that is in use and received an update (replacement) -- the file system can't purge the "running" version of the file until the reboot. Unlike Windows, the Linux file system can replace the directory entry /to/ a file so it points to a different location, but running/open files are still locked to their original entry.

--
AF6VN
Dennis L Bieber
KN2TOD
Posts: 264
Joined: Sun Nov 11, 2018 6:36 pm

Re: Unable to Switch Back to 'rpi-ro'

Post by KN2TOD »

FWIW:

Happened to try out DHCLIENT recently. It appears it can/will inadvertently put the system in permanent RW mode:

Code: Select all

pi-star@pi-star(ro):~$ sudo dhclient -v
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

can't create /var/lib/dhcp/dhclient.leases: Read-only file system
Listening on LPF/wlan0/dc:a6:32:00:d5:f7
Sending on   LPF/wlan0/dc:a6:32:00:d5:f7
Listening on LPF/eth0/dc:a6:32:00:d5:f6
Sending on   LPF/eth0/dc:a6:32:00:d5:f6
Sending on   Socket/fallback
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 5
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 6
DHCPOFFER of 192.168.1.34 from 192.168.1.253
DHCPREQUEST for 192.168.1.34 on wlan0 to 255.255.255.255 port 67
DHCPACK of 192.168.1.34 from 192.168.1.253
/sbin/dhclient-script: 46: /etc/dhcp/dhclient-enter-hooks.d/samba: cannot create /var/lib/samba/dhcp.conf.new: Read-only file system
mv: cannot stat '/var/lib/samba/dhcp.conf.new': No such file or directory
can't create /var/lib/dhcp/dhclient.leases: Read-only file system
bound to 192.168.1.34 -- renewal in 36530 seconds.
pi-star@pi-star(ro):~$ rpi-rw
pi-star@pi-star(rw):~$ sudo dhclient -v
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/wlan0/dc:a6:32:00:d5:f7
Sending on   LPF/wlan0/dc:a6:32:00:d5:f7
Listening on LPF/eth0/dc:a6:32:00:d5:f6
Sending on   LPF/eth0/dc:a6:32:00:d5:f6
Sending on   Socket/fallback
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 6
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7
DHCPOFFER of 192.168.1.45 from 192.168.1.253
DHCPREQUEST for 192.168.1.45 on wlan0 to 255.255.255.255 port 67
DHCPACK of 192.168.1.45 from 192.168.1.253
bound to 192.168.1.45 -- renewal in 32473 seconds.
pi-star@pi-star(rw):~$ rpi-ro
mount: /: mount point is busy.
pi-star@pi-star(rw):~$ 
ummmm.... I wonder what it's up to.
Post Reply