Pi-Star Update hangs on Updating DV Binaries

General support for the Pi-Star System
User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Mon May 20, 2019 1:02 pm

I continue to experience difficulties with manual Pi-Star Updates getting hung on the Updating DV Binaries step. It can take up to 30 minutes for that step to complete (this morning it took 11 minutes on the first update I ran). This has been happening for a month or two now. Once in awhile, a Pi-Star Update will run the way it used to, going through that step quickly, but the vast majority of times, it gets hung, and when it gets hung, there is usually a fatal error message when it finally completes.

This morning's update was with a ZUMspot running on a RPi 3A+ with Pi-Star 4.0.0-RC4, 20190519, but I experience this with all my various hotspots, including those still running 3.4.17.

This is the error message I see:

Code: Select all

Checking nginx config
Stopping Services...
Done
Updating DV Binaries...
fatal: unable to access 'https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin.git/': gnutls_handshake() failed: Error in the push function.
Any ideas about how this frustrating issue could be addressed? I'm guessing there is some issue with GitHub access. Could the Update process include a step that would automatically fail that step if it takes more than a certain amount of time to complete?

UPDATE: See this post for possible solution: viewtopic.php?f=3&t=1794&start=20#p8126
Last edited by KE0FHS on Thu May 30, 2019 4:33 pm, edited 6 times in total.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

User avatar
w7efs
Posts: 245
Joined: Sun Apr 22, 2018 4:26 pm

Re: Pi-Star Update hangs on Updating DV Binaries

Post by w7efs » Mon May 20, 2019 8:19 pm

Is the Pi-Star behind a web proxy? Are there any additional error indications in /var/log/unattended-upgrades/unattended-upgrades.log ? This may be a large file depending on when the last reboot occurred, so look towards the most recent error condition.

Since the problem appears to be with the binaries, I looked in the "pistar-update" script and found this:

Code: Select all

...
        echo "Updating DV Binaries..."
        git --work-tree=/usr/local/bin --git-dir=/usr/local/bin/.git pull origin master
        echo "Done"
...
so I checked the git configuration for that --git-dir:

Code: Select all

$ cat /usr/local/bin/.git/config
[core]                                                                                                   
        repositoryformatversion = 0                                                                      
        filemode = true                                                                                  
        bare = false                                                                                     
        logallrefupdates = true                                                                          
[remote "origin"]                                                                                        
        url = https://github.com/AndyTaylorTweet/Pi-Star_Binaries.git                                    
        fetch = +refs/heads/*:refs/remotes/origin/*                                                      
[branch "master"]                                                                                        
        remote = origin                                                                                  
        merge = refs/heads/master                                                                        
The url indicates the origin access is "https", so it might be an SSL/TLS authentication issue. The way to test this hypothesis is to, as root, increase verbosity on the manual git command and see if it is indeed so, e.g.:

Code: Select all

# GIT_CURL_VERBOSE=1 git --work-tree=/usr/local/bin --git-dir=/usr/local/bin/.git pull origin master
* Couldn't find host github.com in the .netrc file; using defaults                                       
* Hostname was NOT found in DNS cache                                                                    
*   Trying 192.30.255.113...                                                                             
* Connected to github.com (192.30.255.113) port 443 (#0)                                                 
* found 152 certificates in /etc/ssl/certs/ca-certificates.crt                                           
*        server certificate verification OK                                                              
*        common name: github.com (matched)                                                               
*        server certificate expiration date OK                                                           
*        server certificate activation date OK                                                           
*        certificate public key: RSA                                                                     
*        certificate version: #3                                                                         
*        subject: businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=US,jurisdi
ctionOfIncorporationStateOrProvinceName=Delaware,serialNumber=5157550,C=US,ST=California,L=San Francisco,
O=GitHub\, Inc.,CN=github.com                                                                            
*        start date: Tue, 08 May 2018 00:00:00 GMT                                                       
                                                                                                         
*        expire date: Wed, 03 Jun 2020 12:00:00 GMT                                                      
                                                                                                         
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA  
*        compression: NULL                                                                               
*        cipher: AES-128-GCM                                                                             
*        MAC: AEAD                                                                                       
> GET /AndyTaylorTweet/Pi-Star_Binaries.git/info/refs?service=git-upload-pack HTTP/1.1                   
User-Agent: git/2.1.4                                                                                    
Host: github.com                                                                                         
Accept: */*                                                                                              
Accept-Encoding: gzip                                                                                    
Pragma: no-cache                                                                                         
                                                                                                         
< HTTP/1.1 200 OK                                                                                        
* Server GitHub Babel 2.0 is not blacklisted                                                             
< Server: GitHub Babel 2.0                                                                               
< Content-Type: application/x-git-upload-pack-advertisement                                              
< Transfer-Encoding: chunked                                                                             
< Expires: Fri, 01 Jan 1980 00:00:00 GMT                                                                 
< Pragma: no-cache                                                                                       
< Cache-Control: no-cache, max-age=0, must-revalidate                                                    
< Vary: Accept-Encoding                                                                                  
< X-GitHub-Request-Id: CCDC:97DB:1FF17F0:2C9BD33:5CE30851                                                
< X-Frame-Options: DENY                                                                                  
<                                                                                                        
* Connection #0 to host github.com left intact                                                           
From https://github.com/AndyTaylorTweet/Pi-Star_Binaries                                                 
 * branch            master     -> FETCH_HEAD                                                            
Already up-to-date.                                                                                      

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Mon May 20, 2019 8:38 pm

w7efs wrote:
Mon May 20, 2019 8:19 pm
Is the Pi-Star behind a web proxy? Are there any additional error indications in /var/log/unattended-upgrades/unattended-upgrades.log ? This may be a large file depending on when the last reboot occurred, so look towards the most recent error condition.
Thanks for your reply, Tony! As far as I know, Pi-Star is not behind a web proxy. I do use a VPN most of the time (Mullvad), but this happens even if I am disconnected from the VPN.

I also use the 1.1.1.1 DNS Resolver from Cloudflare (https://1.1.1.1/). Do you think that could be the root cause of the issue?

I only perform manual updates via SSH (I don't leave my hotspots running overnight). This issue happens whether I use an SSH app or run the update from within Pi-Star's Dashboard.

Excuse my simpleton's question, but since I'm only running manual updates and this issue happens only with updates and not upgrades, does it still make sense to look in the unattended-upgrades.log? If yes, I'll take a look at that later today when I can sit down at the computer I use for updates. It shouldn't be too large since I shut off my hotspots when I'm not using them.

Thanks again!
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

KE7FNS
Posts: 143
Joined: Wed Apr 17, 2019 11:11 pm

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE7FNS » Mon May 20, 2019 10:53 pm

KE0FHS wrote:
Mon May 20, 2019 1:02 pm

This morning's update was with a ZUMspot running on a RPi 3A+ with Pi-Star 4.0.0-RC4, 20190519, but I experience this with all my various hotspots, including those still running 3.4.17.
Just quickly glancing at things, Pi-Star 4.0.0-RC4 runs a slightly different pistar-update script than the one W7EFS listed.
It points to a completely different repo. So that kinda removes a problem with a repo.

Code: Select all

[core]
        repositoryformatversion = 0
        filemode = true
        bare = false
        logallrefupdates = true
[remote "origin"]
        url = https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin.git
        fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
        remote = origin
        merge = refs/heads/master

So its really interesting that both of them fail at that same point in your systems. Do you by chance have IPV6 enabled?
KE0FHS wrote:
Mon May 20, 2019 8:38 pm
It shouldn't be too large since I shut off my hotspots when I'm not using them.
It won't be large at all, because it gets deleted on reboot/poweroff.

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Tue May 21, 2019 1:48 am

Just ran it again on the same hotspot as above. This time it took a half hour. Here's the full view of the update:

Code: Select all

[email protected](ro):~$ sudo pistar-update
Updating OS...

Get:1 http://archive.raspberrypi.org/debian stretch InRelease [25.4 kB]
Hit:2 http://raspbian.raspberrypi.org/raspbian stretch InRelease               
Get:3 http://cdn-fastly.deb.debian.org/debian stretch-backports InRelease [91.8 kB]
Fetched 117 kB in 1s (66.8 kB/s)                              
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Done
Checking nginx config
Stopping Services...
Done
Updating DV Binaries...
fatal: unable to access 'https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin.git/': gnutls_handshake() failed: Error in the push function.
Updating /usr/local/bin...
From https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin
 * branch            master     -> FETCH_HEAD
Already up-to-date.
Done
Updating Pi-Star Binaries...
No updates for /usr/local/sbin available
Done
Updating Hostfiles...
Done
Updating Dashboard...
Updating /var/www/dashboard...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 4 (delta 3), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (4/4), done.
From https://github.com/AndyTaylorTweet/Pi-Star_DV_Dash
 * branch            master     -> FETCH_HEAD
   8c25afc..8194cd2  master     -> origin/master
Updating 8c25afc..8194cd2
Fast-forward
 admin/configure.php | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)
Done
Updating PiStar-Firewall...
Done
Starting Services...
Done
Updates complete, sleeping for a few seconds before making the disk Read-Only
Finished
[email protected](ro):~$ 
There is no /var/log/unattended-upgrades/unattended-upgrades.log

I have a net now, but when I'm finished, I'll run this on one of 3.4.17 hotspots to see what happens.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Tue May 21, 2019 3:02 am

Well, the update I just ran on one of 3.4.17 hotspots ran fine, so either I'm remembering incorrectly that it previously got hung up on a 3.4.17 hotspot, or this was one of the lucky times when everything runs fine. I'll keep testing tomorrow. Thanks again for the ideas.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

User avatar
w7efs
Posts: 245
Joined: Sun Apr 22, 2018 4:26 pm

Re: Pi-Star Update hangs on Updating DV Binaries

Post by w7efs » Tue May 21, 2019 5:11 am

For some reason you're resisting the verbosity invocation of git as suggested..

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Tue May 21, 2019 12:58 pm

w7efs wrote:
Tue May 21, 2019 5:11 am
For some reason you're resisting the verbosity invocation of git as suggested..
Because I didn't understand that you were suggesting that I do that. I don't read code much at all, except for the very, very basics. Now that I've reread your post, I'm guessing I should run the following, though I'm not sure where (root directory?):

Code: Select all

GIT_CURL_VERBOSE=1 git --work-tree=/usr/local/bin --git-dir=/usr/local/bin/.git pull origin master
I'll try that this morning.
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Tue May 21, 2019 1:19 pm

Code: Select all

[email protected](ro):~$ rpi-rw
[email protected](rw):~$ GIT_CURL_VERBOSE=1 git --work-tree=/usr/local/bin --git-dir=/usr/local/bin/.git pull origin master
error: cannot open .git/FETCH_HEAD: Permission denied
[email protected](rw):~$ sudo su
[email protected](rw):pi-star# GIT_CURL_VERBOSE=1 git --work-tree=/usr/local/bin --git-dir=/usr/local/bin/.git pull origin master
* Couldn't find host github.com in the .netrc file; using defaults
*   Trying 192.30.253.112...
* TCP_NODELAY set
* Connected to github.com (192.30.253.112) port 443 (#0)
* found 151 certificates in /etc/ssl/certs/ca-certificates.crt
* found 604 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: github.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: businessCategory=Private Organization,jurisdictionOfIncorporationCountryName=US,jurisdictionOfIncorporationStateOrProvinceName=Delaware,serialNumber=5157550,C=US,ST=California,L=San Francisco,O=GitHub\, Inc.,CN=github.com
*        start date: Tue, 08 May 2018 00:00:00 GMT
*        expire date: Wed, 03 Jun 2020 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,OU=www.digicert.com,CN=DigiCert SHA2 Extended Validation Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /AndyTaylorTweet/Pi-Star_v4_Binaries_Bin.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.11.0
Accept: */*
Accept-Encoding: gzip
Pragma: no-cache

< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: AD7D:38A5:B353D3:13F96D8:5CE3FAAB
< X-Frame-Options: DENY
< 
* Curl_http_done: called premature == 0
* Connection #0 to host github.com left intact
From https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin
 * branch            master     -> FETCH_HEAD
Already up-to-date.
I should add that I ran an update just before this and it took 15 minutes to complete because it again got hung up on the DV Binaries step, but I also got additional errors:

Code: Select all

Updating DV Binaries...
fatal: unable to access 'https://github.com/AndyTaylorTweet/Pi-Star_v4_Binaries_Bin.git/': gnutls_handshake() failed: Error in the push function.
Updating /usr/local/bin...
error: cannot open .git/FETCH_HEAD: Read-only file system
Done
Updating Pi-Star Binaries...
fatal: Unable to create '/usr/local/sbin/.git/index.lock': Read-only file system
No updates for /usr/local/sbin available
Done
Updating Hostfiles...
Done
Updating Dashboard...
No updates for /var/www/dashboard available
Done
Updating PiStar-Firewall...
Done
sed: couldn't open temporary file /etc/sedfvYctX: Read-only file system
sed: couldn't open temporary file /etc/sedeCeGA1: Read-only file system
Starting Services...
Done
Updates complete, sleeping for a few seconds before making the disk Read-Only
Finished
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

User avatar
KE0FHS
Posts: 846
Joined: Wed Apr 11, 2018 8:40 pm
Location: Colorado, USA
Contact:

Re: Pi-Star Update hangs on Updating DV Binaries

Post by KE0FHS » Tue May 21, 2019 1:24 pm

Ah, after reading the last set of error message, I manually put Pi-Star into read-write mode before running the update, and then the update ran just fine. So perhaps the sudo pistar-update script is, for some reason, failing to put Pi-Star into read-write mode?
73, Toshen, KE0FHS
Playing with Pi-Star (unofficial notes about setting up and using Pi-Star):
https://amateurradionotes.com/pi-star.htm

Post Reply