UPDATE: 10/03/19
Google has posted an update that they found the issue in the Keystone Update and fixed it. They are are pushing the updated version now. The push started last night and the new version should be 1.2.13.79 or 1.2.13.80.
Google has been able to reproduce the bug. Chromium bug – bugs.chromium.org/p/chromium/issues/detail?id=1007358
Google now is recommending that you reinstall macOS to fix the issue. support.google.com/chrome/thread/15235262
Statement From AVID 9/25- avid.com/92419issue
See Index #7 for the Updated Fix from Google Support
#Varsectomy FAQ- Index
- 1. What is going on here? – #Varsectomy
- 2. Which macOS Versions are affected ?
- 3. Specific Mac Hardware ?
- 4. SIP – System Integrity Protection
- 5. AVID or Google Chrome Keystone Update causing this issue ?
- 6. How can I check my /var symlink?
- 7. How can I fix the issue = 2 ways to fix.
- 8. What about Hackintosh users? A fix for you!
- 9. Should I disable Chrome Auto Updates
- 10. Jamf Pro Extension Atribute
- 11. Links
- 12. Credit – Investigation by #MacAdmins & Apple
1. What is going on here ? – #Varsectomy
The purpose of this article is to gather all known information and data around the current issues some AVID and NON SIP protected Macs are experiencing. The issue first started on Monday 9/23/19.
Late Yesterday some #MacAdmins started to report that a few of their systems would not boot properly. They would have the following issues.
- 1. After rebooting the affected system it would Kernel Panic. The system will reboot only to KP again
- 2. User Logs out and the system shows the Setup Assistant.
- 3. The System Kernel Panics into a boot Loop.
AVID users were some of the first to report the issue. AVID.com says
Sometimes AVID Media Creators use 3rd Party Graphics cards connected to their Mac Pro. When the issue hit yesterday, it was thought that AVID was the main cause of the problems since all the users experiencing the issue had AVID software.
Only later after a MacAdmins deep dive investigation was it found that AVID was NOT the cause of the problem. It was Google Chrome Keystone Updater!
Join the #Varsectomy channel in MacAdmins Chat for the latest info!
2. Which macOS Versions are affected ?
MacOS 10.9 – 10.14 Mojave
NOTE: macOS 10.9 & 10.10 so not have SIP. System Integrity Protection was introduced in macOS 10.11 El Capitan.
3. Specific Mac Hardware ?
At first it was first reported that the 2013 Mac Pro was affected, this is not true.
Now it seems the issue affects all Macs that have SIP (System Integrity Protection) Disabled or turned OFF
Check if SIP is enabled by running csrutil status
System Integrity Protection status: enabled.System Integrity Protection status: disabled
4. SIP – System Integrity Protection
All Reports so far look to be from Macs that have SIP Disabled!
Please do not disable SIP, it was created to protect macOS from this very issue.
5. AVID or Google Chrome Keystone Update causing this issue ?
At first the issue seemed to be caused by Avid Media Composer. After further investigation it was found that AVID was not the problem! Some AVID users may need to DISABLE SIP for 3rd party Video Card support.
After investigation from some of the top minds in the MacAmins Slack Chat #varsectomy channel it was found that the Google Keystone Updater was at the heart of the issue.
The Google Chrome Keystone updater lives here (if set to only update for the user who installed Chrome) – ~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/
If set to install for ALL users. (This also prompts for admin access for install) Thanks Jeff Johnson /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/
You can run the Google Chrome Keystone Updater Manually to kick off an update check. (NOTE: Google has disabled the updater for now!) see bugs.chromium.org/p/chromium/issues/detail?id=1007358
~/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle/Contents/Resources/GoogleSoftwareUpdateAgent.app/Contents/MacOS/GoogleSoftwareUpdateAgent -runmode oneshot
Thank you eholtam for the VM investigation and Screenshots!
After kicking off the update you can see the problem below.
Found and deleted symlink at path /var
6. How do I check my /var symlink?
Check to see if your /var symlink was modified by running the following command.
ls -ldO /var
You should get one of the following outputs. The first one below means that your /var volder is SIP protected (notice the restricted flag) and the proper sym link /var -> private/var
lrwxr-xr-x@ 1 root wheel restricted,hidden 11 Apr 1 2018 /var -> private/var
The next one means that your symlink is broken and the folder is NOT SIP Protected.
drwxr-xr-x 5 503 wheel - 170 Sep 24 14:37 /var
If you find /var in this condition you are affected! If you LOGOUT, SHUTDOWN OR RESTART your Mac will NOT Boot! You will need to boot into recovery, repair the /var symlink and reset the restricted flags. Fix below.
7. How can I fix the issue? 2 ways to fix.
UPDATE: 9/25 – Rich Trouton posted a great article showing you step by step how to fix both 10.9-10.10 & 10.11-10.14 systems – derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/
1st Fix from MacAdmins User Juest
First you have to fix the /var symlink so it shows /var -> private/var
Then you can remove the affected LaunchAgents
Boot to macOS Recovery by holding down Command R
Steps for 10.11 -10.14 SIP Supported Systems
1. Boot into Recovery
2. Launch Terminal
(only if you're on 10.11+ with SIP)
# chroot /Volumes/[affected install]
# mv var vv
# ln -s private/var var
# chflags -h restricted /var
# chflags -h hidden /var
# xattr -sw com.apple.rootless "" /var
Steps for 10.9-10.10 NON SIP Systems
10.10 or older:
only do symblink and the hidden chflag and
remove launch agents from
/Users/[affected user]/Library/LaunchAgents/
OR
/Library/LaunchAgents/
- com.google.keystone.agent.plist
- com.google.keystone.xpcservice.plist
# exit
(recommended) # csrutil enable (10.11+)
8. Reboot
UPDATE: 9/28 – 2nd Fix is from Google Support
Google is now recommending that you reinstall macOS if you had this issue.
https://support.google.com/chrome/thread/15235262
NOTE: The information below is from Google. Link Below.
chroot /Volumes/Macintosh\ HD #(Macintosh HD is the default) rm -rf /Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle mv var var_back #(Var may not exist, but this is fine) ln -sh private/var /var chflags -h restricted /var chflags -h hidden /var xattr -sw com.apple.rootless "" /var
NOTE: Commenter stockmind has noted that the GoogleSoftwareUpdate.Bundle could reside in the USERS folder. In that case you need to run this string instead. Change USERNAME to your Username. Thanks for the tip!
rm -rf /Users/USERNAME/Library/Google/GoogleSoftwareUpdate/GoogleSoftwareUpdate.bundle
Official Fix from Google support.google.com/chrome/thread/15235262
news.ycombinator.com/item?id=21066472 & 9to5google.com/2019/09/24/avid-mac-crashing-google/
8. What about Hackintosh users? A fix for you!
I had a few Hackintosh users reach out for a fix. Good News! u/Fargo_Newb on Reddit has a fix for you. – reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/
9. Should I disable Chrome Auto Updates ?
You shouldn’t need to disable Chrome Auto updates at this time. From the Chromium bug report – bugs.chromium.org/p/chromium/issues/detail?id=1007358 Google has disabled Keystone updates for now until the issue is found and resolved.
If you would like to disable Google Chrome Keystone automatic updates, the article below will show you. applehelpwriter.com/2014/07/13/how-to-remove-googles-secret-update-software-from-your-mac/
10. Jamf Pro Extension Attribute
This will help you find machines that are in a state where the /var sys link is broken. Thanks @neilmartin83 & @rmanly
#!/bin/bash
if [[ -h /var ]]; then
echo "<result>symlink</result>"
elif [[ -d /var ]]; then
echo "<result>directory</result>"
fi
exit 0
symlink = System OK!
11. Links
Below are a few of of the reports. It first started out as “AVID Hollywood Mac Pro boot problem“.
Chromium Bug Report 9/25 – https://bugs.chromium.org/p/chromium/issues/detail?id=1007358
Google Official Response 9/25 & 9/27 – support.google.com/chrome/thread/15235262
Official AVID Statement 9/25 – avid.com/92419issue
Rich Trouton step by step fix guide 9/25 –derflounder.wordpress.com/2019/09/25/google-keystone-update-breaks-macs-ability-to-boot-if-system-integrity-protection-is-disabled/
Scott Simmons 9/24 – provideocoalition.com/avid-editors-editors-in-general-be-on-alert-about-this-potential-rebooting-issue/
SlashDot Report 9/26 – https://it.slashdot.org/story/19/09/25/2152236/mysterious-mac-pro-shutdowns-likely-caused-by-chrome-update
Ars Technica Report 9/26 – arstechnica.com/information-technology/2019/09/no-it-wasnt-a-virus-it-was-chrome-that-stopped-macs-from-booting/
Daring FireBall by John Gruber 9/26 – daringfireball.net/linked/2019/09/25/mr-macintosh-chrome-updater
Forbes Report 9/26 – forbes.com/sites/kateoflahertyuk/2019/09/26/google-confirms-buggy-chrome-update-is-breaking-apple-macs/#5aad7747391c
Hacker News Discussion Thread 9/25- news.ycombinator.com/item?id=21064663
Gizmodo.com Report 9/25 – gizmodo.com/whoops-google-says-mysterious-wave-of-unbootable-macs-1838430057
Hackintosh Fix 9/25 – reddit.com/r/hackintosh/comments/d8tm8z/psa_google_chrome_updaterkeystone_rendering/
Updated Report from 9to5google.com 9/25 – 9to5google.com/2019/09/24/avid-mac-crashing-google/
Variety Updated Report 9/25 – variety.com/2019/digital/news/mac-pro-avid-shutdowns-chrome-update-keystone-1203348549/
Variety Report 9/24- variety.com/2019/digital/news/avid-mac-pro-corrupted-hollywood-1203347033/
MacRumors Report 9/24 – macrumors.com/2019/09/24/hollywood-mac-pros-hit-by-avid-issue/
Bleeping Computer Report 9/24 –bleepingcomputer.com/news/software/avid-users-are-suddenly-finding-that-their-macs-won-t-boot/
Some of the first reports on the AVID FaceBook Group 9/24 – https://www.facebook.com/photo.php?fbid=10156759822188022&set=p.10156759822188022&type=3&theater
Disable Google Chrome Keystone Automatic Updates – applehelpwriter.com/2014/07/13/how-to-remove-googles-secret-update-software-from-your-mac/
12. Credit! – Further investigation by #MacAdmins & Apple
It needs to be stated that many MacAdmins and Apple Engineers worked together and after investigating with users who had the issue. Great work guys!!!
- @rtrouton for the cool /var Logo!
- @bradtchapman for #Varsectomy
- @eholtam for investigation and screen shots.
- @Juest for investigation and fix commands
- All active users in MacAdmins Chat #varsectomy
