There are many detailed write-ups of the recent CrowdStrike incident, so I will keep this one brief- On the evening of 7/25/24, CrowdStrike pushed a content update which caused Windows systems to crash and/or enter a reboot loop, rendering the systems unusable.
Unfortunately, the only way to remediate the issue required physical access to the affected machines.
This happened in the middle of the night in the United States, so many users had their laptops/workstations turned off and did not receive the bad content update. Many did however, and Windows-based server infrastructure, often business-critical servers like Domain Controllers, are typically left on 24x7, and unfortunately received the update as well, crashing the systems.
100% of RADICL’s customer base had at least 1 affected host.
Upon learning about the CrowdStrike incident, the RADICL vSOC immediately jumped into action to ensure our customers were able to quickly identify and remediate affected systems- and get back to business as usual:
“RADICL has already paid for itself at BCNS! After the recent CrowdStrike incident, RADICL's vSOC team swiftly notified us of the issue and provided clear, concise guidance on how to fix it," "Our IT team was able to remediate all affected machines promptly and returned everyone to their regularly scheduled duties. We're very pleased we decided to partner with RADICL."
--Thomas Cathey, BCNS Principal
Below is a timeline of events:
6:10 AM MDT: RADICL vSOC becomes aware of the incident
7:17 AM MDT: RADICL vSOC pushes out a Critical Task to all affected customers, detailing the incident and affected hosts, and providing step-by-step remediation procedures:
8:40 AM MDT: ~65% of affected hosts remediated, 100% of business-critical hosts remediated
11:06 AM MDT: ~85% of affected hosts remediated
1:35 PM MDT: ~90% of affected hosts remediated
COB 7/19/24: >95% of affected hosts remediated
The swift response from the RADICL vSOC helped our customers immediately understand what was happening, provided remediation guidance and assistance, and prevented our customers from experiencing any measurable impact on business operations.
While the incident with CrowdStrike was extremely unfortunate, and we wait to hear more about root cause, RADICL still believes CrowdStrike is the premier EDR solution in the world, as described in one of our previous blogs.
This unfortunate incident could have happened to any vendor that has a sensor/agent deployed to endpoints, and we believe CrowdStrike will now be implementing strong controls to ensure nothing like this ever happens again. The case can be made that because of this incident, they are now the LEAST likely EDR vendor to release/push breaking updates in the future.
RADICL will continue to make customer success our top priority. Our engineering team is starting to look into ways to detect this type of 3rd–party outage faster, and our vSOC is working to improve the already-fast response times for all types of incidents.