July 19, 20241 yr https://abc7.com/post/network-issues-linked-microsoft-systems-causing-outages-businesses/15070489/ Fuck Microsoft, this seriously fucks up things at work. We were supposed to have 500 people flying out today. Network issues linked to Microsoft systems causing outages at businesses worldwide LOS ANGELES (KABC) -- A software outage linked to Microsoft systems is impacting businesses worldwide Thursday night. The issue was impacting media organizations, including ABC, as well as banks, airlines and other industries, according to reports by media outlets and technology experts. Earlier in the day Frontier Airlines was among carriers experiencing problems, but it wasn't clear if it was the same issue that impacted a wider swath of the business world Thursday night. The site Downdetector was reporting a spike in outages involving companies such as Amazon Web Services, Microsoft and even Ancestry.com. American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning. The alert did not specify that it was tied to the Microsoft issues. The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues. Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off. At airports like Los Angeles International, social media users were reporting growing frustration among passengers stuck in the terminals. Edited July 19, 20241 yr by RPM
July 19, 20241 yr The headline blames Microsoft but the article says it's impacting AWS and the GCP as well. A Microsoft problem isn't going to impact those other clouds. The last line in the that article is now: Quote Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.
July 19, 20241 yr Author Just beat me to it. A little digging and it looks like it was a bad Crowdstrike update. https://techissuestoday.com/windows-bsod-crowdstrike-update/ Edited July 19, 20241 yr by RPM
July 19, 20241 yr A lot of people are going to have a bad day on Friday. This seems to be a bad Crowdsource update that crashes windows boxes (including VMs). The workaround being reported (as in the techissuestoday.com link above) is to boot into safe mode or WinRE and delete a file. Eazy enough for a techie, but an airline ticket counter worker or supermarket manager isn't going to be doing that. I'm actually surprised, my team has windows boxes are part of our deployments to AWS, Azure, and an internal cloud, but I just checked everything is up and functioning....and my company has some type of relationship with Crowdstrike....The bad update is probably still working it's way around the world and will get to my stuff tomorrow morning. 😞
July 19, 20241 yr I just checked, my work laptop took some updates tonight and is in a reboot loop, due to page_fault_in_nonpaged_area from csagent.sys, aka Crowdstrike stupid motherfuckers
July 19, 20241 yr Hi I’m currently stuck in Charlotte at the airport. Storms caused us to be delayed getting here as our layover, but that fucked us because then our connection to Austin was canceled at midnight this morning. Now waiting for some standbys and also booked a flight in SWA. Seeing which one gets us out of here first. Fuuuuuck Edited July 19, 20241 yr by SubliminalHorn .
July 19, 20241 yr 1 hour ago, 0xdeadbeef said: The headline blames Microsoft but the article says it's impacting AWS and the GCP as well. A Microsoft problem isn't going to impact those other clouds. The last line in the that article is now: you can still blame microsoft for the stupid shit way they handle dynamic linking, which is the root of this problem. On any sane OS, this would not bring the entire system to its knees.
July 19, 20241 yr I'm not a tech guy, but this seems like an undo amount of things being fucked up due to one cybersecurity system's "error." Is it possible that there's something more nefarious going on?
July 19, 20241 yr 11 minutes ago, TexArcher said: Is it possible that there's something more nefarious going on? crowdstrike was secret Chinese plan to take down the whole cloud
July 19, 20241 yr Popular Post Does this mean no Teams meetings for today? Really need to concentrate on the British Open.
July 19, 20241 yr 22 minutes ago, RPM said: FOX4 DFW reporting AA resumed service at 4AM. The first AA flight I’ve seen boarding is doing it now to LA. Looks like things are moving. We’re on standby for the 10:00 flight to Austin 🤞🏽
July 19, 20241 yr Popular Post 30 minutes ago, TexArcher said: I'm not a tech guy, but this seems like an undo amount of things being fucked up due to one cybersecurity system's "error." Is it possible that there's something more nefarious going on? It's not that this is nefarious, it's just how it fucked up that makes it such a pain. The root of the problem is a dll which is loaded at windows boot is causing a memory error that crashes the entire system. It's not hard to fix, just remove the dll and windows will boot like normal. Nothing is destroyed, it's just in a bootlock. Unfortunately, the way windows works, is that you have to use windows to remove the dll or stop it from loading, which you normally can't do because the dll is preventing windows from loading in the first place. The correct solution is to use a special mode called Safe Mode, which prevents any non-critical components (basically any dll) from being loaded at windows boot, specifically so you can fix this stuff. Problem is that getting to safe mode and knowing which dll to delete is above the capability of most people, and on sufficiently important systems, access to safe mode would be restricted to administrators anyways. It's a super easy fix, but it's a pain in the ass to do specifically because of how windows works, so people are going to have to lug their asses physically in front of these computers to make the change. And that takes time. Once it's fixed, it's like nothing ever happened. Nothing is permanently broken. It's just a stupid update that pushed a small piece of code that has memory errors, normally a routine thing but in this specific circumstance a huge problem. It's not nefarious, it's just shitty administrating by lots of these companies. Ideally, you wouldn't just push updates to your critical systems the moment they roll out. You'd sandbox them, test them in a controlled environment, then deploy the update to your system, for this very reason. This isn't really a huge, evil attack. It's more like shitty IT and code deployment in the worst spot. Don't attribute malice where incompetence will do. Edited July 19, 20241 yr by WelfareBuysMyWeed
July 19, 20241 yr 4 minutes ago, TexArcher said: Okay, so I work for a major financial firm. Do I get today off? Depends on if they use crowdstrike and windows. Those aren't the only two options. There are other companies for security besides crowdstrike, and much of the business world runs on linux. But it's also not just your company, it's who they interact with as well. There are also rollback softwares which can undo this immediately. Those can probably be deployed very quickly. There might be regulatory reasons why your company might require the latest security updates. Tl;dr: too many variables to tell. Edited July 19, 20241 yr by WelfareBuysMyWeed
July 19, 20241 yr I got to work hoping everything is fucked and I could just watch TV all day. NOTHING IS FUCKED DUDE!!! Dammit.
July 19, 20241 yr Just got the email from IT about it. Maybe I can leave super early and go play NCAA all day.
July 19, 20241 yr Lol windows is such a fkn pile, it's wild how easily one bad push update can cripple their OS globally. I've seen so many outages caused on Windows fleets just from one bad update that forces you to ol yeller the whole install and restart from scratch Edited July 19, 20241 yr by Captainant
July 19, 20241 yr Using your personal machine to login to your company’s network is one work around that will work for many people. Works for my law firm.
July 19, 20241 yr 41 minutes ago, TexArcher said: Okay, so I work for a major financial firm. Do I get today off? yes. approved.
July 19, 20241 yr It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.
July 19, 20241 yr 17 minutes ago, Buzzrock said: Cue the smug Mac users /smugmacuser 7 minutes ago, Shaggy3.0 said: /linux /both
July 19, 20241 yr 7 minutes ago, Buzzrock said: It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates. Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck
July 19, 20241 yr 2 minutes ago, Captainant said: Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck Is that all it was? A regular definitions update? Jeez. We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day.
July 19, 20241 yr This thing carried over to systems integrated with a lot of EMRs so hospital systems were affected as well, but seem to be mostly functional this am.
July 19, 20241 yr Just now, Buzzrock said: Is that all it was? A regular definitions update? Jeez. We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. Literally a bad DLL file that causes a page fault BSOD on boot
July 19, 20241 yr Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC.
July 19, 20241 yr I guess Texas DPS uses this system because we had a driver's test appointment for noon today for my oldest and that got cancelled at 7am this morning. System won't let me log back in to reschedule.
July 19, 20241 yr My personal Dell machine is not affected. I’m able to login to my firm’s network without any issues.
July 19, 20241 yr 1 minute ago, Captainant said: Literally a bad DLL file that causes a page fault BSOD on boot It appears to be 3rd party kernel-mode driver, not a user level DLL.
July 19, 20241 yr 8 minutes ago, Buzzrock said: Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC. It's usually deployed as part of a managed device, I know it's on my work laptop and virtual desktop as part of the whole minimum required security posture. I get the option to postpone windows and user software updates, but endpoint protection gets pushed through 2 minutes ago, 0xdeadbeef said: It appears to be 3rd party kernel-mode driver, not a user level DLL. Username appropriate lulz
July 19, 20241 yr Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one.
July 19, 20241 yr Popular Post 40 minutes ago, HouTex said: Using your personal machine to login to your company’s network is one work around that will work for many people. Works for my law firm. are you the guy who kept opening the test phishing links at work>? 34 minutes ago, Buzzrock said: It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.
July 19, 20241 yr 3 minutes ago, Buzzrock said: Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. There are for Linux, because it's a OS that's actually built for enterprise usage lol. Windows is basically a full reinstall to unfuck it, and from what I've been reading in /r/crowdstrike most admins are just completely locked out from remote windows systems due to the boot loop.
July 19, 20241 yr 9 minutes ago, Celery Man said: are you the guy who kept opening the test phishing links at work>? the ultimate chaos monkey strategy
July 19, 20241 yr Agile for the win? Happened to me last night around 1130PM CT. Given it was not yet widespread and workarounds prepared, I followed the typical approach for this: reinstall w/ saving personal files. Well, sily me... I neglected to realize that a reinstallation of Windows would NOT reinstall the corporate image. FML. That said, even for technically savvy people, the process of fixing a corporate-image-installed will be a bit painful: request to boot into safe mode, enter the bitlocker key - that you will have to get off of your phone authenticated to your business (using mobile Edge), boot, remove a file. BTW, writing this from my Mac.
July 19, 20241 yr 24 minutes ago, 0xdeadbeef said: It appears to be 3rd party kernel-mode driver. So basically, any random driver in Houston.
July 19, 20241 yr I went to bed around 10pm with a team of juniors cranking away on a thing a client asked for by noon today. At 3am I start getting panicked phone calls from the team that all computers have crashed. Then the IT backup call alerts start. Still down, fun morning!
July 19, 20241 yr Fucking pager-dutied in the middle of the night for this crap. Bunch of servers and laptops down. Luckily I have reinforcements in EMEA, so not as bad as it could be.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.