Jump to content
View in the app

A better way to browse. Learn more.

Surly Horns

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Featured Replies

https://abc7.com/post/network-issues-linked-microsoft-systems-causing-outages-businesses/15070489/

Fuck Microsoft, this seriously fucks up things at work. We were supposed to have 500 people flying out today. 

Network issues linked to Microsoft systems causing outages at businesses worldwide

 

LOS ANGELES (KABC) -- A software outage linked to Microsoft systems is impacting businesses worldwide Thursday night.

 

The issue was impacting media organizations, including ABC, as well as banks, airlines and other industries, according to reports by media outlets and technology experts.

Earlier in the day Frontier Airlines was among carriers experiencing problems, but it wasn't clear if it was the same issue that impacted a wider swath of the business world Thursday night.

 

The site Downdetector was reporting a spike in outages involving companies such as Amazon Web Services, Microsoft and even Ancestry.com.

 

American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning.

 

The alert did not specify that it was tied to the Microsoft issues.

 

The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.

 

Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.

 

At airports like Los Angeles International, social media users were reporting growing frustration among passengers stuck in the terminals.

Edited by RPM

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

 

The last line in the that article is now:

Quote

Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.

 

A lot of people are going to have a bad day on Friday. 

This seems to be a bad Crowdsource update that crashes windows boxes (including VMs).  The workaround being reported (as in the techissuestoday.com link above) is to boot into safe mode or WinRE and delete a file.   Eazy enough for a techie, but an airline ticket counter worker or supermarket manager isn't going to be doing that.  

I'm actually surprised, my team has windows boxes are part of our deployments to AWS, Azure, and an internal cloud, but I just checked everything is up and functioning....and my company has some type of relationship with Crowdstrike....The bad update is probably still working it's way around the world and will get to my stuff tomorrow morning. 😞

  

  • RPM changed the title to Crowdstrike Outage a/k/a Everything's Fucked

I just checked, my work laptop took some updates tonight and is in a reboot loop, due to page_fault_in_nonpaged_area from csagent.sys, aka Crowdstrike

stupid motherfuckers

Hi I’m currently stuck in Charlotte at the airport. Storms caused us to be delayed getting here as our layover, but that fucked us because then our connection to Austin was canceled at midnight this morning. Now waiting for some standbys and also booked a flight in SWA. Seeing which one gets us out of here first. Fuuuuuck

Edited by SubliminalHorn
.

1 hour ago, 0xdeadbeef said:

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

The last line in the that article is now:

 

you can still blame microsoft for the stupid shit way they handle dynamic linking, which is the root of this problem. On any sane OS, this would not bring the entire system to its knees.

I'm not a tech guy, but this seems like an undo amount of things being fucked up due to one cybersecurity system's "error."

Is it possible that there's something more nefarious going on?

  • Author

FOX4  DFW  reporting AA resumed service at 4AM.

11 minutes ago, TexArcher said:

 

Is it possible that there's something more nefarious going on?

crowdstrike was secret Chinese plan to take down the whole cloud

  • Popular Post

Does this mean no Teams meetings for today? Really need to concentrate on the British Open. 

yup, gonna say fuck it and just goof off all day .   buuuurp

drink more.  enjoy life.

22 minutes ago, RPM said:

FOX4  DFW  reporting AA resumed service at 4AM.

The first AA flight I’ve seen boarding is doing it now to LA. Looks like things are moving. We’re on standby for the 10:00 flight to Austin 🤞🏽 

  • Popular Post
30 minutes ago, TexArcher said:

I'm not a tech guy, but this seems like an undo amount of things being fucked up due to one cybersecurity system's "error."

Is it possible that there's something more nefarious going on?

It's not that this is nefarious, it's just how it fucked up that makes it such a pain. The root of the problem is a dll which is loaded at windows boot is causing a memory error that crashes the entire system. It's not hard to fix, just remove the dll and windows will boot like normal. Nothing is destroyed, it's just in a bootlock. Unfortunately, the way windows works, is that you have to use windows to remove the dll or stop it from loading, which you normally can't do because the dll is preventing windows from loading in the first place. The correct solution is to use a special mode called Safe Mode, which prevents any non-critical components (basically any dll) from being loaded at windows boot, specifically so you can fix this stuff. Problem is that getting to safe mode and knowing which dll to delete is above the capability of most people, and on sufficiently important systems, access to safe mode would be restricted to administrators anyways. It's a super easy fix, but it's a pain in the ass to do specifically because of how windows works, so people are going to have to lug their asses physically in front of these computers to make the change. And that takes time.

Once it's fixed, it's like nothing ever happened. Nothing is permanently broken. It's just a stupid update that pushed a small piece of code that has memory errors, normally a routine thing but in this specific circumstance a huge problem. It's not nefarious, it's just shitty administrating by lots of these companies. Ideally, you wouldn't just push updates to your critical systems the moment they roll out. You'd sandbox them, test them in a controlled environment, then deploy the update to your system, for this very reason.

This isn't really a huge, evil attack. It's more like shitty IT and code deployment in the worst spot. Don't attribute malice where incompetence will do.

Edited by WelfareBuysMyWeed

Okay, so I work for a major financial firm.  Do I get today off?

4 minutes ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

Depends on if they use crowdstrike and windows. Those aren't the only two options. There are other companies for security besides crowdstrike, and much of the business world runs on linux. But it's also not just your company, it's who they interact with as well. There are also rollback softwares which can undo this immediately. Those can probably be deployed very quickly. There might be regulatory reasons why your company might require the latest security updates.

Tl;dr: too many variables to tell.

Edited by WelfareBuysMyWeed

I got to work hoping everything is fucked and I could just watch TV all day. NOTHING IS FUCKED DUDE!!!

Dammit.

Just got the email from IT about it. Maybe I can leave super early and go play NCAA all day.

Lol windows is such a fkn pile, it's wild how easily one bad push update can cripple their OS globally. 

I've seen so many outages caused on Windows fleets just from one bad update that forces you to ol yeller the whole install and restart from scratch

Edited by Captainant

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

41 minutes ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

yes.

approved.

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

9 minutes ago, Buzzrock said:

Cue the smug Mac users

/smugmacuser

/linux

17 minutes ago, Buzzrock said:

Cue the smug Mac users

/smugmacuser

 

7 minutes ago, Shaggy3.0 said:

/linux


 

/both

7 minutes ago, Buzzrock said:

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

2 minutes ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

This thing carried over to systems integrated with a lot of EMRs so hospital systems were affected as well, but seem to be mostly functional this am.

Just now, Buzzrock said:

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

Literally a bad DLL file that causes a page fault BSOD on boot

Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC. 

I guess Texas DPS uses this system because we had a driver's test appointment for noon today for my oldest and that got cancelled at 7am this morning. System won't let me log back in to reschedule. 

My personal Dell machine is not affected.  I’m able to login to my firm’s network without any issues.  

1 minute ago, Captainant said:

Literally a bad DLL file that causes a page fault BSOD on boot

It appears to be 3rd party kernel-mode driver, not a user level DLL.  

 

8 minutes ago, Buzzrock said:

Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC. 

It's usually deployed as part of a managed device, I know it's on my work laptop and virtual desktop as part of the whole minimum required security posture. I get the option to postpone windows and user software updates, but endpoint protection gets pushed through

 

2 minutes ago, 0xdeadbeef said:

It appears to be 3rd party kernel-mode driver, not a user level DLL.  

Username appropriate lulz

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

  • Popular Post
40 minutes ago, HouTex said:

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

are you the guy who kept opening the test phishing links at work>?

34 minutes ago, Buzzrock said:

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

image.png.c75ff6749e5281563923ea8f6744c0e4.png

3 minutes ago, Buzzrock said:

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

There are for Linux, because it's a OS that's actually built for enterprise usage lol. Windows is basically a full reinstall to unfuck it, and from what I've been reading in /r/crowdstrike most admins are just completely locked out from remote windows systems due to the boot loop. 

 

9 minutes ago, Celery Man said:

are you the guy who kept opening the test phishing links at work>?

image.png.c75ff6749e5281563923ea8f6744c0e4.png

the ultimate chaos monkey strategy

Agile for the win?

Happened to me last night around 1130PM CT.  Given it was not yet widespread and workarounds prepared, I followed the typical approach for this: reinstall w/ saving personal files.  Well, sily me...  I neglected to realize that a reinstallation of Windows would NOT reinstall the corporate image.  FML.

That said, even for technically savvy people, the process of fixing a corporate-image-installed will be a bit painful: request to boot into safe mode, enter the bitlocker key - that you will have to get off of your phone authenticated to your business (using mobile Edge), boot, remove a file.

BTW, writing this from my Mac.

24 minutes ago, 0xdeadbeef said:

It appears to be 3rd party kernel-mode driver.  

 

So basically, any random driver in Houston.

I went to bed around 10pm with a team of juniors cranking away on a thing a client asked for by noon today. At 3am I start getting panicked phone calls from the team that all computers have crashed. Then the IT backup call alerts start. Still down, fun morning!

Fucking pager-dutied in the middle of the night for this crap.  Bunch of servers and laptops down.  Luckily I have reinforcements in EMEA, so not as bad as it could be.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.