this post was submitted on 20 Jul 2024
473 points (97.8% liked)

Linux

5234 readers
166 users here now

A community for everything relating to the linux operating system

Also check out !linux_memes@programming.dev

Original icon base courtesy of lewing@isc.tamu.edu and The GIMP

founded 1 year ago
MODERATORS
 

A widespread Blue Screen of Death (BSOD) issue on Windows PCs disrupted operations across various sectors, notably impacting airlines, banks, and healthcare providers. The issue was caused by a problematic channel file delivered via an update from the popular cybersecurity service provider, CrowdStrike. CrowdStrike confirmed that this crash did not impact Mac or Linux PCs.

It turns out that similar problems have been occurring for months without much awareness, despite the fact that many may view this as an isolated incident. Users of Debian and Rocky Linux also experienced significant disruptions as a result of CrowdStrike updates, raising serious concerns about the company's software update and testing procedures. These occurrences highlight potential risks for customers who rely on their products daily.

top 50 comments
sorted by: hot top controversial new old
[–] sudo@programming.dev 196 points 4 months ago (1 children)

The analysis revealed that the Debian Linux configuration was not included in their test matrix.

You might as well say you don't support Linux.

"Crowdstrike's model seems to be 'we push software to your machines any time we want, whether or not it's urgent, without testing it'," lamented the team member.

I wonder how this shit works on NixOS.

[–] Flatfire@lemmy.ca 75 points 4 months ago (1 children)

If I'm remembering right, RHEL is Crowdstrike's primary Linux target. And NixOS wouldn't even be a factor since it's basically just not enterprise grade.

That said, they need a serious revision of their QA processes.

[–] circuscritic@lemmy.ca 37 points 4 months ago* (last edited 4 months ago) (10 children)

RHEL, Ubuntu, & Debian cover the vast majority of enterprise installs I imagine, and provide a solid testing base for developers in the Linux business software space.

Maybe you add Gentoo, some post-CentOS clones/forks, or other more niche industry/workload specific distros, but how you do skip Debian?

[–] lemmyreader@lemmy.ml 10 points 4 months ago (2 children)

RHEL, Ubuntu, & Debian cover the vast majority of enterprise installs I imagine, and provide a solid testing base for developers in the Linux business software space.

Enterprises I imagine are using RHEL, Ubuntu, SUSE's SLES and Oracle Linux and probably not Debian. But that's a guess. Where can statistics and numbers be found ?

[–] barkingspiders@infosec.pub 22 points 4 months ago (1 children)

Largish enterprise heavily using Debian, just 1 data point here but we do exist.

[–] lemmyreader@lemmy.ml 3 points 3 months ago
[–] pupbiru@aussie.zone 9 points 4 months ago

consultant for large enterprises in australia, and i literally can’t say i’ve ever seen anyone running anything other than RHEL and amazon linux (so… RHEL) in production… unless we’re talking not for profits, and then that’s been a bit of a mixed bag

[–] BCsven@lemmy.ca 8 points 3 months ago

In the enterprise realm it is typically SUSE and RHEL.

[–] themeatbridge@lemmy.world 3 points 4 months ago (5 children)

I'm not an expert in any sense.

But it was always my impression that Ubuntu and Debian were what you use on personal machines, while RHEL is the baseline standard for professional servers.

Is that not accurate? CrowdStrike's target customer seems to be the sort of company that would insist on using RHEL for the enterprise features.

[–] Skydancer@pawb.social 18 points 4 months ago* (last edited 4 months ago) (1 children)

That is not accurate.

  • RedHat is the standard for high-budget American corps.
  • Rocky and similar for low-budget American orgs
  • Ubuntu Server has a large following with developers who think they don't need sysadmins.
  • Debian Stable is more popular with European orgs that aren't incentivized by US government contracts to go with Redhat. It is much more stable than Ubuntu, has been more reliable in its support promises than Redhat, and doesn't suffer from the NIH syndrome that infects both.
  • Ubuntu is popular with home users
  • Debian Testing is good for workstations and personal machines that need to be a bit more current
  • Debian Unstable for people who like Debian but want to live on the bleeding edge
[–] BCsven@lemmy.ca 3 points 3 months ago

The enterprise systems I see are only certified on RHEL and SUSE, debian is not even a contender. Obviously Americans typically choose Rhel and europe goes for SUSE.

Debian doesn sell enterprise support.

[–] mundane@feddit.nu 14 points 4 months ago (3 children)

I've been using Linux professionally for 15 years. It's been Debian or Ubuntu almost everywhere I have been. Although that might be regional.

load more comments (3 replies)
[–] Scoopta@programming.dev 10 points 4 months ago

A lot of companies run Debian and Debian based distros, Google on their servers for a start

[–] valaramech@fedia.io 9 points 4 months ago* (last edited 4 months ago)

Canonical and Debian both target the professional server space. I've spent pretty much my entire career working on Debian-based distros.

Hell, the one company I worked for that I expected to use RHEL used Ubuntu for everything, so 🤷‍♂️.

[–] LeFantome@programming.dev 4 points 3 months ago

This is accurate.

There is another reply that says “this is not accurate” that includes true information to back you up.

For infrastructure, RHEL is the gold standard for large companies with a budget. The RHEL customer-base probably overlaps almost completely with CrowdStrike.

RHEL imitators are popular with people that value cost savings more than the corporate backing ( beyond individual cases, this DOES NOT describe the enterprise space ).

Ubuntu is very popular with developers in companies of all sizes. Outside of maybe being the base for containers, this is not how “infrastructure” choices are made though.

Debian is popular with Linux enthusiasts and, where they have influence, businesses may use that. In enterprise environments, it is less likely this group is the one making the decisions. Again though, individual cases exist.

load more comments (7 replies)
[–] Telorand@reddthat.com 101 points 4 months ago (2 children)

Users of Debian and Rocky Linux also experienced significant disruptions as a result of CrowdStrike updates, raising serious concerns about the company's software update and testing procedures. These occurrences highlight potential risks for customers who rely on their products daily.

Hot take: maybe bossware is a fucking drain on society, and people should stop buying it.

[–] zelifcam@lemmy.world 89 points 4 months ago* (last edited 4 months ago) (2 children)

Yeah, but our leadership had a really nice lunch with their sales rep! Licenses for everyone!

[–] Telorand@reddthat.com 43 points 4 months ago

It's sad how accurate this is.

[–] slazer2au@lemmy.world 7 points 4 months ago

After getting a referral from your cyber insurance rep right?

[–] dactylotheca@suppo.fi 17 points 4 months ago (2 children)

Well, if the executive leech class wants workers to have bossware, there's not all that much people can do about it. Can't just decide to not use it if your employer demands it

[–] sudo@programming.dev 20 points 4 months ago

Worse, my employer doesn't care about this shit but our clients are demanding we have the bossware installed.

[–] Telorand@reddthat.com 7 points 4 months ago* (last edited 4 months ago) (2 children)

I didn't mean the average worker. I meant the "executive leech class," because downtime of this scale means lost profits, which is something they care deeply about.

[–] dactylotheca@suppo.fi 14 points 4 months ago* (last edited 4 months ago)

which is something they care deeply about.

They care about quarterly profits. Preventing fuckups of this scale requires long-term effort which is not profitable by itself, it only prevents possible future fuckups, and this is why proper QC etc. aren't done. Short term profits over everything else.

[–] technom@programming.dev 7 points 4 months ago (3 children)

In that case, it's time for the average workers to sabotage the bossware. Let the leech class solve the problem they create.

load more comments (3 replies)
[–] SkyNTP@lemmy.ml 89 points 4 months ago (4 children)

The software is not the problem. Software breaks all the time. The problem is monocultures and centralization. Building entire industry ecosystems all around a single point of failure. This is the just-in-time manufacturing supply chain disruptions and fragility all over again.

Who knew, a diverse ecosystem was a strength, not a weakness.

[–] Ooops@feddit.org 46 points 4 months ago (1 children)

The software is the problem if it's produced with a corporate mentality of "ship first, fix later".

[–] SapphironZA@sh.itjust.works 11 points 4 months ago

Yep, at this point the "security" companies can do with imitating malware development practices.

[–] PriorityMotif@lemmy.world 7 points 3 months ago

Everyone got an MBA and failed to realize it was just corporate brainwashing.

[–] madcaesar@lemmy.world 7 points 3 months ago

Nature has been telling us all long, but we don't listen!

[–] possiblylinux127@lemmy.zip 75 points 4 months ago

"I don't test often but when I do I test on the entire planet"

[–] Gloria@sh.itjust.works 62 points 4 months ago

rootkit doing rootkit things

[–] Toes@ani.social 37 points 4 months ago (1 children)

There's a concept in this industry where you eat your own dog food.

Deploying these updates to your own people could have avoided this mess.

[–] themoonisacheese@sh.itjust.works 36 points 4 months ago (3 children)

Oh but they did. Turns out that this is specifically caused by one driver expecting another to be installed, the other one being for another of their products. If you have the other product installed, it doesn't crash, so it didn't crash on their machines because they have all their products installed and apparently not a single element of their test matrix has the single most common configuration they service

[–] Akuma@pawb.social 9 points 4 months ago (2 children)

Do you have a source for that? I'm intrigued. Their own blog post is only talking about a "logic error".

[–] lemmyvore@feddit.nl 5 points 3 months ago (2 children)

I heard a different rumor, that the driver file they pushed was all zeros. I'm inclined to believe that one.

[–] 01189998819991197253@infosec.pub 2 points 3 months ago

They were talking about the Linux instance, not the windows one.

load more comments (1 replies)
[–] themoonisacheese@sh.itjust.works 3 points 3 months ago (1 children)

It's a very educated guess based on the following:

The crash is a null pointer dereference, which a linter ought to catch.

The crash does not happen if you have crowdstrike sensor installed, which is weird because crowdstrike sensor's job is not to prevent any crashes.

Hence the guess: the update the pushed tries accessing memory in sensor, but if it's not installed the pointer is null and that's Bye-Bye.

load more comments (1 replies)
[–] Mango@lemmy.world 5 points 3 months ago

This is the best explanation of this I've heard and you're just like... A dude on Lemmy.

[–] MechKit@beehaw.org 23 points 4 months ago (3 children)

It's a well assembled article, but mostly based on a few comments in a hackernews post from yesterday. I would like to know how widespread it was.

load more comments (3 replies)
[–] LeFantome@programming.dev 8 points 3 months ago

The article implies that CrowdStrike issue impacted only Debian and Rocky 9.4. Debian I can see. But how did something impact Rocky but not RHEL itself or Alma or Oracle?

Is Rocky actually different from RHEL now? Their entire brand promise is that they are the same.

[–] rspfau@ecoevo.social 6 points 4 months ago (1 children)

@lemmee_in I can't find any news about this. Just a statement in a forum and everyone basing subsequent articles on that. It appears to have been limited to a single company? Is there any support for this claim?

load more comments (1 replies)
[–] suction@lemmy.world 4 points 3 months ago* (last edited 3 months ago)

Nobody breaks Rocky ok (spoken in a gruff, low, mumbling voice)

load more comments
view more: next ›