OT: The MS issue and what caused it
folks,i know this is OT but i decided to post here as i’m sure there are a lot of people here that work in IT and know what thay are talking about. The recent MS failure is what i would like your opinion on.
from what I’ve seen, MS declared that a bunch of servers went offline and the load had to be spread on remaining servers. They fail to identify what caused the first bunch of servers encountering issues and they claim it was mot related to a hacker attack (why would they, it would lead to massive panic and distrust). So, then what caused the issue in the first place? Could it have been a piloted denail of service attack?
I’m sure the truth will emerge in coming days but was interested in the more tech savy people’s opinione. In the meantime i rejoyce for not having to depend of third party infrastructure at home, thanks to my little homelab.
A further consideration is that this does once again prove that companies that put all their eggs in one basket, are at yhe mercy of big tech. i experienced this personally at works where comunication by Teams and Outlooks was pretty much halted for hours.
Comments
Info about the root cause: https://infosec.exchange/@k3ym0/117607080678996785
You can tell it’s fake, because the EVP understood what was going on
Too real
Brillant! If only it were true and confirmed. MS stock prices would plumet …
That gave me a good chuckle, thanks!
Sounds like a closed case to me.
Maybe the real treasure was the stock price growth we made along the way?
“we”? XD
Whoops!
By 2030 AI is gonna completely replace sysadmins as the cause of outages.
I’m sure it all came down to poor upper management decisions which they will try to blame on technology
Or on the poor lower level man that forgot to follow procedures …
The main reason for problems: Bad IT administration that relies way too much on Microsoft.
There’s been rumours about insufficient capacity on Microsoft cloud services around Europe for a while. Unofficial of course, and who knows how many “broken phones” has been on the chain before me, so don’t take that as a fact. However, Microsoft stated that the problem started when they increased capacity, so maybe some initial synchronization or load balancing starting up caused extra load on the servers already running on pretty much full load and caused cascading failure.
Of course I don’t know for sure and I don’t expect we’ll get a full technical brief with details and the main reason is management problems no matter what’s the technical cause. I hope at least someone wearing a suit starts to think that maybe cloud isn’t really the answer to everything. And it’s not just 36
54, a shit ton of different cloud services relies on AWS, Cloudflare and other players and if any of those goes down they’ll take considerable chunk of business critical infrastructure with them.Have heard similar rumor about capacity. And yes, MS seems to be especially bad about chasing AI at the expense of their core products.
I blame Jensen’s leather jacket. It’s clearly way too powerful.
EDIT: There’s…a 3rd mistress now? Giggity.
Why speculate when you will get it dfrom the horses mouth in a couple days?
But my 2¢ is a bad code being deployed.
just one?
Because the horse lies
Upper management should do things and moves to avoid bad code deploying.
Why should upper manglement have a say in this? They only thing any manglement should say is Yes to a digital twin for a test environment to catch crap like this before it becomes a global outage.
Preventing expensive accidents is a part of survival strategy. Should be interesting topic for any CEO and every level of management. In reality they just hope accidents won’t happen?
Something something health control on too much servers shut them down, which made the others overloaded. My money is on the health monitoring being vibecoded.
I think it is a cascading effect some unmaintained and forgotten server gave out causing others to fall.
Thats what think not based on any facts.
I worked at a place that was running COBOL written in the 1970s that supported thousands of other services across the globe.
it was not redundant. it was not HA. there was just one.
the man who wrote the software died in 1995.
nobody understood what it did, or how to maintain it.
I would say, with the recent layoff from Microslop, your assessment might be more accurate than most think.
I doubt Microslop would pass up the opportunity to blame a hacker group or AI at this point. they look like a bunch of incompetent assholes. since they are being vague about it, and won’t immediately state a root cause, that can only mean it’s incompetence.
All I can say for sure, is that nobody knows for sure.
At the moment, we can only guess. My guess: an update (maybe of a load balancer) went wrong. We will never know for sure.
Maybe this is theproblem with big tech. They will never confess and what strikes me is that businesses and governments just won’t wake up
Sign in to comment. Sign in