My job puts me in frequent contact with the safety team at a chemical plant. There are sensors all over the plant and warehouse and if the tiniest bit above nominal levels of quite few different chemicals is detected in the air, an audible alarm sounds and lights flash, along with automated reporting to safety team members.
My job does not involve interacting with chemicals, and my duties on the production floor are basically just rebooting printers and plugging computers back in as needed, so my instructions are basically “if the alarm goes off, immediately return to the office and listen out for any further instructions”
The fact that the monitoring mechanism literally only reported overage by email is a terrible design and the organization deserves whatever regulatory and legal backlash they receive for that decision
No mails arrived due to the disruption […]. So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
Kinda scary to rely solely on email for such alarms.
I think that automation is fine, but when you start talking about life-critical systems, the bar on failure modes and reliability becomes a lot higher. Like, if you have a sensor for excessive chlorine levels, the default probably should be not to keep pumping more of the chemical in, but to have some “hard” upper level where no more chlorine goes in (and if the two systems aren’t directly-integrated, to at least let the maintenance safety system power off the chlorine system), and for a secondary alarm to be triggered (like, start playing a loud audible alarm) if maintenance doesn’t respond to the email notification and indicate that they’ve dismissed the alert within a given amount of time. That’s disruptive and undesirable compared to just emailing maintenance, but as long as there aren’t any failures, you shouldn’t ever see it. But if you do have it come up, you don’t want the failure mode to be chemical-keep-being-dumped-into-the-system-and-people-being-unaware-of-it.
Automation can increase safety by decreasing time until an issue is addressed, for example. Automation isn’t intrinsically incompatible with safety. Aircraft computers today use TCAS to automatically instruct pilots what to do if they are about to collide (and in fact, today the automated system is explicitly given higher priority than human air traffic controllers — if the two give pilots conflicting directives, the human is the one to be disregarded). But you gotta make your automated procedures have high reliability and handle failure the way you want when the consequences of a failure are high (and that goes for non-automated procedures too).
When these things go off a literal alarm needs to be screaming that there’s an issue.
Yes, you can set up additional email alerts and logging. But the primary mechanism needs to be an in your face loud uncomfortable alarm, and safety cutoffs like you said.
It’s not excessive automation – it’s a lack of redundancy for critical safety features.
Anything safety-critical should never have a single point of failure if you can help it. And there was definitely an opportunity here to have some kind of secondary backup alarm system.
Excessive automation? That actually seems like a great thing to have automated… just not by email. Nothing stopping them from having a sub ghz radio that sets off a physical alarm somewhere. Improper automation I think would be more apt. A purely local automated alarm sounds great, but the internet shouldn’t be involved imo, at least not on the critical path to remediation.
I for one think that IT team should be kicked to the curb, but they will probably get raises to figure out that they can use email, healthchecks.io, and ntfy all at the same time for redundancy or they will say we need money to switch providers and set up a gmail account to do the same thing and call that redundant cause you know getting two emails saying the same thing is not going to end up just being ignored.
I used to have IT monitors that would email but also text me. Most folks don’t know they can send an email to (phone_number)@(provider).com and that shows up as a text.
Most of those email to sms portals no longer work, at least on the US carriers. Unless you are on a corporate account and your company pays for that additional service.
I know he tried calling 0118 999 881 999 119 725 3 first, but had he not ended up with another country on the line it would have been a much quicker response.
The technical standards for Email would allow for 3 days of delivery failure before finally giving up. That’s the amount of leeway email has (in theory - practically we assume that everything longer than 3 minutes means something has gone wrong). Sending alerts by email is fine for non-critical applications; if one of my docker containers crash, an email is fine because the amount of damage is nil.
Why wasn’t there an actual fail-safe mechanism employed that measures the chlorine levels and shuts the whole thing down BEFORE it gets dangerous?
This is not a Microsoft problem. This is a negligence problem on Phantasialands side, using inappropiate tools for a system that can hurt people on failure, and should be followed up by lawsuits.
Message: To whom it may concern, an elevated carbon dioxide level was measured in the walk-in fridge in the bar kitchen. Probably one of the keg lines is leaking pressurised gas. Gas PPM is classified as alert level 5 - Immediately harmful to life. Please evacuate staff from the area and ventilate the fridge thoroughly before attempting entry with ELSRD hood donned to shut down gas flow.
Oh, well everyone just ignores them, then Jeff who isn’t even on the IT team but was on the mailing list was getting them. So Jeff marked them as spam instead of asking to be removed from the mail list, and now our corporate email provider just black holes them.
Your answer is 100% incorrect. The hotel is solely responsible for the incident as it is their responsibility (liability) for maintaining the system. The hotel failed to properly maintain the system and provide adequate monitoring of said system. If the alerts were delivered by traditional mail and the letter was late, you wouldn’t blame (i.e. sue) the post office.
If the hotel wants to sue Microsoft, they can try. They will get the damages from the missed SLAs, everything else will be laughed out of court.
Your post reads like anti-Microsoft hate rather than actual legal argument. As much as MS sucks, there’s no jurisdiction here much less a burden of care or duty on MS’s behalf.
Yup. They put a safety critical system behind an alert system that is roughly the equivalent of sending a 6 year old across town to deliver a message. Sure they failed to deliver a message, but they never should have been relied upon to do that in the first place. The failure isn’t the kid’s fault; it’s the idiot’s fault that trusted them with a life and death situation.
I have to agree with the other commenters. I feel like this is a task for a loudly audible alarm system and maybe a flashing strobe in the operators office, not an email.
Industrial automaton engineer here, I’ve worked on a bunch of safety critical systems and audited systems after incidents.
Best practice in situations were you are sending a remote start command, is for the stop command to be local to the pump controller.
I audited a system where the main control system sheet a start command to the dosing shed (which stored the concentrated chemicals), unfortunately the stop command was over the same link that got interrupted. The start was received but the stop was never was. Radio is not a reliable link.
For chemical dosing, I’d expect at least two types of command signals, such as concentration and Max allowed time; you could also measure the weight loss of the concentrate as a third check. Loss of communications to the main controller isn’t necessity an instant stop, but I would expect a prolonged loss to trigger a stop. As long as the local controller has stop control, dosing can usually continue.
Yeah, standardly you’d have someone manually check chemicals daily, and the automated system would correct chemicals when not in use, but where I grew up it isn’t so much of a pool being “opened” as it was just that it’s open around 365 days a year and less people just use it in “winter.”
But if its 81F/27C outside on Christmas day, people will be swimming on Christmas…
Wow. Why wire up a blinking lite and speaker in the managers office to the system when you can just rely on fucking email. Pretty sure whoever made it possible to use email intended it to be a backup solution in case no one in charge was on site when the alarm went off.
We were all exposed to this problem: on Friday there was an outage that crippled the US software company Microsoft, including the email program Outlook.
Phantasialand was affected as well. Because of the disruption, no emails arrived, the company told WDR on Saturday. So nobody noticed that the maintenance safety system for the sauna area had long since raised the alarm by email. The automatic dosing system had pumped too much chlorine into the pool.
Comments
My job puts me in frequent contact with the safety team at a chemical plant. There are sensors all over the plant and warehouse and if the tiniest bit above nominal levels of quite few different chemicals is detected in the air, an audible alarm sounds and lights flash, along with automated reporting to safety team members.
My job does not involve interacting with chemicals, and my duties on the production floor are basically just rebooting printers and plugging computers back in as needed, so my instructions are basically “if the alarm goes off, immediately return to the office and listen out for any further instructions”
The fact that the monitoring mechanism literally only reported overage by email is a terrible design and the organization deserves whatever regulatory and legal backlash they receive for that decision
Kinda scary to rely solely on email for such alarms.
Yeah, I hate Microsoft as much as the next person, but this is 100% on Phantasialand for trying to cut corners with excessive automation.
I think that automation is fine, but when you start talking about life-critical systems, the bar on failure modes and reliability becomes a lot higher. Like, if you have a sensor for excessive chlorine levels, the default probably should be not to keep pumping more of the chemical in, but to have some “hard” upper level where no more chlorine goes in (and if the two systems aren’t directly-integrated, to at least let the maintenance safety system power off the chlorine system), and for a secondary alarm to be triggered (like, start playing a loud audible alarm) if maintenance doesn’t respond to the email notification and indicate that they’ve dismissed the alert within a given amount of time. That’s disruptive and undesirable compared to just emailing maintenance, but as long as there aren’t any failures, you shouldn’t ever see it. But if you do have it come up, you don’t want the failure mode to be chemical-keep-being-dumped-into-the-system-and-people-being-unaware-of-it.
You want fail-safe behavior.
Automation can increase safety by decreasing time until an issue is addressed, for example. Automation isn’t intrinsically incompatible with safety. Aircraft computers today use TCAS to automatically instruct pilots what to do if they are about to collide (and in fact, today the automated system is explicitly given higher priority than human air traffic controllers — if the two give pilots conflicting directives, the human is the one to be disregarded). But you gotta make your automated procedures have high reliability and handle failure the way you want when the consequences of a failure are high (and that goes for non-automated procedures too).
It’s like those supermarket fridges.
When these things go off a literal alarm needs to be screaming that there’s an issue.
Yes, you can set up additional email alerts and logging. But the primary mechanism needs to be an in your face loud uncomfortable alarm, and safety cutoffs like you said.
It’s not excessive automation – it’s a lack of redundancy for critical safety features.
Anything safety-critical should never have a single point of failure if you can help it. And there was definitely an opportunity here to have some kind of secondary backup alarm system.
Nothing safety critical should ever go through microslop
On top of that it should fail safely… and not simply stop working without anyone knowing
Excessive automation? That actually seems like a great thing to have automated… just not by email. Nothing stopping them from having a sub ghz radio that sets off a physical alarm somewhere. Improper automation I think would be more apt. A purely local automated alarm sounds great, but the internet shouldn’t be involved imo, at least not on the critical path to remediation.
I disagree, email definitely had a place. Documenting the fault and time it happened.
The way they did it was stupid.
Sure, for record keeping or for distribution.
But not for safety, durable delivery or for actually taking action off of.
No… with the wrong kind of automation.
Safety relevant systems do not belong on public infrastructure
Outlook isn’t public infrastructure lmao.
I wonder if they could have done something like [puts on big brain hat] have an actaul fucking alarm the pool staff and guests can hear.
In proper SRE, there is a rule - mails are not alerts and must not be used as such.
I for one think that IT team should be kicked to the curb, but they will probably get raises to figure out that they can use email, healthchecks.io, and ntfy all at the same time for redundancy or they will say we need money to switch providers and set up a gmail account to do the same thing and call that redundant cause you know getting two emails saying the same thing is not going to end up just being ignored.
I used to have IT monitors that would email but also text me. Most folks don’t know they can send an email to (phone_number)@(provider).com and that shows up as a text.
Most of those email to sms portals no longer work, at least on the US carriers. Unless you are on a corporate account and your company pays for that additional service.
I think major American carriers stopped allowing that, actually
does that only exist at places where SMS is free?
100%. This is not because of an outage, this is because whoever set this up is an idiot.
That’s actually terrifying a lot of people don’t even care on checking their mails on a daily basis
Subject: Fire.
Dear Sir/Madam,
I am writing to inform you of a fire that has broken out on the premises of 123 Cavendon Road.Fire! Fire! Help me!
123 Cavendon Road. Looking forward to hearing from you.
Yours truly,
Maurice Moss
A fire? At a sea parks?!
I know he tried calling 0118 999 881 999 119 725 3 first, but had he not ended up with another country on the line it would have been a much quicker response.
So long as he organized and put all the objects on fire together, he’ll be fine. Also don’t use fire extinguishers from Britain apparently.
The technical standards for Email would allow for 3 days of delivery failure before finally giving up. That’s the amount of leeway email has (in theory - practically we assume that everything longer than 3 minutes means something has gone wrong). Sending alerts by email is fine for non-critical applications; if one of my docker containers crash, an email is fine because the amount of damage is nil.
Why wasn’t there an actual fail-safe mechanism employed that measures the chlorine levels and shuts the whole thing down BEFORE it gets dangerous?
This is not a Microsoft problem. This is a negligence problem on Phantasialands side, using inappropiate tools for a system that can hurt people on failure, and should be followed up by lawsuits.
From: noreply@carbondioxidedetector.com.au
RE: Carbon Dioxide leak detected
Message: To whom it may concern, an elevated carbon dioxide level was measured in the walk-in fridge in the bar kitchen. Probably one of the keg lines is leaking pressurised gas. Gas PPM is classified as alert level 5 - Immediately harmful to life. Please evacuate staff from the area and ventilate the fridge thoroughly before attempting entry with ELSRD hood donned to shut down gas flow.
Folder: Spam
Yeah
“Why do all our alerts go to spam”
Oh, well everyone just ignores them, then Jeff who isn’t even on the IT team but was on the mailing list was getting them. So Jeff marked them as spam instead of asking to be removed from the mail list, and now our corporate email provider just black holes them.
Can’t put any blame on MS here. Using email for safety warnings with no redundancy is negligent.
One can put some blame on them, legally speaking.
Which of those claims will stick remains to be seen, but “no blame” is incorrect.
Your answer is 100% incorrect. The hotel is solely responsible for the incident as it is their responsibility (liability) for maintaining the system. The hotel failed to properly maintain the system and provide adequate monitoring of said system. If the alerts were delivered by traditional mail and the letter was late, you wouldn’t blame (i.e. sue) the post office.
If the hotel wants to sue Microsoft, they can try. They will get the damages from the missed SLAs, everything else will be laughed out of court.
Your post reads like anti-Microsoft hate rather than actual legal argument. As much as MS sucks, there’s no jurisdiction here much less a burden of care or duty on MS’s behalf.
Yup. They put a safety critical system behind an alert system that is roughly the equivalent of sending a 6 year old across town to deliver a message. Sure they failed to deliver a message, but they never should have been relied upon to do that in the first place. The failure isn’t the kid’s fault; it’s the idiot’s fault that trusted them with a life and death situation.
Yeaaah not MS fault here, no sane person would rely solely on email for something like this
This is why I always use Yahoo mail for my critical life safety notifications.
Make sure to throw the system status dashboard up on GeoCities
fuck microslop but that one is 100% on Phantasialand
I’ve been to Phantasialand. It was a nice park.
I have to agree with the other commenters. I feel like this is a task for a loudly audible alarm system and maybe a flashing strobe in the operators office, not an email.
Industrial automaton engineer here, I’ve worked on a bunch of safety critical systems and audited systems after incidents.
Best practice in situations were you are sending a remote start command, is for the stop command to be local to the pump controller.
I audited a system where the main control system sheet a start command to the dosing shed (which stored the concentrated chemicals), unfortunately the stop command was over the same link that got interrupted. The start was received but the stop was never was. Radio is not a reliable link.
For chemical dosing, I’d expect at least two types of command signals, such as concentration and Max allowed time; you could also measure the weight loss of the concentrate as a third check. Loss of communications to the main controller isn’t necessity an instant stop, but I would expect a prolonged loss to trigger a stop. As long as the local controller has stop control, dosing can usually continue.
Email is not a guaranteed delivery system. Now they know.
What the fuck?
did not expect that on my bingo card.
Those who do not study history are doomed to repeat it.
This isn’t MS fault, what happened to Normal Alarming System
Don’t be so NASty
Why the fuck would you open the pool without getting a mail saying it is safe?
The email is that there is a dangerous condition, not that it is safe.
I’m thinking maybe that should be the other way around.
That’s such a rookie mistake to make in safety design…
Using email is the rookie mistake. Email does not even guarantee delivery, let alone in timely fashion.
Meh… Doesn’t have to be guaranteed because you assume it’s unsafe anyway. So if you don’t get the mail, go to the machine and see what’s happening.
Yeah, standardly you’d have someone manually check chemicals daily, and the automated system would correct chemicals when not in use, but where I grew up it isn’t so much of a pool being “opened” as it was just that it’s open around 365 days a year and less people just use it in “winter.”
But if its 81F/27C outside on Christmas day, people will be swimming on Christmas…
Damn, Microsoft hurting people they didn’t even mean to (in addition to the ones they mean to).
Wow. Why wire up a blinking lite and speaker in the managers office to the system when you can just rely on fucking email. Pretty sure whoever made it possible to use email intended it to be a backup solution in case no one in charge was on site when the alarm went off.
Does seeing memes count? Otherwise, not really. I wouldn’t even have known it was down.
https://feddit.uk/pictrs/image/d6dd77b1-5a98-4a0b-a293-d62cd7c61674.webp
https://lemmy.world/pictrs/image/a43b66e3-654c-4d36-ba79-6be578780a98.png
GTFO of the cloud, you idiot humans
In English: https://timesofindia.indiatimes.com/technology/tech-news/a-hotel-in-germany-reports-chlorine-gas-poisoning-after-email-warning-system-fails-due-to-microsoft-365-outage-guests-taken-to-hospital/articleshow/135864509.cms
mff 2014 ass bullshit
[deleted]
Sign in to comment. Sign in