Comments

  • By bgentry 2026-03-0120:245 reply

    The important quote from the timeline:

    Mar 01 9:41 AM PST

    We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.

    • By dehrmann 2026-03-021:291 reply

      > impacted by objects

      Well that's a way of phrasing it.

      I assume these were missiles.

    • By jiggawatts 2026-03-0121:412 reply

      This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…

      Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.

      A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.

      “Oh, that’s in case there’s a fire!”

      “What does it do?”

      “It cuts… the power… uhh… for the safety of the fire department.”

      “So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”

      “… yes.”

      “Not 100% then, is it?”

      • By protocolture 2026-03-0123:362 reply

        Equinix in Sydney plonked 2 datacenters right on top of each other, and still insists that they are useful as redundant sites.

        There was a locally very funny situation for a while when a tech influencer was insisting both equinix sites could be shut down by a single building collapse. He was wrong, but he wasn't so wrong that people shouldn't make better infrastructure decisions.

        • By bigfatkitten 2026-03-025:472 reply

          The two in Alexandria are of course just down the road from the airport. A widebody jet cratering in that location will put them both out of business.

          • By protocolture 2026-03-0222:231 reply

            None of the runways point towards them, its hard to imagine that scenario.

            • By bigfatkitten 2026-03-031:37

              34R does. The immediate right turn in the 34R ILS and RNP missed approach procedures takes you pretty much directly overhead Equinix.

              Aircraft routinely overfly the location either on departure from 34R or approach for 16L.

          • By jiggawatts 2026-03-0211:031 reply

            They're also below sea level.

            • By protocolture 2026-03-030:011 reply

              Theres an Energex office building in Brisbane that has a backup generator below sea level. (Guess when they need to use the generator lmao.)

              • By jamesfinlayson 2026-03-040:29

                If it's a once in 25 year flood then it won't be a problem for the person who made that decision!

        • By hi_hi 2026-03-022:591 reply

          I'm intrigued...how was he wrong?

          • By protocolture 2026-03-023:19

            The building in question wasnt really tall enough. And would have to be precision demolished to collapse in the way he was afraid of.

            It would still cause chaos and possible power issues.

            Needs to be taken in context with some Sydney buildings having maintenance defects a few years after they open. Largely due to inferior materials imported from china. The building in question developed some cracks in supporting beams and was briefly evacuated. There was never a chance it was going to topple on its own in a way that impacted more than 1/2 datacenters, so he pivoted to possible terrorism, but even thats largely nonsensical.

            I just went hunting for the case and couldnt find it. The gentleman in question had published the claim to his business that was it happens trying to build contacts with defense and intelligence agencies for third party threat assessment. As far as I can tell the business no longer exists and he has deleted their footprint.

            But he also posted the claim on public mailing lists so I can probably trawl it up if necessary.

      • By vntok 2026-03-0122:082 reply

        Should have pushed it.

        • By dminvs 2026-03-0123:032 reply

          at The Planet in Dallas c. 2002 the EPO button was exposed with no cover, and in very very close proximity to the "Exit" button for the doors...

          one day, a colo customer hit the wrong button on the way out, and uhh, there was an outage

        • By skywhopper 2026-03-0210:33

          My boss at my first job hit the Big Red Button by swinging his arms too big in our datacenter one day, shutting down hundreds of servers and the mainframe, wreaking havoc for days!

          That was when we installed the Big Clear Button Cover.

    • By xg15 2026-03-0215:51

      Impact assessment: yes.

    • By Imustaskforhelp 2026-03-0120:334 reply

      > we will ensure we restore power and connectivity safely

      this would require human intervention and I am a bit worried what if the strike can happen again and human lives might be lost.

      IIRC there have been cases in history where sometimes a same location is targeted across multiple days. Obviously, AWS might have local employees working in the region but would there be an evaluation of this threat itself within the relevant team in AWS. What if they try to bring the service back but then missiles are struck again and what if human lives might be lost on it. Let's just hope that it could be part of a evaluation as well.

      • By subscribed 2026-03-0123:12

        I wouldn't risk it.

        Both Americans and Israelis are known for double taps. Surely Iran can adapt their tactics too.

      • By pwarner 2026-03-0122:101 reply

        But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?

        • By Imustaskforhelp 2026-03-025:40

          > But I mean,are the employees safe at home? I guess if the really targeted the data center then home is safer, but in the fog of war maybe the data center wasn't the target?

          My gut feeling says that they would be safer at homes than at datacenters. The only large info I have heard is attack on hotels, this datacenter etc. (atleast till right now).

          > but in the fog of war maybe the data center wasn't the target?

          We can't say this for sure but even if that was the case, I do think that they would see some damage was caused and then, try to double tap it for even more damage. So chances are, even if it wasn't the target previously, it might be the target now?

      • By zozbot234 2026-03-0123:20

        > this would require human intervention

        Amazon has self-propelled robots that handle their logistics and fulfillment, don't they? Send in the robots.

      • By tokyobreakfast 2026-03-0121:362 reply

        > this would require human intervention

        that's the difference between heroes and ordinary employees who bitch about having to go into the office twice a month.

        same as the stories you hear of guys taking snow-cats up a mountain in a blizzard to restore phone circuits or radio transmitters gone offline.

        • By flymasterv 2026-03-0121:451 reply

          Man, don’t be a “hero” trying to restore a lower ping to someone trying to buy a kindle in Jeddah.

          • By ok_dad 2026-03-0121:493 reply

            What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.

            • By stonogo 2026-03-0122:101 reply

              In that case, the hero was the person who avoided relying on a single AZ when they deployed to cloud.

              • By Imustaskforhelp 2026-03-025:55

                100% absolutely but its a bit worrying if in the future multiple AZ/datacenters could be start to get targeted?

                attacking datacenters within a particular region so that service would have a hard time.

                I guess someone can use some other regions DC to have more than (regional?) AZ but for mission critical infra, I can see that having sometimes issue too and you genuinely can't predict any of all of this.

                That being said, There should be more than one AZ reliance but IMO also off-site or multi-cloud backups should also be preferred/used as well.

            • By flymasterv 2026-03-020:37

              Their lack of multiple AZ’s isn’t the guy making 30k a year’s problem.

            • By Imustaskforhelp 2026-03-025:49

              > What about local hospitals which may have service from that data center? There are heroes needed everywhere, all the time.

              Off-site backups/Multi Cloud Strategy while encrypting data (and keeping the key safe, key point) might be a better strategy for such mission critical infrastructure.

        • By thatguy0900 2026-03-0121:501 reply

          I'm sure bezos will be really happy someone is being a hero for him in a war zone while he sails his newest yacht to wherever the new version of the island is.

          • By tokyobreakfast 2026-03-0122:04

            on second thought there is a difference between restoring critical infrastructure in times of crisis vs restoring bot infrastructure for indian spamming operations. choose wisely

  • By p-o 2026-03-0120:149 reply

    Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future.

    Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services.

    Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.

    • By swiftcoder 2026-03-0120:413 reply

      The more interesting question, is how many datacenters are just plonked next to a high-value military target?

      For infrastructure reasons, we plonk datacenters down next to airports big enough to fly major hardware into, and near where the big oceanic cables come ashore… and for strategic reasons those are also the perfect places to place military bases

      • By throw475787 2026-03-0121:221 reply

        Is there acrually some meaningful physical separation between military and civilian server deployments?

        We seem to be really bad at separating those two. For example Starlink is basically military infrastructure now, used to guide bombs.

      • By pinkmuffinere 2026-03-0220:421 reply

        You sound like you know things, and I have questions! What are the chances that we see infrastructure move to less-developed-but-more-stable regions of the world? What even are the candidate locations that wouldn't be an absolute headache to set up? I'm thinking perhaps West Africa? Or South Africa if it can stabilize a bit? Maybe other coastal locations that have good-enough transportation hubs nearby?

        • By swiftcoder 2026-03-048:33

          AWS spreads a pretty wide net - there are already datacentres in Cape Town, São Paulo, Jakarta... You could certainly deploy your app far enough to dodge any regional instability.

          However, proximity to money/power tends to be a factor in business, and the bulk of datacentres cluster around US/EU/MENA/China/Japan

      • By cherryteastain 2026-03-0121:42

        A datacenter IS a high value military target.

    • By Imustaskforhelp 2026-03-0120:242 reply

      > Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.

      The bigger part of me seems that if we someone nukes 50 datacenters all at once or say all of Amazon's datacenters at once, then the data stored in there would simply be gone and given so many datacenters are located in Virginia,USA iirc or just so many companies being reliant on few datacenter providers.

      The larger threat to me with the lose of data is firstly the panic within public fronting services but also, with Hedge Funds, Pension funds or banking datacenters who might be using these and if they lose the data, then its gonna cause even more public mayhem.

      Some might be saying oh off-site backups exist but there has atleast been one instance, where a single Google accident had led to massive issues for a 135 Billion $ pension fund.

      Relevant Kevin Faang video about it: https://www.youtube.com/watch?v=3GOAUyipnM4 [Google Accidentally Deletes $135 Billion Pension Fund, Chaos Ensues]

      • By treesknees 2026-03-0212:001 reply

        In my experience many middle eastern companies tend to only operate out of the Middle East AZ’s. They’re not backhauling their data and customers to us-east-1. If the goal is to severely disrupt middle eastern rivals, then you don’t need to hit every possible AWS datacenter.

        • By Imustaskforhelp 2026-03-0214:031 reply

          Interesting, I didn't know that so thanks for telling me something new.

          But why is this the case? Like, saving costs? Doesn't this recent attack on AWS DC does show that they aren't as safe as previously thought especially in a region of conflict.

          Is there any particular reason as to why this is the case?

          • By jamesfinlayson 2026-03-040:44

            No idea what the laws are like but maybe data sovereignty? I'm in Australia and certain industries require Australian data to stay in Australia, which until ap-southeast-4 came along meant relying on one AWS region.

      • By fc417fc802 2026-03-0123:01

        Notably they did have backups. As you would expect for a $135 billion dollar undertaking. It's just that restoring from a calamity tends to be time consuming (a key difference between failover and a backup).

    • By roncesvalles 2026-03-0121:02

      Exactly. 2 is only sufficient for HA against random failures. It's not enough for HA against a determined adversary willing to use targeted force.

    • By tbrownaw 2026-03-0120:26

      > Seems like it should be somewhat easier to nuke 50 datacenters than it would be to hack and disrupt 1000s of different services.

      Previous outage news makes it sound like the cloud providers still have quite a few logical single points of failure.

    • By Zeyka 2026-03-0120:281 reply

      That's so interesting. Are any of the US military (or other satellite state of the US) systems running in "normal" datacenters or do they have a few protected DoD datacenters in the US?

      • By Imustaskforhelp 2026-03-0120:411 reply

        Found this relevant article: https://serverlift.com/blog/military-modular-data-centers/ (AWS Military Modular Data centers)

        I do think that though, atleast from the Anthropic decision prior, we know that Anthropic which was used by DoD should be on normal AWS datacenters.

        I am saying this because, Dod Threatened to force take the source code of Anthropic if they don't agree to aggregious demands so that means that they don't have the source code.

        Perhaps DoD used Anthropic within AWS Military modular DC's but I find it extremely unlikely.

        I am almost certain that even with OpenAI who bent its knee to DoD, its still hosted on regular infrastructure and DoD is using these AI models on pretty sensitive tasks (During the Venezeula Maduro's capture, Anthropic/Claude were used iirc to handle some data analysis)

        IMO Tho, Any Employee from Anthropic/OpenAI might know better tho about how these models are deployed.

    • By breppp 2026-03-0210:20

      > doesn't seem it was the case in this instance

      To me it absolutely looks like the case in here, what are the chances three availability zones across two different countries have been attacked?

    • By ineedasername 2026-03-0123:311 reply

      Data centers in space no longer look so unreasonable when the requirement is “redundancy against multi site bomb strikes mid op”. A little depressing when some pieces start to fit together.

      • By Kye 2026-03-021:012 reply

        How would that help? The ground station that handles local routing to and from the orbital station becomes the target.

        • By vineyardmike 2026-03-023:15

          I’m not exactly fully bought into the idea (for many practicality reasons) but it seems easier to build many (and replace) ground stations than data centers.

          Additionally, StarLink et al are now able to directly communicate with cell phones. It therefore should be possible to route entirely in space between “data center satellites” and communications satellites and communicate directly with an end user device, avoiding the entire terrestrial internet.

        • By ineedasername 2026-03-0217:55

          Routing? Why route? the link at the point of comms is direct. There's no extra ground station, unless you mean the person/people doing things. But either they're communicating direct to a DC on the ground, or one in space, or wherever. Setup so that-- sure-- they can work and coordinate better if someone isn't dropping bombs on ground nodes but that all nodes have some independent capabilities as well.

    • By jcgrillo 2026-03-0120:29

      IIUC part of the reason ballistic missiles have multiple warheads is that some of them detonate high up to knock out air defenses and other electronics allowing the rest to fall through to their targets. The last time we tried this experiment as a species was the starfish prime tests in 1962 which caused some electrical havoc in Hawaii. These days our systems are probably more delicate and sensitive? All that is to say, in a scenario where nukes are going off I'm not sure you'd even need to target any datacenters in particular.. they're probably all toast by default.

    • By roxolotl 2026-03-0120:56

      This is the data center version of https://xkcd.com/538/. Realistically if there is a hot war what you’re saying seems accurate.

  • By ejdyksen 2026-03-0120:01

    Just one AZ, not the whole region:

    > The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.

HackerNews