What patterns do we see in RUM data during the April 28th power outage in Portugal and Spain?
By Philip Tellis on
On Monday this week, the Iberian Peninsula suffered a major power outage that disabled many services across these countries. In this post I'll look at the patterns we saw in mPulse data during this time.
Overall traffic
I first looked at the overall data patterns for traffic originating from Spain & Portugal for several weeks prior to the outage. While other regions were also slightly affected, these two countries suffered the worst of it.
The image above shows traffic patterns for these two countries for 7 days around the event at 1 hour granularity. We see that on average (in that mode I mean the medians), Spain has about 4.5 times the traffic of Portugal. This is expected, since Spain's population (48m) is 4.5 times that of Portugal (10.5).
Identifying the Anomaly
I then ran mPulse's Anomaly Detection against this data.
The Anomaly Detection logic shows that traffic dropped below expected levels between 12pm & 1pm CEST (local time in Madrid), and then returned to normal sometime around 2am on the 29th. This chart only shows traffic from Spain, but you get the picture.
Drilling into the details
I then looked at the data at finer granularity. This time at 5 minute intervals comparing April 28th to the week prior. Ideally I'd have used the previous Monday, but as that was the Easter Monday holiday, the traffic patterns were unusual on that day as well. I settled on Wednesday, April 23rd as a representative day for the prior week. Just to be sure, I also looked at data from France & Germany to make sure we weren't seeing a common change in pattern.
Also, to be clear, for any of the following charts that refer to a single country, the time axis will be in that country's time zone. This means that Portugal shows up in WEST rather than CEST. The outage started at 11:33am in Portugal which is 12:33pm in Spain.
I looked at several different dimensions, but the first one to stand out was Device Type.
Mobile Lunches
For all 4 countries that I looked at, there is a sharp drop in desktop traffic around lunch time. France and Portugal have a 2 hour lunch break from about 12-2pm. Germans take a 1 hour lunch from 12-1pm. Spaniards have a later lunch, starting around 1pm, and going on until 4 or 5pm. This could possibly be due to the tradition of afternoon siesta.
During this lunch break, we see traffic moving to mobile devices.
During the outage
At the start of the power outage, we notice that while traffic drops 80% on desktop, there is only a 40% drop on mobile. At this time, and going on for a few hours, there is more mobile traffic than desktop traffic. In fact mobile traffic remains higher than desktop traffic until the next morning, but this appears to be a common pattern where traffic typically moves to mobile as people leave work around 5pm.
I was curious about that intial part of the outage. If there wasn't power, then how were people still connecting to the Internet? Naturally their phones still had sufficient battery power, but were there cell phone signals or other networks that were usable?
Connection sources
For these charts I looked only at mobile devices to see how they were connecting to the Internet.
For both Spain & Portugal, we see that Wired connections (this includes WiFi you may have at home, the office or a coffee shop since it uses NAT over a wire, but not ISP provided WiFi networks where you get a public IP over WiFi) drop off significantly. There's about a 50 percentage point drop in wired traffic. At the same time, Portugal saw a 30 percentage point increase in Cellular network usage. There were even a few wireless hotspots that appear at this time.
Destinations
What kinds of sites were most interesting at this time? What were people willing to use their limited electric reserves on? I grouped destination sites by Industry to try and make some sense of it.
There are different patterns for Spain & Portugal, so I'll look at them separately.
Remember that in Portugal the outage started at 11:33am local time. Almost immediately we see an increase in traffic to news websites with people trying to find out what was going on. There are also periodic updates on government websites that see an increase in traffic.
There's a drop in Retail, Entertainment and Financial activity.
There is also a sudden new stream of visits to sites dealing with Food Safety and the safe handling of food, possibly because people have to figure out what's safe to hold on to without refrigeration. I should mention that there were no visits to these sites at any time prior to this event for the several weeks that I looked at the data.
Food safety and Government websites seem to be common with Spain as well.
However Spanish users do not visit news websites as much. Most of their information comes from government websites.
I also found the steady stream of travel websites interesting. People needed to know if the outage affected their travel plans.
How long could it last?
Finally I wanted to know how long folks could keep using the Internet from their phones, so I looked at battery percentage data. One caveat is that we do not have battery data from all users, so this is a much smaller sample. Once again, I compared the data during the outage with the previous week, and also with other countries in Europe to make sure we weren't catching common patterns.
For other countries in the EU, we see that the change in battery levels is consistent week over week.
For Spain & Portugal, we see a deviation from the prior week's pattern just as the outage starts. At the peak of the outage, phone battery levels are about 10 percentage points lower than the week prior. As partial power starts to come back, first in Portugal, and then in Spain, we see levels get back to the previous week's pattern.
There's another minor drop later at night when more power is restored and more people start to come online, but things start to normalize soon after that.
In closing
This is one of the few posts I've written that doesn't directly talk about any of the performance metrics we collect. I did look at the change in network latency during the event, but all visible effects appeared to be a side-effect of reduced traffic. It's important to look at traffic as well as other metrics because you do need a large enough sample to have stability in everything else.
In a world that's so dependent on having access to electricity, it's interesting to see how some information services were still able to stay online and available. These sites were served by Akamai, and having battery backups, generators, and edge nodes geographically dispersed helps our customers stay online during local events like these.
Read the original article
Comments
By codetrotter 2025-05-0512:488 reply > some information services were still able to stay online and available
I’m in Valencia, Spain.
The mobile internet connectivity here during the power outage was very unstable.
Cellular phone signal strength was also very very low for the majority of the time.
Even sending SMS or WhatsApp messages would not work most of the day, because of just how unusable mobile connection was for me and my girlfriend and our families here.
And I only managed to load news pages, national or foreign, a few times during the hours of outage, to try and get some information on what cause, how widespread, and how long it would probably take to restore power.
On the plus side I did get to try my little solar panel for the first time to try and charge one of my power banks using solar power. And it did seem to get some juice out of it.
The biggest problems of all from my pov was:
- We live on the 8th floor with a 1 year-old baby. Going 8 floors of stairs with the stroller was not fun.
- All my money is electronic, except from one 50 euro bill I had in my wallet. How was I going to pay for water and food if this outage would go on.
- What’s going on? How bad is it? How long is it going to last? Very unstable mobile internet as mentioned.
In the end we ended up staying outside going for a walk and meeting up with my mother a bit and then me and my girlfriend and our baby going to the beach and sitting there until late. Finally when we came home lights were starting to come back on. And the elevator was working again too!
The next day the first thing I did was walk to the nearest ATM and withdraw several hundred euros, and I bought a bunch of water. We don’t have a car, so I used one of my big bags with wheels to be able to bring more water home than usual.
By sillyfluke 2025-05-0513:103 reply > All my money is electronic
Yes, one positive aspect of these types of events is that the hazing against the cash-first minority worldwide has ebbed slightly. Sweden seems to be backtracking from their cashless push due to the threat of Russian cyberattacks as well.
In related news, high-speed trains appear to have been sabotaged in Spain today, causing transportation chaos again. This happened while they have not been able to conclusively determine the cause of the blackout.
The plot thickens...or gets sidetracked, depending on what the truth turns out to be.
By makeitdouble 2025-05-0513:403 reply > hazing against the cash-first minority
That's...a pretty strong opinion.
Otherwise cash will still have it's issue during a blackout. For instance I'm not sure most shops would operate their POS during a blackout or without any connectivity, at least if there is any hope of resuming normal operations within days, it would screw the ledgers. ATMs of course are dead. Vending machines are also probably not ready for that (Japan has emergency ready ones, I can't imagine other countries doing that)
We're already in a world where cash is second class citizen, and it won't just get back to the "good old days" because of a temporary outage.
And it will also be a different story altogether if power/internet never comes back. Having cash stashed somewhere might not help you that much.
It's interesting and informative to watch how different places handle power failures (which where I am in the USA are not common but not entirely rare, either).
Most of the bars keep serving to cash customers, and use paper to make notes for future bookkeeping. Some even start using paper tabs.
Big companies switch to backup generators (Walmart) or immediately cease business (also Walmart, because the card communication failed).
Some smaller ones had no lights to continue to be safe inside, so chased everyone out.
Other ones had enough windows and kept selling on a cash basis, making notes by hand. Some of these could open the cash drawer others couldn't, but made do with what they could.
Which IMO highlights what the money is. It’s a trust bond, nothing more and nothing else.
Paper money isn’t guaranteed to be always accepted. It only takes most governments a one day to say something like “because of <big crisis> we’re no longer accepting <currency> as they’re subject to <something bad>”. Paper money becomes fireplace material in an instant. (Or dystopian scenario - dictator decides that currency transactions can be done only by citizens, all other transactions are void).
What is really weird is this dualism in trust. On one side there’s distrust to government controlled organizations about online money, and on the other there’s trust that government (own or foreign) won’t regulate it into oblivion for some silly reason.
If we get to apocalypse level crisis one is much better off on stocking cigarettes, coffee and chocolate anyway…
By bombcar 2025-05-0614:10 It's even more amusing to realize that multiple times in history, paper money has continued to "have value" and be used in exchange even after the government devalued it completely - usually by ceasing to exist.
> That's...a pretty strong opinion.
If you're a person who uses cash a lot, the comments you hear do start to feel a bit like hazing. You very often hear jokes like "who uses cash anymore?" both directed at you and not, like you're a crazy person for preferring not to support Visa's advertising empire with a ~1-3% tithe on every purchase.
By makeitdouble 2025-05-061:48 I think there's the disconnect between the US perspective and the other countries which makes the discussions difficult.
The EU caps the credit card rate at 0.2-0.3%, so the moral argument for not using card payments is a lot weaker, and most countries have also widely used non credit card electronic payment.
SEA countries are mostly in the same boat, with higher credit card fees but more competition.
By 3D0MR3991N 2025-05-174:43 No one reputable cares about cash anymore. The people clinging to paper money are analog conspiracy nutters, the kind of people who hoard pennies in mason jars and refuse convenient contactless payments. I'm pleased to see them awkwardly counting out wrinkled bills while the rest of us tap and go at the spaces I inhabit.
By sillyfluke 2025-05-0513:521 reply >That's...a pretty strong opinion.
I'll go out on a limb and say it's only a strong opinion for anyone who isn't familiar with trying to use cash exclusively for all physical transcations under 1000 dollars in their day-to-day lives.
In London, they have tube stations with a single coffee stand on the platform that's card-only. It's a fucking outrage in my humble opinion. and just another form of debanking, pure and simple.
By Symbiote 2025-05-0516:10 Denmark is often cited as an example of a society that's advanced for electronic payments, and it is — but there's a law here that means, in most circumstances, a business must accept cash.
The official advice was changed from "keep some cash for emergencies" to "keep some cash in small banknotes for emergencies" to "pay in cash at least sometimes, to keep the systems that deal with it functioning".
By blockmarker 2025-05-0520:20 It is not at all certain that there was any sabotage. Supposedly it was sabotage because important wires were stolen, but wire has been stolen by criminals for decades to sell for the materials. And for the last few years there has been an increase of delays, breakdowns and failures in the whole railway network. It is far more likely that common theft on a decaying system caused the problems, but that would pin the blame on the government for this decay. As such they prefer to blame anyone else, including shadowy enemies sabotaging the country.
The cause is the frequency drop that was not compensated by the inertia of rotating turbines due to increasing use of photovoltaics. See https://x.com/shellenberger/status/1916893181876326868?t=32a... A high level engineer in a Spanish generation plant confirmed this to me.
> The cause is the frequency drop that was not compensated by the inertia of rotating turbines due to increasing use of photovoltaics.
But what caused the frequency drop? Large-scale grids are designed and operated in such a manner that any single fault, even one which causes a frequency drop (like a generator or a power line getting disconnected), will not cause a blackout. Which means: if there isn't enough inertia to compensate the frequency drop caused by a single fault anywhere in the grid, the system operator will either order photovoltaics and wind turbines to reduce their generation to a safer level, or order traditional rotating generators to operate as synchronous condensers (which adds inertia without adding generation).
Which means that either there was a double fault (two faults close enough in time that there wasn't enough time to reconfigure the system to a safer state before the second fault), or that the modeling of how the photovoltaics and wind turbines would react to a single fault was incorrect (for instance, expecting them to stay connected for longer on that level of frequency drop). My personal guess is that we're going to see a repeat of what happened here in Brazil in 2023, as I explained in another comment on an earlier thread (https://news.ycombinator.com/item?id=43821801), where a single fault was enough to destabilize the system because the inverters in wind and solar power plants disconnected earlier than expected.
According to my friend, the freq drop was caused by a sudden large supply surplus over the instantaneous demand. Nuclear plants were offline and there was nothing to absorb the freq drop at that moment.
> According to my friend, the freq drop was caused by a sudden large supply surplus over the instantaneous demand.
Wouldn't a supply surplus cause a frequency increase, not a frequency drop?
By cft 2025-05-0515:39 In the case of the rotating generators, yes. In the case of the solar panels, I do not know: I guess it depends on the inverters characterists? Spanish is not my native language, so I may have mixed it up when talking to him.
By otherme123 2025-05-0514:58 Nobody knows the cause at the moment. All that we have are guesses and FUD. Even "high level engineers" don't know for sure what happened.
> - What’s going on? How bad is it? How long is it going to last? Very unstable mobile internet as mentioned.
Silly question but do you have AM or FM radio? When the lights went out in the northeast blackout of 2003 we turned to our cars to put on AM radio. Even after Hurricane Sandy my mother was without power for 3 weeks and she was running a battery powered radio.
I shudder to think of a future where moving information requires high performance digital electronics vs. a crystal radio set.
By codetrotter 2025-05-0516:14 It’s a very valid question.
I don’t have one currently. But I did hear later that others were using radio to get news.
Thank you for bringing it up again. I’m gonna buy a small battery powered radio :)
My Android cell phone has a FM radio app that was pre-installed at the factory. It requires the use of wired headphones to act as an antenna, but otherwise works fine.
By Vilian 2025-05-0523:19 Sadly some new models are removing that function :/
By prof-dr-ir 2025-05-0513:441 reply > The next day the first thing I did was walk to the nearest ATM and withdraw several hundred euros, and I bought a bunch of water.
That is a very good idea for everyone. Putting together an emergency supplies kit is what various European governments, and now also the European Commission, are beginning to officially recommend:
https://www.euronews.com/my-europe/2025/03/26/brussels-ask-e...
> What’s going on? How bad is it? How long is it going to last?
I think some governments suggest that people buy a battery-powered or hand crank radio to address exactly this issue.
By mjevans 2025-05-0514:11 Many of these also have small solar panels. Enough to recharge the device and sometimes build up a charge for other devices like a tablet or cell phone. It wouldn't be enough to continuously run that greedy screen, but it would be enough to maintain standby radio contact.
> And I only managed to load news pages, national or foreign, a few times during the hours of outage
I think this is a problem with https. I remember intermittent connectivity as way better before Google forced the issue.
And yes I like https. But it comes with drawbacks. E.g. no isp caching.
I don't think ISP caching would be a thing without https. It would bring a lot of additional complexity and resource requirements for them. I can hardly imagine that being worth it to save some bandwidth. Maybe it made sense in a world where bandwidth was very limited.
Also I am very happy that it is not a thing and that ISPs cannot do that. When I go to a website I want to get the website from the webserver exactly as the server delivers it and not some other page that my ISP thinks is how the website should look.
Besides with global CDNs we have something very similar but better anyway. I don't get the site from the other side of the world but from the closest CDN server that does caching. The important difference is that the CDN server is authorized by the website to cache the page and the webmaster has control over what it does.
> I don't think ISP caching would be a thing without https. It would bring a lot of additional complexity and resource requirements for them. I can hardly imagine that being worth it to save some bandwidth. Maybe it made sense in a world where bandwidth was very limited.
Transparent squid proxies were common back when most sites were on http. They let ISPs reduce the use of their limited upstream bandwidth, while also making sites load faster. The complexity and resource requirements were modest: install squid on a server, and configure the router to redirect (masquerade) all outgoing TCP port 80 connections to the port configured for squid on that server.
By blahaj 2025-05-0521:52 The internet is much bigger, more diverse and complex today. You need a lot of storage to get any meaningful impact. Caching the http of Wikipedia won't get you much. You need to cache lots of YouTube videos. Or you just get them from the data center you peer with over the fat link you built.
With bandwidth usage the diversity of the data retrived over the internet has also gone up. You can't just cache the few most popular websites and save most bandwidth. But bandwidth capacity has scaled a lot so you probably also do not need to.
> When I go to a website I want to get the website from the webserver exactly as the server delivers it and not some other page that my ISP thinks is how the website should look.
You could have some hash check to prevent hijacking. The old method would be naive today.
There would be some privacy concerns I guess. But it could be opt-in on the site owners part. I think caching some videos and pictures would save a lot of power.
> Besides with global CDNs we have something very similar but better anyway.
Sure but they are some switches away.
> You could have some hash check to prevent hijacking. The old method would be naive today.
But how do you know that the cached site is up to date? How does the ISP know that? What about dynamic content? What about consistency between different requests that are part of the same page load?
> Sure but they are some switches away.
My point is that this does not matter much. Usually, at least in non sparsely populated parts of the world with modern infrastructure, these switches are close and there is lots of bandwidth capacity.
I just don't think it makes sense for ISPs to save bandwidth on these links by building their own local data centers when they peer with a CDN data center anyway.
By rightbyte 2025-05-067:11 The root html would govern what caches are up to date with the hash for some non-dynamic payloads and the root html would not be cached. Etc.
It would be interesting to know how much bandwidth would be saved by caching X gb of the most downloaded films, pictures and textfiles at a neighbourhood level.
In the 90s early 00s I think the share was way bigger than it would be now.
By heraldgeezer 2025-05-0519:58 In Sweden we all get this. Saying you should have water, food, radio and cash and more so maybe Spain or EU needs this too :) Now do I have this, no. But we are further.
https://rib.msb.se/filer/pdf/30874.pdf
EU has started this a bit, we are waking up. EVROPA.
https://www.dw.com/en/european-union-response-disasters-war-...
By gus_massa 2025-05-0516:11 > And I only managed to load news pages,
Did you try with HN? I remember a long time ago I was in a hotel with bad connectivity, and one of the few sites that loaded was HN (no images, almost no JS, ...). I was able to read the comments, but it was difficult to read most of the articles.
By bluesmoon 2025-05-0513:27 Thank you for your personal story about this. It helps to put things in perspective.
I'm in Barcelona (Sabadell specifically,) and the cellular networks were down. Luckily I have a generator and Starlink.
Curious if anyone was able to use their Meshtastic radio to contact anyone.
By tiagod 2025-05-0517:09 I live close to Lisbon (in the South margin of the Tagus river) and I can say Meshtastic was very active throughout the outage.
I received news of power coming back in the first few towns through it, before FM radio.
I needed to get closer to the river to get reliable contacts, but I've now ordered a nice antenna and a solar kit to mount a repeater in a mast on my roof so I can cover the center of my town more reliably.
Isn't Starlink using country-local ground stations for the Internet connectivity? Likely they had power-backup but could it switch to foreign country stations?
By genewitch 2025-05-0514:47 They have lasers on the satellites so they can relay to another ground station. I "come out" in either Dallas or Georgia somewhere. Once I exited in Wyoming or thereabouts.
By madaxe_again 2025-05-0521:35 It’s wild how different experiences can be of the same thing - we didn’t even realise anything was amiss until we went to pick our kid up from kindergarten, and everyone was stood around on the streets looking a bit lost.
We live off grid - independent power supply, starlink, no cell reception in our valley, EV charged off the panels. Just another day.
Worryingly, after about four hours of power cut the local town had already run out of water (they pump up to a relatively small (100m3) holding tank), so we donated our stash in the car to the kindergarten for the kids staying later.
No run on the banks here though - we are super rural and pretty much everyone keeps wads of cash (land deals etc. are almost always done with the official bit and the under the table bit) and has a full pantry at home.
By TrianguloY 2025-05-0513:51 I work on the University, and there I recovered wired internet rather quickly probably due to backup generators. At home most routers stopped, some even took until the next day to be functional again.
As for mobile connectivity, the main issue was the congestion. The cell network didn't fail, usually, but in most places either your phone wasn't able to connect or had no internet. Too many people trying at the same time, I guess. On the University on the other hand it worked perfectly. Maybe because it's a usual crowded place and there are more resources, but I think it was also because a lot of students (even teachers) went home, so those who stayed were mostly alone with a good internet...but less people to talk to.
I was in Spain during the blackout nearby Valencia. My phone had 3G data connectivity from 12:30 to 18:30 despite the outage. Same for the fiber signal, powering the modem&router with batteries allowed me to a working fiber connection for 4 hours. Some neighbors with different mobile operators told me they did not have signal. It might be some operator had backup diesel generator that lasted 4 hours.
By pmontra 2025-05-0513:28 Not only the backup generator of that base station but the backup power of all the network hardware up to it. The base station could have outlasted some other parts that run out of diesel before it did and yet it did not have connectivity.
