Infrastructure at Roblox

2025-06-2522:587466corp.roblox.com

How Roblox's Infrastructure Supports Record-Breaking Games

Roblox’s ability to scale and support tens of millions of users playing together across millions of unique experiences isn’t the result of a single innovation. It’s the sum of a broader culture of innovation and a thousand small things done well across the company. This is how we’ve built the infrastructure that’s currently supporting record-breaking traffic to many of the experiences on Roblox. One of those experiences, Grow a Garden, recently broke the Guinness World Records® achievement for most concurrently played video game, with 21.6 million users playing simultaneously. And in the process, the Roblox platform has continued to hit new peak concurrency records (as it has over nearly two decades), most recently exceeding 30 million concurrent players.

Roblox faces unique challenges in building and maintaining infrastructure for millions of creator-built experiences, including Dress to Impress, Adopt Me, and Dead Rails, requiring innovative engineering methodologies. The platform supports dozens of hourly updates and more than 30 million concurrent users with infrastructure that scales during unexpected traffic spikes. This infrastructure must support thundering herd situations where more than 21 million users join a single experience simultaneously (and the update code came from independent creators). Roblox engineers innovate solutions by challenging traditional wisdom—solutions that are inspired by our four core values.

Roblox engineers manage 24 edge data centers around the world, which run the game servers. When a user joins an experience, they’re matched to the nearest data center and the most appropriate instance within the center to minimize latency. We also manage two core data centers, which are much larger and run centralized services like the website, recommendation algorithms, safety filters, virtual economy, and publishing platform, which are necessary for the edge data centers to function. A global private network interconnects all the edge data centers to the core data centers, with edge data centers serving as a firewall to protect services running in the core data center.

In an ideal world, our creators should never have to think about capacity—the infrastructure should be invisible to them, working behind the scenes. When a creator publishes an experience to Roblox, our work is to support the capacity needed, no matter how many players show up. In the early days, we planned capacity once a year for the year or two ahead. But in recent years, successful experiences like Dress to Impress, Fisch, Dead Rails, and Grow a Garden have led us to rethink our framework for capacity planning.

In line with our value of taking the long view, we now predict capacity needs up to two years in advance, balancing user demand with efficient server utilization. Our planning cycle involves data center acquisition, server hardware refreshes, and physical networking, with new data centers like the one in Brazil being planned years ahead. The networking team also maintains “dark” capacity to ensure continuous operation despite issues like network cable cuts.

The capacity Roblox has today is based on predictions made two years ago, when we couldn’t have predicted experiences growing from unknown to huge popularity within weeks. Popular games like Dress to Impress and Grow a Garden, which helped double Roblox's peak concurrent player count from 13.9 million in April to 30.6 million in June 2025, didn't exist when these capacity predictions were made. For example, in March 2025, Dead Rails spiked to 1 million concurrent users, using all available CPU capacity.

Learning from these types of popularity spikes, we’ve moved to a more agile planning cycle. To consistently support record player counts on Roblox, engineering employs a rigorous weekly cycle of planning, testing, and capacity adjustments. Monday is dedicated to incident reviews, followed by capacity planning on Tuesday. Throughout the week, there’s continual chaos testing. Thursday focuses on reviewing capacity for any large updates our creators have told us to expect. On Friday, additional cloud resources are provisioned to ensure that the platform is prepared for peak weekend usage. Throughout the week, we continue to release entirely new features, and we do not lock the continuous deployment by all engineers.

Throttling is a very accepted concept in computer science. But this is the most misused and misunderstood lever of computer science. When new engineers join Roblox, their first solutions often include, “If we could just tell our creators to tweak this config or slow down their events…”. Veteran Roblox engineers then gently explain our value of respecting the community and that we don’t tell our creators what to do.

For example, most gaming systems have a simple solution for matchmaking when millions of players click play simultaneously. They throttle the joins, make players wait, or send them to random servers by skipping the matchmaking algorithm. At Roblox, we do the opposite. We redesigned our entire matchmaking systems for thundering herds of players. At peak, this system evaluates up to 4 billion possible join combinations per second. Years ago, we set the objective of 10 million joins in 10 seconds, and we continue to iterate toward that goal.

To avoid throttling due to capacity, we’re experimenting with cloud bursting as part of our transition to a cellular infrastructure, allowing for dynamic and compute-efficient scaling. This architecture handles peak demand by matching users to both on-premise and cloud edge data center cells. We’re working toward a fully automated bring-up and tear-down of cloud-based edge data centers that are fully abstracted for the matchmaking algorithm.

Another example is our text-filter system, which at peak handles 250,000 requests per second. That’s a large model inference running 250,000 tokens with constantly expanding context windows. And with more than 300 AI inference pipelines running in production, Roblox service owners invest a lot of time in finding the ideal mix of inference profiles between GPUs and CPUs. Even under peak loads, Roblox engineers respect the community by prioritizing creator freedom and user safety.

With our planning, we build up the capacity and algorithms to support the most exciting updates from creators. But we need to be sure these systems can hold up under even the largest peaks or single service outages. Information gathered from peak usage on more than 1,600 microservices helps identify services to further stress test.

True to our value of getting stuff done, every day we take a few of these services and constrain their capacity in production. We observe the attributes, then fix them before the weekend. We call this “test actual capacity on” (TACO) Tuesdays. Our reliability team also runs continuous capacity correctness (C3). Each engineering team uses a C3 dashboard to predict and manage their services’ CPU capacity. This enables service owners to continuously learn from the last peak to increase or decrease capacity for the next peak. We’ve also launched a system that traces call patterns in the core Roblox engine for new releases. This helps ensure that we’re better prepared during an update.

Even with all this preparation, we still occasionally run into scenarios where the unpredictable nature of traffic patterns could cause a single service or product flow to bring the platform down. For example, the 2 trillion event analytics pipeline could see 30% more traffic due to a popular update. This is where our resiliency mechanisms, such as adaptive concurrency control (ACC), circuit breaker, and shedding retries, kick in to protect the platform. This year, we also built a chaos-testing platform to strengthen our infrastructure’s resiliency and scalability by randomly injecting faults, exhausting resources, and randomly terminating processes in production.

We spend all week testing and preparing for these big weekend updates. But when the weekend comes, we still have work to do. Ahead of weekend updates, Roblox engineers collaborate to monitor upcoming changes and predict remaining capacity, provisioning additional cloud resources as needed to accommodate millions of extra players via virtual edge data centers.

On Friday, we decide whether we need to add extra capacity with cloud resources. This process gives a clear direction to our hybrid cloud team to bring up enough extra capacity to accommodate millions of additional players. At any point, our 24 physical edge data centers are running, but after all the testing, we might decide we need additional edge data centers. There’s no way to rack and stack servers in 12 hours, so we work with our cloud partners to build multiple virtual edge data centers. We test them on Friday, and then we’re ready for the weekend.

In the true spirit of taking responsibility, everyone, including our highest-level executives, take on-call rotations—even on weekends. The surge of millions of users on Saturday can often trigger hundreds of alerts. Teams preemptively resolve these alerts, enabling us to handle challenges during a big update or a platform-wide all-time high.

As Leonardo da Vinci is often credited with saying, “Learning never exhausts the mind.” Each peak has inspired us to learn and invent new techniques to make our infrastructure more dependable and invisible. Our creators publish or update, and through the magic of invisible infrastructure, tens of millions of users start enjoying an entirely new experience almost immediately. We are eternally grateful to our creators and users for challenging us to push the boundaries of computer science.


Read the original article

Comments

  • By nick_m 2025-06-295:283 reply

    Hey Roblox Engineers on here - a brilliant article, by the way - and I want to chip in here, I'm a senior engineer and understand how tech works all the way down. NAND gates, flip-flops and I/O schedulers, and networks? No problem.

    I have two young children, a boy and a girl. They both love playing Roblox, and I play along with them too, and their friends join in as well. Yes, they both always want more Robux, but let's look at this from a different perspective:

    They create their own worlds - often amazing, it's not like they can run out of LEGO pieces, their creativity is their only barrier. In COVID lockdown, they could carry on playing with their friends, despite not being physically together. Humans still monitor and care for the "game", yes, some bad actors might get through occasionally, but on the whole, it's a safe and well-controlled, fun place to be. I used the concept of a Roblox Avatar to gently explain to my children, that people online might not be all they pretend to be - after all, in some games, I'm a super weight-lifter with a six-pack, and I have wings too :-O We all laughed. It's already taught both my children some genuine life-lessons - working in a pizza shop and doing deliveries, earning money, deciding how to lay out their dream houses (and Theme Parks!), and so on - plus, the importance of locking the door to keep the "bad guys" out.

    All this, whilst having fun. Roblox is a force for good - if you pay the odd time for some credits, then so what, developers and us creatives also have to keep the roof over our heads.

    • By yurishimo 2025-06-2910:281 reply

      I don’t think anyone has anything against Roblox as a sandbox game, but your glossing over of the predatory monetization is the real issue most people have with the game.

      For tech literate parents, setting a spending limit is easy enough and we understand how these games are engineered to be addicting. But for the normies, they don’t know how it works and how to set appropriate limits. Personally, I think kids games with such heavy monetization should be outlawed. The pseudo-gambling aspects of these games is also highly problematic and we see this playing out now as sports gambling and crypto speculation are off the charts in our society with young adults.

      • By paulryanrogers 2025-06-2915:11

        It is amazing how gambling has so thoroughly infiltrated the US. When my older brother was young the cartoons could not have (or consist of) toy commercials, and often included at least a token moral lesson. Now my kids see games and YouTube channels that are filled with thinly veiled gambling tricks. MrBeast has shamelessly built an empire on it. (His charity washing not withstanding.)

    • By jokab 2025-06-295:55

      > NAND gates, flip-flops and I/O schedulers, and networks? No problem.

      That's kind of a weird flex.

    • By fRobloxinTheD 2025-06-2916:39

      [flagged]

  • By mrbluecoat 2025-06-292:22

    > This infrastructure must support thundering herd situations where more than 21 million users join a single experience simultaneously

    > we now predict capacity needs up to two years in advance

    Those two statements taken together are mind boggling

  • By kgwxd 2025-06-2523:213 reply

    I really love the core idea of Roblox, if only it weren't a scam targeting kid's birthday money.

    • By pkulak 2025-06-292:006 reply

      If kids pay 5-10 bucks every once in a while to support their entertainment and pay engineering salaries… is that the worst thing in the world? I’d rather my kids play Roblox than scroll Instagram. One of those is “free”.

      • By op00to 2025-06-292:345 reply

        I recently attended a class with my kid on internet safety taught by an attorney specializing in defending children accused of crimes, especially those involving computers.

        Roblox is like the #1 cesspool for bad shit going down with kids on the internet. Social media is bad as well, and my kids do neither. No, Roblox is not the end of the world, but there’s no real compelling reason to open up to it.

        They play with their friends on XBox Live, with voice chat limited to friends we personally add.

        • By Manuel_D 2025-06-293:551 reply

          The #1 by raw numbers, or normalized for userbase?

          • By margalabargala 2025-06-295:03

            Raw numbers, of course. There are places on the internet dedicated to crime, so if you normalize by userbase all the 100% criminal sites come out on top.

            Only some of Roblox is trying to get children addicted to gambling.

        • By UnlockedSecrets 2025-06-292:517 reply

          As someone with no experience of roblox but has heard this a few times, what exactly about it makes it 'such' a bad thing??

          • By Rotdhizon 2025-06-293:573 reply

            Roblox has a seemingly impossible to solve problem with child predators. Kids have the oddest talent of being able to stumble into the most unknown, niche games where these people lurk. Roblox now has voice chat as well and you have grown adults talking to teenagers in private, unsupervised and away from parental attention.

            Roblox currently has a massive event going on that was hit with a huge controversy. They picked over 1000 games to be advertised and someone accidentally (or not) picked a child romance themed game to put on the list. Without explicitly having sex, people have figured out how to game the system by letting players use emotes to do normal things but really they are intended to role play sexual activities (pushup emote while over another player laying flat on a bed for example). There's a specific term for these games but I can't recall it at the moment. It was called something like "condo games", the genre that refers to these romance games. The people who make these games do it intentionally, some of them make literally dozens/hundreds of these games, all aimed at children. Some of the game developers absolutely need to be investigated by law enforcement. Look up the video on youtube called "How One Developer RUINED Roblox's Biggest Event Overnight" and skip to around 2 minutes and he explains this in detail. This developer made children themed romance games that gained a quarter BILLION plays.

            As an adult that does occasionally enjoy some Roblox experiences, there's no problem for me. Stick to the front page, featured stuff and you're fine. The problem is solely with kids/teens going where adults don't realize and getting caught up with this predators.

            • By diggan 2025-06-2911:141 reply

              > you have grown adults talking to teenagers in private, unsupervised and away from parental attention

              Sounds like the problem is the lack of parental attention, and like many things, that isn't a problem with a technological solution, but requires parents to actually pay attention to what their children is doing.

              • By op00to 2025-06-2920:30

                It’s important to be involved. It’s also important to give kids safe freedom with guardrails. It’s a balancing act.

            • By Manuel_D 2025-06-294:041 reply

              If tens of million of people are using a service, statistically at least a few of them are going to be bad actors. The stuff you're describing doesn't really seem much different that any popular internet spaces in the 2000s. People were "cybering" in World of Warcraft. 4chan was raiding habbo hotel and club penguin with Nazi memes. Kids were chatting with strangers on AIM and ventrilo. If anything, these services probably had considerably worse moderation given the language processing rooms we have today.

              I'm not seeing any evidence that the bad actors are a proportionally larger problem, or just the fact that more people on the internet. E.g a city with 1 murder out of 100k and another with 100 murders out of 10M are just as safe.

              • By sothatsit 2025-06-294:272 reply

                Roblox specifically markets itself to children, and 40% of its playerbase is 13 years old or younger. Therefore, it is reasonable to hold it to a higher standard than other games.

                • By Dilettante_ 2025-06-295:141 reply

                  IIuc, the original point/implication of this thread of conversation was more like "there's an unusally high concentration of child predation on Roblox", which, while not invalidating it, is a considerably different problem than "there is more child predation than there ought to be on Roblox".

                  The former implies that rblx has some attributes that are conducive to child predation, which would be worth teasing apart out of scientific interest, while the latter is a very general problem, as (I dare take this as self-evident) any place that has greater than 0 child predators has more than there ought to be.

                  • By sothatsit 2025-06-296:37

                    I was just responding to this point:

                    > The stuff you're describing doesn't really seem much different that any popular internet spaces in the 2000s

                    I think there is a big difference to some of the examples they gave, because of the uniquely young age demographics on Roblox. The only example that seems comparable was Club Penguin.

                    Now, I agree that there is interest in teasing out whether there are problems with Roblox specifically, or if it is just a problem with having an online space with such a high concentration of kids in general. But that high concentration of kids does make it much more of a concern either way.

                • By Manuel_D 2025-06-294:413 reply

                  Do Minecraft, Fortnite, and countless other games not also market themselves to children?

                  • By Rastonbury 2025-06-295:29

                    In those games its more difficult to create a private hangout space or "GTA for kids". Haven't heard of the weird romance thing, but seen my nephew playing a roblox game where the goal is kill as many people wheelchairs as you can. I saw I guess the humor in it because I played San Andreas when I was his age but him mom might have been shocked. Those other games are much more restricted in the possibilities, moderation seems impossible

                  • By sothatsit 2025-06-296:09

                    Roblox has a very young playerbase, even when compared to Minecraft and Fortnite. Roblox is also unique in the sheer quantity of offences that happen on their platform, and that is why they are often singled out.

                    But this can be a problem wherever kids are online. Discord also has huge problems with child predators. And any platform that caters to children should be held to very high standards of child safety.

                    The games that I think shouldn't be held to such a high standard are games like World of Warcraft. That game is not targeted at children, has far fewer children players, and therefore it is unreasonable to hold them to as high of a standard as Roblox. (Although they do still have some responsibility to make sure their platform is safe.)

            • By taneq 2025-06-294:28

              > child romance themed game

              Well there’s four words I never expected, not wanted, to see arranged like that. :S

          • By op00to 2025-06-2920:29

            I think it’s easy for people to contact each other in game and this to get kids to move over to a discord away from moderation. That’s the basic MO the instructor said how “grooming” in Roblox works.

          • By superb_dev 2025-06-293:281 reply

            The YouTube channel “People Make Games” had a really great video on the exploitation going on

            • By bentcorner 2025-06-294:15

              IIRC that video is about how young content creators get exploited, which is indeed a bad thing but not exactly what GP is asking about (young players getting predated upon)

          • By jovial_cavalier 2025-06-2916:27

            Because the company knows that there are child predators on their site, and not only do they not care, but that is basically their tacit business model.

            https://hindenburgresearch.com/roblox/

            edit: some quotes from the article, attributed to concerned employees -

            “You’re supposed to make sure that your users are safe and but then the downside is that, if you’re limiting users’ engagement, it’s hurting your metrics. It’s hurting the [daily] active users, the time spent on the platform, and in a lot of cases, the leadership doesn’t want that.”

            “You have to make a make a decision, right? You can keep your players safe, but then it would be less of them on the platform. Or you just let them do what they want to do. And then the numbers all look good and investors will be happy.”

          • By charcircuit 2025-06-293:081 reply

            13 years old and above can talk and dm with any other account also 13 years old and above. Unless a parent account disables it.

            • By ocdtrekkie 2025-06-294:111 reply

              I once overheard our nine year old explaining to a friend how to lie about their birthdate to create a YouTube account. Age checks are not meaningfully a working thing.

              • By charcircuit 2025-06-294:36

                Parent account can be used for any age and can disable chat.

          • By cokeandpepsi 2025-06-293:46

            [dead]

          • By loloquwowndueo 2025-06-292:56

            [flagged]

        • By landl0rd 2025-06-293:46

          This is because roblox is the #1 place for kids to interact and play and such. The creeps and weirdos and predators are going to follow wherever they go.

        • By lokar 2025-06-293:271 reply

          AIUI, they let stranger adults give kids in game currency.

          What do you think that leads to?

          • By Manuel_D 2025-06-294:09

            The same was true in World of Warcraft. What did that lead to?

        • By jjani 2025-06-293:491 reply

          I'm sure it's the #1 platform involved in "defending children accused of crimes involving online platforms" in whatever US state that attorney operates in. But even that sounds better than the lifelong dopamine system disabilities that the likes of TikTok causes. Every other adult I talk to about social media admits having had to very deliberately uninstall/disable TikTok, else they just couldn't detach themselves from the screen. Now imagine what it does to kids. And how big of a % of users ends up like this compared to the % of all Roblox users traumatized by it.

          Before anyone comes in accusing me of unfairly pointing fingers when Meta (Instagram reels) and Google (Youtube shorts) are just as bad, truth is the latter two just haven't managed to make their algorithms half as addictive. I'm sure they'd love to, there's nothing inherently better about them, but so far they haven't cracked it.

          • By op00to 2025-06-2920:33

            They can both be horrible! My kids do none of any of that. Works well for us!

      • By Aurornis 2025-06-292:372 reply

        > If kids pay 5-10 bucks every once in a while to support their entertainment and pay engineering salaries

        If this was a game that had a subscription fee or where you had to buy the new version every year, that might be true.

        These games don’t work like that, though. They rely on maximizing the take from each individual player. They’re designed so that the sky is the limit. You pay 5-10 bucks here, then you’re enticed to pay 5-10 bucks there, then you’re scrounging or begging for another 5-10 bucks for the next thing.

        • By moregrist 2025-06-293:331 reply

          The amount a child can spend on Roblox is entirely up to the parents.

          Children, especially pre-teens, don’t have access to credit or money without their parents consent, and a pretty easy way to limit the amount of money your child can spent on Roblox is to just make them have to ask you to buy the in-game currency.

          There are problems with the game but the amount of money going into it is something a parent can and should 100% control.

          • By ocdtrekkie 2025-06-294:131 reply

            Clearly you aren't aware how many kids steal their parents' credit cards to buy Robux. It's a very common thing.

            • By Dilettante_ 2025-06-295:21

              How common is it, and how specific to Roblox purchases? What about 19$ Fortnite cards, or whatever currency those "Tiktok NPCs" that make noises at children are paid in?

        • By landl0rd 2025-06-293:49

          If toy stores had a subscription fee or where you had to buy a new access pass every year, that would be fine.

          They don't work like that, though. They rely on maximizing the take from each individual child. They're designed so that the sky is the limit. You pay 5-10 bucks here, then you're enticed to pay 5-10 bucks for another toy, then you're scrounging or begging for another 5-10 bucks for the next toy.

      • By Spooky23 2025-06-292:50

        Roblox is awful with lots of weird crap and opportunity for abuse.

        Fortnite is way better and safer.

      • By bowsamic 2025-06-295:45

        There are better alternatives to both of those that we should guide our children towards rather than just throwing up our hands and giving them their screens

      • By jmyeet 2025-06-292:113 reply

        Roblox engages the exact same psychological manipulation that gambling does.

        Would you let your children go nuts at a casino? Or online with sports betting or crypto slots?

        The problem isn't the $5-10. It's the psychological manipulation.

        • By ryandrake 2025-06-292:232 reply

          As a parent, if you sit down with your kid and curate (or at least actively check/approve) what they are playing on Roblox, it’s fine. Mine just uses it to socialize with her real life friends, who are also all there. She sticks to the wholesome stuff, it might as well be Minecraft.

          • By krapht 2025-06-292:251 reply

            Sure, but at that point, why not just Minecraft then?

            • By ryandrake 2025-06-292:26

              She does! A lot of her friends left Minecraft though, but we still play together.

          • By loloquwowndueo 2025-06-292:57

            The problem is, most parents don’t.

        • By Dilettante_ 2025-06-295:25

          I actually don't know very much about Roblox. I was under the impression that it was more like a storefront sort of thing, with 3rd party games being hosted there? Kind of like one of those 'old' flash game portals, except you buy the games and ingame items.

          Where do the gambling tricks come in?

        • By gehwartzen 2025-06-293:33

          It sounds more like the same psychological manipulation that underpins the IRL world (at least the US) in the form of capitalism. Engineered towards adults to buy as much useless crap as possible.

          Teach kids that it’s the same kind of manipulation as in their games (and how to spot it) and perhaps they will be better equipped to not get fucked by adult life.

      • By loloquwowndueo 2025-06-292:56

        [flagged]

    • By MOARDONGZPLZ 2025-06-292:24

      imo it’s so bad that employees might as well be working at Meta. It’s an ethical decision. Maybe their ethical framework supports it, but mine doesn’t.

    • By gavinray 2025-06-299:32

      My mom paid my $5/mo Runescape subscription when I was a child and those are some of the fondest memories I have.

HackerNews