That's between $572,000 (500 instances, 30 days) - $2,863,800 (2500 instances, 30 days), per month at current prices, and seems like it's only for one aspect of their infrastructure.
That seems .... excessive? Is that a typical spend with a game server system like this? That does seem to suggest that once this becomes less than profitable, it's all going away ...
Those costs can easily be cut in half with reservations. Its likely there's a lower-bound of the number of reserved instances they use as a baseline performance guarantee, then they use on-demand to scale above that.
No one at their scale actually pays the list price.
Basic math: There are ~100 players in each game. At a 30 tickrate, that's 3000 RPS per game minimum. Each of those requests likely involves a number of 3D math calculations, including hit detection, collision detection, real-time cheat detection, etc. Those updates then need to batch back to the players at 30 tick. All of this needs to happen with as little eventual consistency as possible; a difference of milliseconds degrades the player's confidence that the server is correctly calculating what is happening in-game.
Point being; multiplayer game programming is an entirely different beast than normal web programming. The same rules don't apply. Its an n^2 problem where every additional user in one "lobby" actually increases resource utilization exponentially because you need to update every other user's game-state with the actions of that new player. Additionally, Battle Royale style games are the most demanding multiplayer games ever created. Games like WoW have way more players per realm, but servers only need to worry about the interactions of a select few in your surrounding area, and it doesn't have as stringent real-time requirements. Games like CoD only load in 5-20 players.
I'd like to draw your attention to Planetside 2. A pretty recent FPS that was run a lot like WoW, with 3 'continents' and complete seamlessness on those continents. Firefights where 100 people were actively engaging each other were an almost constant experience back when I played it. I think there was a cap of 1000 players per continent. Those players could gather wherever they wanted. I'm pretty sure I was in some 300 player battles, and nothing but lag would prevent all 1000 people on a continent from going to the same base.
It has to be said that in the big battles (at around a 100) lag became an issue. I'm not sure whether this was clientside, serverside or both.
(There is also eve-online where recently there was a 'fight' with 5000 players in the same location, but that is a different beast from FPSes, and the main logic of that game is single-threaded)
We generally didn't go higher than 10Hz since you'd start saturating lower end connections and any well written game will be good at reconciling state via dead-reckoning. You have to handle 100ms+ spikes anyway so it doesn't make sense to run the network super-fast.
As for your last comment it's wrong, a game like Battlefield is probably more demanding compare to a battle royale type of game, a FPS like Battlefield runs at 60hz and is using a lot of physics everywhere at fast pace, battle royal games don't have that much action going on.
And we do indeed take advantage of reserved instances. We’re also looking at other options along these lines.
Blizzard is absolutely stringent about their real time requirements for WoW. It's crucial for PvP, Raids, etc.
Fortnite doesn't need to broadcast nitty player info for players on the other side of the map to you, either, I would think. That should definitely be optimized away
> actually increases resource utilization exponentially
I think you'll have to choose there.
I've always wondered what AWS rates do large users receive from negotiations?
Quadratically actually.
Exponential would mean each user doubles the amount of calculations, for instance.
Auto-scaling instances on EC2 only makes sense if you don't have the in-house skills to troubleshoot/manage hardware servers or if the traffic is unpredictable and the bursts are infrequent (e.g. once or twice a week or just busy for a short time period of the year).
There's a difference in the level of respect each company gives for its customers. I play PUBG a lot, but I want to see Epic win in the long run.
They don't do QA, and their 'deploy process' takes everything down for hours. It's embarrassing.
I mean, what did you expect?
two very different games fundamentally
However, as for:
Why does it need to be a zero sum game with one winner
A "battle royale" if you will.
When Epic started the UE4 project, they promised to take care of us gnu/linux users. I bought in the second I could. Immediately they started ignoring us. The first year and a half or so most of us Linux people were using a community fork because Epic refused to merge changes. We banded together and we're forced out of their irc and had to form another channel. I tried to be understanding, at first thinking they were low on resources. As time went on and games like Paragon (which I payed $20 for at release and which has now been abandoned, and I'm still waiting for my refund for over two weeks), and Epic started showing off how well things were going, they still basically abandoned us. There is still no marketplace/launcher on Linux, so to retrieve my $500 worth of assets I have to use a windows computer. Major bugs persist in all branches, and not just for the native editor... games like pubg would love to ship for Linux if the crosscompile tool gain wasn't an exercise in cryptic puzzle solving (which is why so many UE4 games are windows only not by choice. All pleas for more resources and love on the forums are met with comments about marketshare and how unworthy Linux is of their time and resources.
They promised us all this love and then after many of us spent lots of money on them they just ignored us. I'm thankful for the attempt for a native Linux editor, but crashes and uncompilable projects have essentially halted dev for me to the point I'm having to consider godot and blender even though they aren't nearly at feature parity, and due to epics licensing all that money on assets is wasted since those assets can only be used in UE4...
I love UE4 when it works. Blueprints, the animation and rigging system, the camera system are all wonderful to work with. I want to use it... but I'm feeling increasingly taken advantage of by Epic.
Tim, if you're reading this, how about a post mortem of the abandonware that is the Linux editor?
This clearly shows how complex a system is needed that has to handle 3.4 million concurrent, connected users. I think the connected part compounds any scale problems you have since it is implied they are connected to each other.
The biggest problem here is that it largely used to be a distributed system. You used to just be able run your own dedicated server on whatever provider you liked. The dev would just run a single server list. Now many game developers have decided they're the only ones who get to run servers - primarily because they can charge more for micro transactions and private servers this way as far as I can tell.
It really hurts games - PUBG is the best example I've seen - constant lag issues, complete lack of server side checks for things like shooting through the ground (because hey, that costs CPU and every additional cloud server they need means less profit), etc. It's basically made the game unplayable.
Game developers are unfortunately stuck between immersion in their games and the rage that leaves players with when technical issues occur. The more immersed your players are, the more rage they'll experience when your game crashes or lags at the wrong time.
It also hurts the game's image if many servers spam players with "Donate $20/mo for the MASTER rank and get exclusive weapons that deal twice the damage of normal weapons!"
I haven't played in a while but I remember playing Minecraft when I was still in school and servers were filled with this- many would let you donate hundreds for admin/OP/whatever, many would restrict core parts of the game to "donating" members (such as mining certain ores, going into certain areas, etc). Eventually it got so bad that Mojang actually had to step in and (vaguely) threaten to sue any server owner who does this - http://andrewtian.com/mojang-threatens-lawyers-against-pay-t...
Also, admins that can ban players for any reason, who are picked by the server owners (and not the devs), can result in backlash if someone influential is banned.
Community/matchmaking servers don't distribute ("run your own server") too well. If you only ever want to play with and against people you know, that's fine, but a lot of people don't want that: some want to play on randomized (or ranked, if the game supports it, across a large number of players; not just the ones in the potentially-altered rankings on a single person's server) teams. Others (more commonly) want to play against new and different teams.
Unless your graph of social connections willing to agree to use a given server is big enough and the right shape, hosting your own matchmaking/community services just results in isolated islands without a vibrant overarching community.
Put another way: the separate-server approach of e.g. TF2 works fine for some, but a lot (judging by play counts) of other folks prefer games with a centralized community.
To be clear, I think it's fine to have the option of a separate community server. But if a game developer already has the centralized infrastructure, is making tons of money off of it, and spending tremendous resources keeping it intact (as Epic apparently is), it's a pretty tough sell to say "oh and you should also spend the resources to make a hosted option for this".
"Take the complex thing designed for centralized, single-instance internal use with a team of support staff, put it in a box, and let users run it on their own hosts" is far from a simple proposition.
I'm unsure if any companies doing centralized community servers/community federation also have a large portion of their actual game servers not hosted by the company; I'd love some examples of that phenomenon. The examples of Minecraft and TF2 come to mind as doing some subparts of that, but those games only have limited matchmaking capabilities.
I believe its more about standardizing the user experience. Now that enough games have dedicated servers with a standard expectation of latency your game looks poor if your user experience is beholden to community servers with selfish moderation and varying stability.
The added benefit of centralized server hosting and control is exactly what you point out though.
I'll never accept donations again for a game I've made available for free from the amount of problems someone has caused me after they gave me $5.
It's one of those things, though, that reveals the human inclination to remember negative events more even among a sea of positive events... Frequently as a game developer you will get loads of positive and helpful feedback but the small minority that want to shit on you for no good reason stick in the front of your mind.
Anecdotally, I'm seeing more and more peers leave the industry as a result of being disgusted and worn down by the constant, unending abuse that game developers receive from users.
0: https://medium.com/@morganjaffit/the-cost-of-doing-business-...
The games as service part prevents players from keeping proper emotional distance to games by not allowing them to have closure; to look at the game, say you've beaten it, and move on. Every game needs some kind of grind or persistence beyond the actual play, through online connectivity and often actual player competition. So you kind of prime players through negative play models very often, especially when in-game purchases exist.
What's worse instead of an end to a game, devs are forced into chasing the mythical idea of balance instead. This actually sets players up into warring factions because balance is often zero sum; my warrior gets buffed at the expense of your paladin. There's no existing balance state, ever; you can look at Overwatch to see it just ending up into random changes to heroes that please and incense players in equal measure.
These things combine to make players strung out over time; their nerves are jangled through constant grinding, death (screw roguelikes!), or competition, and then you end up with developer as God who made decide to alter everything to please a particular player faction, or even no one at all but themselves. You make your players invest a hundred hours in your game, and they aren't going to be disinterested enough to take changes in perspective; you don't give them the chance to be, they have to be logged on every day doing busywork either through ranking or grinding.
I think this explains why modern gamers can be so enraged. It's more than personality; a lot of the structure of many modern games can really create this. The games make players too invested in them too easily, over too long a time.
A whole lot better than spoon feeding customers bullshit for weeks while hamstringing your product rather than investing in it (looks at EA, mumbles about SimCity).
The success or failure of the recent Sim City game was hardly EA's number one concern. Fortnite, however, is probably extremely important to the folks at Epic.
I'm not defending EA here, but it was definitely in Epic's best interests not to piss off Fortnite's player base given that there are some other, ahem, similar titles on the market right now.
Epic is a pretty big operation too.
2. they actually care
3. they enjoy what they are doing
One thing I really miss working solo is working with a team of smart folks to solve a complex problem together. It's so damned fun.
(I work at Epic on Team Online, aka the team that this whole PM is about.)
They talk a lot about reducing operating complexity and scaling their infrastructure, I wonder what the cost of their current infrastructure + the staff to maintain it might be vs the managed solutions that cloud providers offer now.
For example, using cloud datastore or spanner or big table as a persistent layer, these managed services can definitely scale to the current need and I've seen them go much higher as well.
For logs ingestion and analysis, big query can be a very powerful tool as well, and with streaming inserts that data can be queried in near real time. For things that are less urgent, batch queries. For other things dataflow can help with streaming workloads.
I think one of the problems they alluded to though was that at the moment they're on a single provider, and what they're looking for is a multi cloud strategy which totally makes sense. A lot of the above products create some kind of locking, with some exceptions, like using hbase as an interface to big table or beam as an interface to dataflow. Though I don't know what the other providers offer that may have these same interfaces.
Another option is kubernetes, which I believe all providers are pretty strongly embracing. Having most of the supporting infrastructure be brought up with a few kubectl commands could help them scale across several cloud providers quickly.
Usually there is a cost/scale threshold with managed providers where it is cheaper do DIY than to pay thousands upon thousands per month for say log ingestion.
The other problem is that most of these things aren't easy to migrate to. AWS RDS is much easier because its just managed whatever you're already using. But cloudspanner? DynamoDB? You have to completely re-architect your application. Then you have to move your application, and data, to this new system...without massive outages. It's a lot of work and a lot of cost. So until things go HORRIBLY sideways, most companies don't have the spare time/money.
Been there, tried that.
At my current company we have access to AWS support. I'm not high enough on the ladder to know the specifics (separate contract, size threshold?), but when we've had issues in Aurora we've had personalized support. I have no doubt if you're scaling to thousands of instances you would have personalized support, if for no other reason that a cloud provider would not want to lose such a huge customer.
You think GCP offers better support on spanner et al when customer is having performance problems? In this case probably yes, because an Epic sized monthly spend is highly effective at escalating through support.
It takes low effort to find <cloud persistence horror story> around here so we know cloud is not a special magic that is immune from integration performance problems. But the economic incentives are meaningfully different and especially so at runtime.
Disclaimer: Riot Games Engineer :-)
I've always thought that XMPP would be useful for games, just surprised to hear that people are actually doing it.
it looks like a great business model to take something open source and hard to use and make a proprietary version thats easy to install and use
https://en.wikipedia.org/wiki/Signal_Protocol https://en.wikipedia.org/wiki/WhatsApp https://www.cisco.com/c/en/us/about/corporate-strategy-offic...
You couldn't see friends lists at all during that time period. So you couldn't queue up in a friends / people you knew at all in a match, the only options were either playing solo or using a "filled" team with random players.
I've been playing fortnite as one of the early 60k concurrent users all the way to the 3.4M, so its been interesting seeing their load / server issues over time and then reading this (Granted, I don't understand everything discussed in their blog). They've done a outstanding job handling their growing traffic.
One thing I've noticed with Fortnite, compared to PUBG or other MMOs, is how large their patch updates are. Its usually several GB large, and it comes fairly frequently about once a week.
Most notably, whenever you wanted to add someone to your party. You had to do the following.
1. You Send user friend invite
2. User would have to accept invite
3. User would have to disconnect to refresh your friends list (due to friend-service issue mentioned in blog)
4. User would relog back on (took approximately 3 mins)
5. You could then see them on friends list
6. Send party invite
[1] Getting ~16 FPS on medium settings, with the high-end late 2016 MBP 15".
Note: I did tweak some minor settings like AA to get better performance.
- Followed by removing Nginx + Memcached couple altogether out of equation.
I think this is where Stacktical helps with proactively detecting performance regressions at the CI level, before they hit production: https://stacktical.com
Disclaimer: I am Stacktical's CTO
I wonder whether Epic can solve its problems by rearchitecting more into a CQRS driven system with event sourcing: store events in a more write optimized DB (e.g. Cassandra) and then process the events for fast reads through whatever is required for the usecases. Maybe they touched the limits of MongoDB to handle both, reads and writes at their scale.
edit: nevermind, it is available again - weird