What we have considered is not doing any synchronous writes and just queuing it up and handling it async. There's definitely some questions about whether our current approach will scale 10x but it should be fine for the next 2-3x.
Just to shed a bit more light, we break things up into figuring out which ad to show and then handling when that ad is actually seen. The second part is mostly async already but the first part is the harder part. You want to choose the best ad for the content, the geographic targeting needs to match, and the ad campaign has to have budget left on the hour/day/total. If we only checked the budget every 5 minutes, that would be a problem. At some point, multiple servers need to know that there's still budget on a campaign and this requires some amount of synchronization.
This is a common pattern, however. The easiest thing to set up is to do a customer database query for every request, while the far more efficient option, in many, many cases, is to cache the 1000 most-likely queries and serve up static or pre-rendered data. Not cache like one update per day, but perhaps one update per minute. Only you and your team can decide whether it would cost more to serve up a few free ads (over a customer's budget) because the cache was a minute or two old, vs doubling/tripling your infrastructure costs in order to make the system more precise. Sounds like you've got a good roadmap for making that decision.
So hopefully I'm not that totally rude person, no-shade intended pointing out that if for some random reason you haven't seen ClickHouse you should check it out.
Solves problems that I parsed from your blog post and originally built for that use case.
100 recs/s for an ad network is super duper mega tiny so I hope you're successful and get bigger!
