I'm currently working on a software project where I need to query datasets that could be very large (maybe hundreds of thousands per single context), and then do some computations on the results. It's basically, find some sort of "median" from the set, but it could be a bit more complex than that, like find the smallest, most common value. My impression is that most modern databases should be able to handle queries like this with some built-in mechamism. However, one of the concerns is that, because the datasets could be very large, queries would end up taking very long. The data being queried is also highly dynamic, so caching maybe a little tricky.
I'm pretty sure this isn't something unique to this project, but I'm interested to know how other practitioners address this kind of situation. Also, to note, while I'm asking this in general terms, it'd be interesting to know how MongoDB users in particular handle this.
