The "bug" with the OOM killer that i implied is that what you describe
does not happen. Which is not surprising because disk cache thrashing is normal mode of operation for serving big files to the network. An OOM killer acting on that alone would be problematic, but without swap, that's where the slowdown will happen for other workloads, too.
Its less a bug but an understood problem, and there aren't any good solutions around yet.