http://www.cs.vu.nl/~herbertb/download/papers/anc_ndss17.pdf
XI. CONCLUSIONS
In this paper, we described how ASLR is fundamentally insecure on modern architectures. Our attack relies on the interplay between the MMU and the caches during virtual to physical address translation—core hardware behavior that is central to efficient code execution on modern CPUs. The underlying problem is that the complex nature of modern microarchitectures allows attackers with knowledge of the architecture to craft a carefully chosen series of memory accesses which manifest timing differences that disclose what memory is accessed where and to infer all the bits that make up the address. Unfortunately, these timing differences are fundamental and reflect the way caches optimize accesses in the memory hierarchy. The conclusion is that such caching behavior and strong address space randomization are mutually exclusive. Because of the importance of the caching hierarchy for the overall system performance, all fixes are likely to be too costly to be practical. Moreover, even if mitigations are possible in hardware, such as separate cache for page tables, the problems may well resurface in software. We hence recommend ASLR to no longer be trusted as a first line of defense against memory error attacks and for future defenses not to rely on it as a pivotal building block.
So, outside of memory unsafety, is there another threat profile where ASLR gains you something?
It requires knowing something about the topic, and maybe some of those who don't would have to look up and read about 1, 2, 5, 10 different topics that show up in the article that they didn't know about before.