Sure, but those same speculative execution attacks also trivially defeat ASLR. And practically speaking, memory safety in formally proved software tends to be pretty bullet proof.
So, outside of memory unsafety, is there another threat profile where ASLR gains you something?