It seems normal phones are able to do it, though. At least it seems normal phones suffer less latency problem.
In a way, simplicity in technology often means better performance.
Google is well-aware of this, thus Fuchsia.
SeL4 would make a good base for such a device.