This is kind of the wrong question. You can trivially run openstack ontop of mesos and vice versa.
Mesos is bigger on services, openstack is bigger on virtual machines. They are different solutions for overlapping, but fundamentally different problems.
Here is an example of Ebay running mesos ontop of openstack:
http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-s...