If performance is a concern then you should still avoid std::set though by default. Logarithmic worst case when it's just always slow isn't really useful.
There may be a benchmark out there where std::set can beat std::unordered_set, but you'll be really hard pressed to find it