Yes it matters on MS-DOS like OS design, like some embedded deployments and that is about it.

It is even impossible to guarantee a process doesn't get rescheduled into another CPU with the performance impact it entails, unless the process explicitly sets its CPU affinity.