It happened to Jack Dorsey. And attacks tend to become easier over time. Any employee of an at&t store could do it to you right now.
The reason we know Dorsey was the victim of a sim swap attack is probably that he's important enough that when he was hacked he couldn't be dismissed with the "You probably messed up and leaked your password" dismissal.