Have you ever called into a bank or brokerage? Most ask "security questions", often ones that you can't even choose, like your address or how many accounts you have with them. It's arguably far worse than speaking your password into the phone.

>2. Is there a way for phone call man in the middlers to get that info without wasting a ton of time listening to calls?

Automated speech recognition has been around for decades. Even before that signals intelligence agencies have shown that widespread wiretapping/eavesdropping is possible and effective.