My wife and I do this, but we still got scammed. My wife's debit card got skimmed, and there wasn't much money in that account but it gave the thief enough info to do a social engineering attack at the bank (USAA in this case). Took several calls before they found a cooperative representative, but they managed to convince them to reset the password, tell them the username, reset the PIN, and disable the two factor token (which is a Symantec app that runs on a smartphone). When the bank fails this badly, nothing can really protect you.
Sure was an awkward call with the USAA security department after they worked out the exact timeline and actions of the attack. They didn't really have much to say in their own defense. Cost them about five grand in the end. The really annoying part is that now if my wife needs to call them for anything on the phone they make her jump through many hoops to identify herself. E.g. she has to answer the credit bureau-based questions every time, plus a PIN, plus a phone password. We may move banks just so we're not stuck with an online-only relationship because of this experience.
