This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.