In that time I've had a single incident (earlier this year) where spammers got it and sent 20k spam mails in less that 24h mails. Luckily I noticed this and stopped it pretty quickly.
I had mails to some destinations bounce for around a week after that.
Beyond that incident, it's very rare to have trouble delivering mail - I think maybe once in the past 5 years there was a company who I had trouble with.
Having said all that, the spam incident this year highlighted how fragile hosting your own mail server can be if things go wrong, and I felt really bad about the spam mails sent from my server - continuing to run it almost feels like a liability, and when time permits I plan to move our mailboxes to O365