The downsides are greater too. It's one thing to miss out on TV or drinks with friends; quite another to miss out on the one evening you might get to sit with your wife and have dinner, or not see your kids for a week.
It doesn't matter how rich or respected my side-project startup would make me, I wouldn't be able to get that time back.