Actually, you often do. That's the point really. You should decide if it's an operation you might want to retry, you might also want to just flat out error and do nothing more, maybe you want to provide degraded functionality, like provide some default answer.
I think errors as values cause you to always think about this, which makes you handle errors in a more sensible way, instead of just bubbling up. Sure, 90% of situations you will bubble up, but in my opinion it's still worth it.
