You are allowed down time. If people don't respect/understand that, it's their problem, no yours.

I don't answer the phone when I eat or after 10pm. Same thing with text messages/emails. "Do not disturb" mode is there for a reason.