The problem is that then some people opt out. If enough people do that, it's no longer financially viable to have a fire service.
> I absolutely reject the idea of a "social contract"
Which is fine, but then don't live in the society. As a society, we have decided what is important for the benefit of the society; fire departments, law enforcement, schools, roads, etc. We then attempt to gather money from everyone in that society to pay for those things, because that's the only way it's realistic. Not everyone in the society agrees to all the details, but (in theory) the majority of people agree to them.
It's not all that different than living in an apartment with a few other people. If most of the people agree that everyone has to split the electricity bill, but one person says they don't want to... then it's reasonable to have that person leave, and replace them with someone that is ok with the rules set by the group.