If the government froze my bank account so I couldn't pay my phone bill, I wouldn't be free to speak to you right now.
I don't have the right to speak in the Queen's bedroom in Buckingham Palace, either.
Neither of those things are significant limitations on the freedom of speech.