So, even though most people cannot understand some southern dialects when they hear it, they can fluently understand other dialects without explicitly learning to do so when it's written down. Even when you explicitly write down exactly what you say in the dialects (like the Cantonese version of Wikipedia pages), people can still understand it, without much difficulties. And yes, there're some words used differently from standard Chinese, but the difference is not much bigger than the difference between UK and US English.

For that reason, we define them as dialects not languages. We can "speak" other dialects but we can certainly "read" and understand them.

If you find that definition unsatisfying, sure you can call them different languages. We just call them dialects and that's not going to change.