"My logs and firewall are less cluttered" is not at all the correct metric to measure the security of your box.
IP address spoofing is a thing. Blocking CIDR ranges might protect you from low-effort, drive-by botnets that constantly scan the entire internet (which all should be completely mitigated by using certificate based auth anyway), but blocking based on IP address is absolutely not an effective control against a determined hacker.
You must consider your threat model. For your personal instance that you host hobby things on, you probably won't be targeted via IP spoofing. For any type of company, you should not be relying on CIDR blocking as part of your security layers. CIDR blocking is only effective at reducing the clutter of your logs, which is a convenience, not a security control. The real security control is using proper auth methods, which are so easy to do at this point that it's ridiculous for even a hobbyist to not do them.