I wouldn't say it's that drastic. Also, SELinux can give you a false sense of security. It's best to harden the system overall instead of relying on one security feature (however good it might be).