No port can be used by any other computer or server (since all are required by Switch), and any port listening on your Switch will be automatically reachable by anyone which is a potential security risk. Does a Switch work if you are IPv6-only? Cause if it'd work with IPv6, a mitigation could be to only use IPv6. It wouldn't solve the security issue, but it would solve the problem of all (external IPv4) ports being used.