To add on to your point,
even if you allow evil.com to only access evil.com and not any subdomains, your browser is still vulnerable because of short TTLs on DNS resolution.
evil.com can set a short DNS TTL, and after you access it, it can rebind its address to 127.0.0.1. Then subsequent requests to evil.com go to localhost (e.g. fetch("evil.com", ...) on evil.com will go to 127.0.0.1 if the DNS rebound successfully).
Caching a website's IP on first use doesn't help, either, because it breaks long sessions on websites that use DNS rebinding for legitimate purposes (load balancing, fallover).
The only real way to fix this is for the local webserver to check the Host header on the HTTP request... or look at IP addresses. But building a global registry of IP addresses is hard, so we're stuck with trusting application developers (and malware writers) who run servers on localhost to use good security practices.