But remove the CPE router. Where do you think that fiber goes? To "the internet"? It's going to yet another box owned and managed by your ISP. And from there, probably yet another box owned and managed by your ISP. And then another black box maybe owned by yet another ISP, and then another black box owned by maybe yet another ISP. Each one of these could let an attacker come between your network and "the internet". You have no control over them. You don't patch them, you don't configure them, you have no say over the services running on them. If they're compromised, you likely wouldn't know.

The CPE just moves the first black box inside your home, but there's always some ISP black box you're connecting to. Even if you're a top tier network, it's not like you control every box between you and every other site you want to go to. You're going to eventually have some handoff at some peering location, and once again your traffic goes to a box you don't control just waiting for an attacker to manipulate and mess with your traffic.