If the NFT does actually reference an IPFS CID, that amounts to a hash of the image, so that particular issue isn't there.
It's just kind of nuts to talk about "everyone stopped pinning my NFT" as being a "rugpull"- anyone can pin a piece of content in IPFS, including anyone with an interest in the NFT existing, such as the owner. So the only thing you have to do if an NFT falls off the IPFS network is just to take your copy and pay someone else to pin it. It's only a "rugpull" if you, you know, forgot to keep a copy of it anywhere.