I think the main problem is that people can be affected that aren't even using it, which is why it is such a big problem. You can't really hide it's GrapheneOS either, even just by virtue of the features available on the device, you'll be able to deduce what it is.

I understand the idea behind it but it simply isn't realistic to provide and can put people in danger - the very thing it's meant to prevent.