Perhaps not perfect but in consideration of some of the barriers other OEMs may posses with a security device I think the situation reads as 'being better than the alternatives'. Pixels are also an reference device by Google for Google so that is another incentive for them to suck less.
I'm quite sure the hardware is fine. From what I recall google phones actually have some of the best security hardware. It's one of the reasons GrapheneOS chose Nexus/Pixel phones as official base as well as the long support for those devices.
What is the alternative? Running some chinese hardware and chips? I guess iPhones would work but you have no real softwarecontrol over those.
And what do you worry about regarding the hardware if I may ask? That there is some way the hardware bypases the software running on the device and send out information?
> "And what do you worry about regarding the hardware if I may ask? That there is some way the hardware bypases the software running on the device and send out information?"