The Federal Reserve is also intimately bound up in fractionally reserved expansion of the paper currency, so it is reasonable to assume that they might use the same basic technique with gold. And that, in fact, is the historical origin of banking, in the modern, leveraged sense: gold assayers began loaning out gold on a fractionally reserved basis, allowing them to collect interest beyond their core assets. Worked great until people ran on them.
So I assert that it is less irrational to believe that there is a possibility that central banks engage in manipulation of gold prices than it is to believe that there is a dinosaur living in the subway under manhattan.
YMMV.