https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
It's similar to the way in which when we're presented with words that are jumbled up apart from their first and last letter, our brain fills in what's happening. Vision in general is not just the mechanical encoding of sense data from light. It is as much psychological as physiological. We see the world in stable colours and contrasts because our mind controls for illumination and processes the colour of one detail relative to the whole picture. I think our mind's eye depends on similar kinds of psychological support.
My actual complaint runs deeper though. My experience of visualizing can't be a delusion because I'm describing my internal experience. You can claim I'm lying about my experience, but what you can't do is claim I'm actually experiencing something else and not noticing. If I didn't notice, then no, I did not experience that other thing.
Also, I can imagine my wife's face almost perfectly, but any attempt to draw her will result in a monstrosity because I suck at drawing.