Well, the instrument wouldn't me measuring yellowness... it would be measuring electrical impulses that (in some individuals) correspond to the (verbally asserted) perception of "yellow". "Yellow" is not a characteristic of the world; it's a convenient label that humans apply to some bucketed sets of sensory perceptions.
I agree that there are sensory perceptions humans are capable of perceiving and labeling as colors that cannot be attributed to external physical phenomena, but those are largely artifacts of the way our brain processes signals. For example if you stare at a purple dot for some time, then look away, you'll perceive a yellow dot where there is no external set of photons corresponding to the wavelengths that normally trigger the sensation of yellow striking your retina.
This is just more explanation about how "yellowness" is a characteristic of our brains, not of the external world.
Or did you mean something other than what I'm referring to here? I think that for the vast bulk of humans, the vast bulk of the colors they perceive regularly are due to photons striking rods and cones in their eyes at various intensities, causing color sensations to occur in the brain.
Do you think something else is happening?
You seem to understand how the eye works, and some neuroscience, so I don't understand how you can have the questions that you raise about whether we can build cameras that sense "color" instead of "light"