There is a fair point there, but a few things - HSV and Lab are only models, they don’t necessarily capture all visible colors (esp. when it comes to tetrachromats). Brightness is a dimension, and can affect the perception of a color, esp. as you get very bright - HSV and Lab are 3D spaces. Arguing that brightness should be ignored or factored out is problematic and only a small step from arguing that saturation should be factored out too and that color is mostly one dimensional.

According the opponent process model of colour perception you need three axes to represent all colours: luminosity [L+M+S+rods], red-green [L-M] and blue-yellow [S - (L+M)].