Because they're not letters of the German alphabet, by definition, and they're not German vowels, by definition.
They're characters that represent vowel sounds, sure.
It's kinda the same thing as how in the English alphabet, Y is by definition a consonant, despite being used to represent a vowel sound in many words. (beauty, many, cyst, tryst, for example)
By which definition is "Y" a consonant? It represents the vowel sound much more often than it does the consonant sound, so if you go by that metric, it's a "vowel letter".
Of course, the more accurate take is that it's by definition neither a vowel nor a consonant because those are properties of sounds. They only apply to letters to the extent that letters consistently represent sounds, but when they do not, it doesn't make any sense to categorize letters like that.
At this point, I can only suspect (based on ample anecdotal evidence) that an inclination towards prescriptive linguistics is a quirk of German speakers ;)
