Not sure if that's too much of a crutch for you, but it's quite easy to create an "Ask Gemini" shortcut that calls a Cloud Function and returns a spoken response. I use this on my HomePods all the time, and it's working great.