When my sister visits, it almost never responded to her for example.

And in my personal experience, I used to have a very deep voice and it always responded to me. I decided I would rather have a more androgynous voice and did some voice training to accomplish that and use a pretty neutral pitch. Now it only responds to my normal voice about 50% of the time, so I intentionally drop my voice an octave whenever I speak to it.

But somehow it's not just about pitch either. I have friends who are trans men, and speak in a deep voice but they still have trouble getting Mycroft to respond.