Well the case is different here.
OP was present in the conversation and was able to figure out it's important without knowing the language. Otherwise they can just say they had a very important conversation in a place where OP was not present.
Also curious what happened after OP figured it out and asked them to switch to English. Did they refuse? Did OP reach out to his manager? Did manager ignore OP? Did OP reach out to skip or HR about the manager?
Lot of missing details.