Your latency numbers for OpenAI (and Azure's equivalents) seem really high, I run time to first token tests and I see much better numbers!
(Also are those numbers average, p50, p99, etc? I'd honestly expect a box plot to really see what is going on!)
