Depends on the field, python is almost never the bottleneck in CV - but it almost always a bottleneck in NLP or Search research.

Regardless, the code in the github PR should be running strictly in pytorch for the heavy lifting. Clearly something is touching python that shouldn't be, or must be for frustrating reasons.