That's essentially what we ended up doing, selectively using Serpy for certain slow responses. It still felt hackish though, as though we were taking the brakes off certain responses because we knew they needed little validation (IIRC). DRF serializers still need work; marshmallow provides a similar feature set to DRF but serializes almost twice as fast:
https://voidfiles.github.io/python-serialization-benchmark/
Disclaimer: The above is not my project