Is it the PostgREST part? Are you using it for simple queries, or are you trying to use it for complex business logic?

Asking because PostgREST is great when you use it the way it’s intended but like any tool it will underperform when used in a way it’s not supposed to. It’s a screwdriver that you shouldn’t use to hammer nails.