Sweden has in the past illegally handed political asylum seekers over to the CIA
without extradition requests, knowing these people would be handed back to the regime they escaped from, to be tortured.
Later Wikileaks released documents that showed that years after this had supposedly been stopped, Swedish military intelligence uncovered that CIA rendition flights were still ongoing, with the cooperation of Swedish airport staff.
So there might not be much for Assange to worry about legally, but that doesn't say much.
He might very well be paranoid, but Sweden hasn't exactly acted in a way that does anything to give someone in his position reasons to trust they'll even follow their own laws.