It's not that many years ago that Swedish police cooperated in illegally black-bagging two asylum seekers and illegally handed them over to the CIA who shipped them off to Egypt - the regime they'd fleed - where they were promptly tortured by Mubaraks secret police.
If Swedish police was willing to effectively kidnap political asylum seekers and hand them off to a foreign government, in violation of Swedish law and international treaties, and not punish anyone for it, why should Assange trust them?
We know that CIA rendition flights continued for years after the above became publicly known and the Swedish government promised rendition flights would end. We know this because of Wikileaks documents. So why should Assange trust the Swedish and US governments not to conspire?
Whether or not there actually is some conspiracy (I don't believe there is), Assange has plenty of reasons to be paranoid about it.