Reports indicate Siri computes primarily in the cloud (the whole sound file is sent), and that it's not on iPhone 4 for other reasons (e.g. reduce server load, increase 4S sales)

The A5 has several integrated processors for specific purposes, specifically relating to audio and video. The Audience chip is likely one, but there are also processors specifically for the purposes of handling video encoding or decoding, and other things, probably like cryptography.

Apple was not specific about what all the coprocessors are or what they do, which is why I'm being vague, but they have publicly announced that these are integrated into the SoC.

So the "Rumor" part is that one of them is from Audience. That there are custome co-processors has been revealed publicly.