Perhaps we are arguing the same thing ultimately, but it's important to understand the distinction Google is attempting to make. They're basically trying to cover themselves by asserting that they are not granting any additional freedom beyond what the designer intended. Relicensing without permission is copyright infringement-- Google doesn't want the headache. Their safest course is to specify that unless you've been given license to do certain things by the content owner, that Google themselves is not licensing you to do those things.

Think of it like an affirmation from an open source project licensed under an MIT or BSD license. Although they might include GPL code from elsewhere, they'll generally state that they're not relicensing someone else's GPL code under the less restrictive license.

My personal opinion is to leave things as they are until you hear from Google. Simply be aware of the possible interpretations of the ToS. I do believe Argorak has the correct sentiment: that it's ultimately up to the individual font designer/cretor.

Your temporary caching, to me, seems fine. I read the ToS as saying that permitting your end user to obtain permanent copies might be disallowed ... at the discretion of the font designers. To be absolutely certain, you should check each font's license. Considering the fonts aren't Google's to license, I doubt you'll hear from them.