At least with gmail and chat clients etc. things are somewhat put in compartments, one of the services might screw up and do something with your emails but your Messenger or WhatsApp chats are not affected by that, or vice versa. But when you bake it into the OS (laptop or phone) you're IMHO taking a much bigger risk, no matter what the intentions are.
Whereas with apps like Gmail and WhatsApp on an iPhone, you must trust Google and Meta in addition to Apple, not in place of Apple. It doesn't distribute who you trust, it multiplies it.
In essence, what you're doing is training an assistant to learn all of your details of your life and habits and the question is if that "assistant" is really secure forever. Taken to the extreme, the assistant becomes a sort of "backup" of yourself. But yeah it's an individual decision with the pro's and con's of this.
Yeah, pretty much
Also, your grandma might not setup a VM, but it sounds like the off-device processing is essentially stateless, or at most might have a very lightweight session. It seems like the kind of thing one person could setup for their family (with the same tamper-proof signatures, plus physical security), or provide a privacy focused appliance for anyone to just plug into a wall, if they wanted to.
There have been many cases recently of compromised code being in the wild for quite some time and then only known about by accident.
I have no idea what code is running on a server I can't access. I can't exactly go SSH into siri.apple.com and match checksums. Knowing Apple's control freak attitude, I very much doubt any researcher permitted to look at their servers is going to be very independent either.
Apple is just as privacy friendly as ChatGPT or Gemini. That's not necessarily a bad thing! AI requires feeding lots of data into the cloud, that's how it works. Trying to sell their service as anything more than that is disingenuous, though.
That's like... the whole point? You have some kind of hardware-based measured boot thing that can provide a cryptographic attestation that the code it's running is the same as the code that's been reviewed by an independent auditor. If the auditor confirms that the data isn't being stored, just processed and thrown away, that's almost as good as on-device compute for 99.999% of users. (On-device compute can also be backdoored, so you have to trust this even in the case that everything is local.)
The presentation was fairly detail-light so I don't know if this is actually what they're doing, but it's nice to see some effort in this direction.
E: I roughly agree with this comment (https://news.ycombinator.com/item?id=40638740) later down the thread -- what exactly the auditors are verifying is the key important bit.
Apple has developed some of the strongest anti tampering compute on existence to prevent people from running code they don't want on hardware they produce. However, that protection is pretty useless when it comes to protection from Apple. They have the means to bypass any layer of protection they've built into their hardware.
It all depends on what kind of auditing Apple will allow. If Apple allows anyone to run this stuff on any Mac, with source or at least symbols available, I'll give it the benefit of the doubt. If Apple comes up with NDAs and limited access, I won't trust them at all.
