I suppose the models might theoretically at risk if the learning data can be extracted, but I don’t think this will practically happen because it’s so far different from current GDPR practices. Someone would have to prove their protected information is inside of a model before they might have a chance. After that, I am not a lawyer.

Of course you can’t prove that some data cannot be de-anonymized unless there are duplicate entries. However, GDPR explicitly encourages anonymization, or “pseudonymization”, which therefore suggests that reasonable attempts to keep data generic are considered legal by this particular law. People have already pointed out that GDPR’s language here is too vague and makes bad assumptions about how identifying multiple quasi-identifiers can be.

> How do you prove there is no PII in the ML model?

Is "innocent until proven guilty" not a maxim in European justice?