Apache Kafka and GDPR compliance (opens in new tab)

As everything in life, to gain something, you need to sacrifice something else. With RDBMS you get mutability; but to go 10x or 100x faster/larger you need to make hard decisions.

HDFS, S3 and other systems have immutability in-built. Immutability is not bad per-se, as it give (some) assurance that data has not been tampered with, and although it could be implemented, the system cost could be significant.

Stricking the right balance is the challenge

It's not that simple. For example in my business, we may give some money to help someone "once in its life" (the law says so). Therefore, if the persons asks to be deleted, then we might not apply the law anymore because it'll mean we won't remember the decision... I think GDPR is a good thing, but at some point, in my business, those who write the laws will have to be aware of it (and the legal teams is miles away from the IT stuff, sadly).

Where integrity matters, you never want data to be mutated with no trace. An audit trail is almost always needed - it’s not an extreme leap, then, to say “why don’t we just replay the audit trail to arrive at the current state?”

>What, none of these people /ever/ foresaw the need to delete some data?

It's a performance trade off, and not a very surprising one. Hard Disk Drives have always been known to never actually delete data (if you want the data gone, you overwrite it with 0s). It's not unimaginable that this performance trade-off found its way up the stack.

And just like a regular HDD, you can forcibly delete the data, it's just a very expensive operation that isn't needed 95% of the time.

I asked a GDPR consultant we hired about backups. The answer was basically that the letter of the law requires data to be deleted from backups, but people aren't going to do that, and there is some language about "reasonable" effort or something like that.

This might be useful: http://www.davidfroud.com/does-right-to-erasure-include-back...

Deletion doesn’t have to be immediately complete; if you rotate backups on a six month period then that’s okay.

At least, that’s what I heard about how a BigCo is doing GDPR.

My understanding is that the GDPR “right to be forgotten” does not cover backups. There may be some exceptions, but there are practical limits on its reach.

> How exactly do you "forget" the data on the logs?

If we think around the options, you can have either:

i) eventual deletion (log retention policy) ii) compacted topics (and push null values) iii) expensive re-processing of the entire log iv) expensive segment re-write operation

with each option bringing in a new set of challenges

Dear System Administrator,

We've just hacked your server and wiped the crypto keys for your users. As you know, all your backups are now useless.

Send us $1 million in Bitcoin to get your crypto keys back.

Sincerely,

Hacker McHackface

I would consider that already covered under the GDPR. Most machine learning approaches today make little to no guarantees about differential privacy and allow for (partial) extraction of the training dataset, which would mean that the request for deletion was never fully fulfilled.

if your PII has been incorporated into a model, let's say giving a likelihood to buy red cars based on 100 data points, then it's fairly safe to assume your PII is anonymised - i cannot imagine a way back from model that each input

GPDR itself doesn't specify cookies use. "Cookie law" is defined in ePrivacy Directive (2002/58/EC) which to be replaced by ePrivacy Regulation which is an addendum to GPDR. Actually, it's going to be much saner approach than the joke the current "cookie law" is:

"Simpler rules on cookies: the cookie provision, which has resulted in an overload of consent requests for internet users, will be streamlined. The new rule will be more user-friendly as browser settings will provide for an easy way to accept or refuse tracking cookies and other identifiers. The proposal also clarifies that no consent is needed for non-privacy intrusive cookies improving internet experience (e.g. to remember shopping cart history) or cookies used by a website to count the number of visitors."[0]

To answer your question "do we need to get consent from users before we can set any cookies?"

It depends: yes for tracking cookies, no for others. How to tell them apart is another question..

0. https://en.wikipedia.org/wiki/EPrivacy_Regulation_(European_...

Isn't that already an EU requirement?

I wish you wouldn’t call it irreversible. Every large public claim of that sort has proven false. Consider the Netflix case, where the separate IMDB review dataset allowed reconstruction of pseudonymous movie watching records.

These approaches may help with compliance, but they’re the opposite of real safety.