I believe it is you missing the point. They're his keys. If I quit the company I'm working for, they can't just take all all my keys because at some point they gave me some.
This is no different than my old neighbor, Mr. Taylor. He used to let us use the pool in his backyard until one of the kids pooped in it. He stopped letting those kids come to the pool.
It's his pool, and it's his choice who can and can't use it. In this case, David Pollack has just decided nobody can use the pool, and he's put a fence around it.
You're just pissed you can't use the pool for free anymore.