Epstein was running what he said was legal tax avoidance system for his clients.
I'm not saying they were breaking any laws in it, but Epstein is a dodgy guy (also he used to be involved in IIRC a Ponzi scheme and was previously sacked from a big firm - red flags galore with this guy) and the scheme worked off asset prices and trusts.
If a bunch of billionaires could manipulate asset prices (selling illiquid assets like mansions and artwork between each other and their trusts) I suspect they could really bring down their tax bills. This would be illegal (I think) but you'd need to untangle a large web of transactions to prove it.