No, he isn't reading it incorrectly. Buffett is claiming that on aggregate, because of the fees, the passive investors will be better off than the active investors.
From page 23:
A lot of very smart people set out to do better than average in securities markets. Call them active investors.
Their opposites, passive investors, will by definition do about average. In aggregate their
positions will more or less approximate those of an index fund. Therefore, the balance of
the universe—the active investors—must do about average as well. However, these
investors will incur far greater costs. So, on balance, their aggregate results after these costs
will be worse than those of the passive investors.
