Black money largely isn't held as large stacks of suspicious looking cash - that's a Bollywood trope.
It's invested domestically in real estate, art, gold, or other assets, or held offshore, either in the form of freely convertible currency or other assets, which is why 98.96% of the cash was returned.
I strongly doubt a significant number of criminals openly advertised their possession of black money by returning it themselves.
Modi's ploy was an unmitigated failure.