You're only seeing the direct radiation risk here .. chances of being struck by a decaying particle from a hot pile of short half life material breaking down.
There is a significant ongoing risk from toxic isotopes that aren't especially radioactive but nonetheless pose a toxic health risk (akin to straight up rare heavy metal poisoning).
There is a related risk from 'biological isotopes' that are taken up within the food chain and concentrate in plants, animals and perhaps ultimately human bodies where slow damage is caused by slow breakdown (again, not highly radioactive).