This is unlike, say, naturally radioactive carbon or potassium isotopes, that have a relatively constant concentration in the animal's body over it's life time - if a fish eats some high potassium food, it will excrete an equivalent of his own equally radioactive amount to maintain homeostasis.
It's for this exact reason the net "banana dose" of radiation, unless you are potassium deficient, is in fact zero.