"Wouldn't this concern be on par with consuming things that contain deadly things but in ppb or ppt quantities?"
It's not going to be enough to be toxic to humans. The issue is whether it's either introducing antifungal resistant fungi into your body, or causing fungi already in your body to become antifungal resistant.
