It's just diatoms. We all eat all sorts of things and it's fine for the most part. As long as you aren't eating actual poison you are fine.

"The dose makes the poison." Even ionizing radiation (sunlight) in small doses over a whole lifetime isn't really harmful. Especially given that it's how we naturally make vitamin D.