Totally, I also think there's a huge vicious cycle risk with these tools.

The more we rely on them, the more our own mental model of existence and knowledge will adapt to become little more than statistical aggregates of data. Science has already trended this way in certain areas. We used to reason from deductive principles, look for data, and fit data to both inductive hypotheses and deductive principles. Steadily we've come to care less and less about the deductive part and think of everything in terms of probabilities and statistical models. While this is not bad per se, I do think our growing ignorance of deductive reasoning causes us to miss some important aspects of existence.

That reminds me of the whole most real numbers are irrational, in fact the vast-vast majority of points on the number line you see before you, are not rational numbers. Humans don't have a good intuition for maths. So guess what happens when you brute-force a neural network machine into an AI?