We don't completely ignore math until high school, suddenly say "so there are these things called numbers," expect to get along to algebra in one semester, and then throw our hands up in the air when that doesn't work out.

At first they are. This is different than the way I learned math as a kid. Today (at least at the elementary school that my kids attended), math starts out with relatively undifferentiated exploration, and hops around from one topic to another. It gains rigor more gradually.