I disagree. People haven't had a Mathematics before Pythagoras. All Mathematics people had at that moment was that 1 golden coin + 1 golden coin = 2 golden coins or one copper knife. But trading is not mathematics if not talking about modern things like HFT with broad using of probability theory (and we obviously are talking about ancient times).
Physics is understandable even with way more simple (and way more ancient) species then humans. For example, an ape can take a stick and use more heavy end to make a more powerful hit. Birds seems to understand physics quite well - starting from setting wings while flying and ending to general intelligence of crows.
And BTW why are you concerned that in the pyramid of knowledge (Biology is a science > Chemistry is a science > Physics is a science > Mathematics is a language) there are three sciences and one a non-science?