Counters are a different story than numbers. The numbers are highly regular, with the tiny exception that numbers 4 and 7 technically have two variants, where in certain areas one is preferred more than the other, and the pronunciation of 9 when talking about time. But that's really minor.

As soon as you're starting to count it's different.

> I think every natural language is irregular when it comes to numbers.

But.. it's not. Japanese is not. Just because it's not metric with grouping doesn't mean it's irregular.