The thing is that numbers are usually wrapped in other things, like objects, hashtables, arrays, etc, and OCaml is a beast at dealing with that kind of code.

From a purely numbers perspective, its operations on integers have to use the LEA instruction instead of ADD (for example) because of the 1-bit tag, which slows things down a bit, but the speed at dealing with symblic code as I explained above more than makes up for it.