It was one of the first games and easily outclassed by later releases so I'm sure there are many optimizations that could be made.

I hope we get the same with Waverace one day, nothing since has matched the feel of the wave physics in that one, it would be interesting to see what they did.

My unfounded suspicion is that optimizations might have been turned off to avoid bugs in the toolchain. The developers would have had to deal with very early versions of these tools before the comsole was launched and in development itself.

Apparently some later releases of SM64 turned on optimizations and have less lag than the earlier ones.