Yes, objects have the cruft and primitives don't. So you use primitive types whenever you can, including mathematical operations (unless the numbers are big and you need a BigInteger or BigDecimal, but that's another story). So far so good. But then, because of type erasure, collections (and literally anything generic except arrays) only accept objects, meaning that if you need a set of longs, you use a set of Longs instead. And if you are using some complex algorithm that relies on maps or sets of primitive collections, where you are constantly adding or removing elements, boxing and unboxing will kill your performance. I have been hit by this more than once. The last time I measured this (about half a year ago, which was the last time I encountered this in my job), the difference in memory usage was several Gb, and in particular there were about 7Gb used just for the Long class.

There are third party libraries that alleviate this. I use koloboke a lot, for example (which works for maps and sets, but lacks sorted collections and multimaps, which I also need to use very often); but the problem remains for anything slightly complex, and it's not uncommon to find yourself writing two almost identical copies of a relatively complicated method or class, one for objects and another one for ints (and maybe a third one for longs, a fourth one for floats...), because otherwise you hit the same problem.

So yeah. Primitive types are not objects and that limits them because Java doesn't work well with things that are not objects. C++ is much better in this sense, because generic code is actually generic and any type will do.