At more advanced levels the theories are based on differential geometry and operators on Hilbert space. I'm not sure if fully worked out finitist versions of these even exist. Where finitist versions do exist, they're often technically more difficult to use than the standard versions, which is the opposite of an improvement in my view.

Whether it's undesirable for your mathematical foundation to prove the Banach-Tarski paradox is debatable. It's counter-intuitive, but doesn't lead to contradictions, as far as is known. It doesn't apply to physics because the construction uses non-measurable sets.