That's only kind of true. Standard continuous-variable quantum mechanics has all sorts of consistency problems, and the only way to get reasonable predictions out is to paper over infinities and pretend they don't exist.

I know there are what are called C*-algebras, which help solve some of these issues, but I don't know anything about them. I do know that the Hilbert Space approach is not sufficient.

True, but from a mathematician's point of view, the theory quickly becomes complicated (and in some sense limited) if you really want to do things rigorously when working with continuous systems, something that does not happen with finite dimensional systems as the parent comment probably alludes to.