I think you're looking for the Procrustes transform, which uses SVD to optimize a minimal transform for rotation, translation, _and_ scale.

https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem