Even with tarballs all it takes is one fuck-up in a script and your homedirectory is gone. In that sense nothing is safe. You could of course audit all the code that the makefile or build script executes but I really don't know anybody that would do that for anything beyond the trivial.

We've been totally conditioned to just wget some archive, unpack it and build it, and even if git clone takes it one step further in day-to-day practice there is no difference between the two.