PhantomJS is pretty big. IIRC, building it takes quite some time. I think they bundle webkit and the necessary parts of Qt, and you'd have to be out of your mind to build that from source if you can avoid it.

Using official distribution packages would be a better idea, but their freshness can vary, especially on RHEL.