PhantomJS also runs some ancient version of WebKit, which means it is not compatible with modern browsers.
(I keep intending to hack Blink into a state where it can run truly headless, like a library, so that hacks like the above aren't necessary.)
It's only ever worth using Selenium when you literally can't do anything else to get your testing done.
Edit: Going through my bookmarks bar and found this, from 2008 - http://www.alittlemadness.com/2008/03/05/running-selenium-he...
Might help anyone if they run into issues.
That's not to say there aren't good use cases for running headless browsers with Selenium. One huge benefit is not needing to run Windows VMs.