Fortunately you can do it by using phantomjs directly instead of going through the Selenium WebDriver API. Maybe one day the phantomjs WebDriver API implementation (ghostdriver) will extend the API to pass HTTP status information back to the caller. Until then, this API is unusable (at least for me).
There are tools, such as BrowserMob Proxy, far better suited for monitoring HTTP traffic. And they'll get you all the headers. You can even capture to HAR so you measure performance.
Also, phantomjs has access to all the information you want and the WebDriver API already has a capabilities negotiation facility.
[Edit] Don't forget that the original URL is the only one supplied by the client of the API. It may be incorrect for very different reasons than all the other resources included by the page itself. That's why it is justified to treat it as a special case.
For example: here's the wiki on network monitoring including HAR: https://github.com/ariya/phantomjs/wiki/Network-Monitoring
The API seems pretty clean to me but I guess that is a matter of opinion.
See: http://voorloopnul.com/blog/a-python-proxy-in-less-than-100-...
In theory you could recreate this in another language such as Python but you would have to both parse the JavaScript from the website and implement a full browser.
Selenium is terrible, performance wise, and requires a significant investment in environment in order to work reliably. I try to avoid it except when I absolutely cannot.
It's still "in an early stage of development" but it's on my list of libraries to keep an eye on for when I have time to tackle the JS-heavy sites of the world.
For testing on browsers other than WebKit (or vendor specific WebKit edge cases) use Selenium. Harder to setup, more complex, probably faster (still slow for testing) but not limited to WebKit.
[1] Sorry folks but some JavaScript is required to programmatically interacting with the web - also need some HTML and CSS.
I prefer using lxml myself, since I like using XPath queries, but bs4 sometimes parses broken HTML better than any of the provided lxml parsers do.
Minor nitpick: I don't think it is a good idea to copy a binary directly to /usr/bin, without a package manager. You could just put it into /opt and symlink to /usr/(local/)bin.
Also, PhantomJS works fine in this case because the binary in the tarball is statically compiled. You can find a whole lot of qt stuffs inside PhantomJS source repository. There ain't no such thing as "truly headless".
@Steven5158 thanks for the share.
If anyone here wants help in building SMS apps do contact me.
Very, very good to know when diving into scraping.
