Straight off, no on at Sauce Labs hates browser testing, that's just a stupid thing to posit.
Test::Right is a really cool mental model of testing, and it's going to make people's lives better. Robust tests, parallelism, immediate cross-browser compatibility (this is an issue people without a lot of experience don't really understand), a lot of cool shit.
I love the promise that if I write my tests in this pattern/framework, my tests will always be good. It's not there yet, but it's still damned cool.
We also have several volunteers who're excited to port this to other languages, but we started with ruby because it's a fertile playground, and the important piece here is the pattern, not the language/framework itself. It'll end up being the flagship browser testing framework de facto, but that's not the goal here.
I played a big part in putting some pieces of Test::Right together, full disclosure.
I like this a lot.
Prior to that approach, we tried everything: we actually wrote our own test framework that used javascript to drive the browser from inside the browser, then we moved to using selenium, but they all suffered from that same problem of incredibly fragile tests. Over time, test maintenance is easily 10x the cost of initial test development, so it's a huge, huge problem.
So I see this as a promising direction, but building and maintaining that (essentially) Page Object model by hand is going to result in its own set of maintenance problems. How do you know when the model is up to date? If someone changes the model, how do you know which test that affects if you're in a dynamic language like Ruby? It solves one set of horrific problems, and replaces it with another set of problems that are hopefully, with enough developer discipline applied (and guidance from the framework), ultimately slightly less horrific. Being able to automatically generate the POM via metaprogramming by statically analyzing your application is awesome, but it's not a technique that really works with any standard web framework that I know of.
Web app testing really still has a long, long, long way to go.
That said, it's going to be a lot of fun, and I'd love to have someone with your experience and insight join in the conversation. I'll have to think of a way to do it - a Test::Right convore, or mailing list, perhaps?
I especially like the decoupling of intentions from mechanics. It lets us improve the mechanics without breaking or rewriting all the tests. Maybe the mechanics are implemented with JS DOM-traversal code, maybe it's CSS selectors, maybe it's SIKULI-like image recognition (http://sikuli.org/).
Looking forward to trying it.
We all sat down a bit ago to reflect on 16 month of non-stop in-the-trenches browser testing for all kinds of people, to integrate it all. We talked about how we wanted to make things better for the people out there who know what they want to do, but are continually screwed over by the how.
The goal "impossible to write bad tests" floated out, and it was almost a joke. But we started to wonder it would take, and it grew more serious. It's a mantra now.
And we have so much cool stuff flowing out form that mantra in the next few weeks/months from that attitude, it's insane.
And excellent point about intentions/mechanics - Jason Huggins is all over Sikuli. We're watching very patiently :)
Thanks for the work you've already done on this, and will no doubt do in the future.
JavaScript also makes creating a very clean testing-focused DSL easy.
I'm hoping to open source the project soon. If anyone would like to know more give me a shout (@andrewdavey).
That said, it's not the important part. Selenium (and watir, windmill, etc.) is an execution tool, it doesn't help you avoid some very, very large pitfalls waiting for you down the road. Too often your tests are tightly coupled with the dom, data and selectors are strewn all about, you have manual pauses and timeouts set by someone who left years ago, and you have so many that the whole thing takes two weeks to run (yes, I've seen this a number of times).
Basically, I don't care about the tool to execute the tests anymore. They'll all become interchangeable with Selenium 2 anyway. The much more valuable piece here is in creating tests that are maintainable and a joy to work with.
Certainly best of luck, please be sure to share you experiences and insights! There's always more to be learned.
I've developed a rather nice asynchronous, sequential, execution model for test code. So they avoid the callback-hell you'd see with multi-step async tests. The spin assert idea mentioned in the article is very useful, and I'll be borrowing it for sure! :)
I'm wary of running tests in parallel, due to concurrency issues. However, I guess spinning up multiple web servers and in-memory databases would work fine.
In other words, shut up and show me. (This isn't an attack on saucelabs)
The github page is sparse on documentation, and the wiki page is basically a copy of the README.
So don't make vacuous claims. Write an article demonstrating how this system works, how it compares to X, and why it's better than X.
We err'd on the side of getting this out (several developers were chomping at the bit even with just a bit of editing on the post) - rest assured that we'll have more coming. In particular, we'd like to have some more documentation, examples, and best would be quick screencasts talking about and demonstrating the techniques/benefits going on in here.
No worries about 'shut up and show me' - but the counter-challenge is 'dive in and try it' :)
