Faulty Hopes for UI Testing Tools
Michael Feathers wrote a tough post recently on UI testing tools.
The fact of the matter is that UI based testing should be used for UIs: that’s it. You should not be testing your full application end-to-end through a UI Testing tool. First of all, that sort of testing couples some of the most important tests in your system to one of the most volatile parts of it.
I understand the frustration he speaks from, but I’ve always realized that you don’t really want to completely try to test through the front-end of the application. It can be useful with legacy systems to get a minimal test harness in place. With the right testers I’ve even seen some of the Mercury products used effectively for UI testing, though it probably required a 3-1 ratio of QA to developers and the tests were not customer tests, but written more as a regression suite.
My use of functional testing tools like Selenium has generally been for a spattering of end to end tests and some amount of customer tests. I’ve never really tried to achieve a high level of coverage with these tests. Obviously testing the business logic through something like Fitnesse is more effective or even setting up BDD style tests tend to make better customer tests.
I’ve actually been saddened over the past five years or so to see that tools like Fitnesse haven’t seen more adoption. I think the BDD development has put more of a focus on developing customer style tests, but in general if there are automated end-to-end test suites in place there often overly reliant on something like Selenium.
A couple of thoughts, Ed. Cucumber is getting a lot of adoption in the Ruby world. It’s my favorite tool bar none for acceptance tests. We use primarily to drive Rails integration tests, but also have it driving a growing Selenium suite since we have a decent amount of javascript-heavy functionality on Weplay. You and Michael are right that these tests are more fragile than tests that skip the UI, but some of this functionality has to be tested, IMO.
Cheers from NYC,
Luke
Hope NYC isn’t too cold this time of year. Anyway your right about Cucumber, but then Cucumber is giving you BDD style tests and avoiding the UI again like FIT and others. I think using something like Selenium for has to be done for now when you’re developing javascript heavy front-ends. The difficult part is if you setup an extensive test suite you suffer from brittleness in these sorts of tests. So I’m pretty much on the same page with your approach.