Writing Maintainable Automated Acceptance Tests

November 23, 2009 at 1:33 pm — Testing — Tags: , , , , ,

I’ve posted “Writing Maintainable Automated Acceptance Tests” on my articles page.

The article demonstrates how to make automated acceptance tests more maintainable by:

  • Hiding incidental details
  • Eliminating duplication
  • Naming essential ideas

Though the examples in the article use a very nice testing framework called Robot Framework, the ideas work just as well with other other popular open-source testing frameworks, such as FitNesse and Cucumber.

You will be able to follow the article even if you don’t know Robot Framework. But don’t be surprised if it inspires you to give Robot Framework a try.

RSS feed for comments on this post. TrackBack URL

20 Comments »

Comment by Markus Gärtner — November 23, 2009 at 2:15 pm

I liked your paper. It reminded me on the tutorial session with Elisabeth Hendrickson I joined at the Agile Testing Days in Berlin this year. She had the same example and it was very easy for me to follow. Maybe I’ll be able to take some time and give Robot Framework a try in a week or so, when I’ve dealt with the last conference for me this year.

Comment by Dale — November 23, 2009 at 2:20 pm

Markus, yes, you’ll see lots of similarities between what I do and what Elisabeth does. It was Elisabeth who turned me on to Robot Framework, and we often co-teach her wonderful Acceptance Test Driven Development class. And of course I stole the “login” example from her.

Trackback by eungju's me2DAY — November 30, 2009 at 3:11 am

EP의 생각…

Writing Maintainable Automated Acceptance Tests…

Comment by Dale — December 7, 2009 at 5:41 pm

Bob Martin posted a great video showing how to apply these ideas using FitNesse.

Someone will translate this to Cucumber any day now…

Comment by Andras Hatvani — December 8, 2009 at 5:21 am

I became aware of this article by Uncle Bob’s blog and just want to express appreciation for this gap-filling work!

Comment by Andras Hatvani — December 8, 2009 at 7:11 am

Assuming a project will be implemented using Acceptance Test-Driven Development (ATDD), then analogous to Test-Driven Development (TDD) the steps of development are the following:
* Write a failing acceptance test
* Develop production code using TDD (test, code, refactor) satisfying the test
* Refactor the acceptance test.

So basically, the missing step was the last one, namely the refactoring of the acceptance test, which includes techniques and patterns similar to those applied during the development of production code.
I just wanted to point this out, because the word refactor or its derivatives aren’t present in the paper. Since “test automation is software development”, therefore improving the design and structure of the tests is refactoring.

Comment by Chris Falter — December 8, 2009 at 8:01 am

Great article, Dale! I would change just one small thing: I think the tests might be a bit more expressive of system behavior if you changed the variable names “${aTooShortPassword}, ${aMinimumLengthPassword}, ${aMaximumLengthPassword}, and ${aTooLongPassword}” to ${aPasswordWith5Chars}, ${aPasswordWith6Chars}, ${aPasswordWith16Chars}, and ${aPasswordWith17Chars}”. The reason is that as I looked at the tests you presented, I found myself wondering, “so just what are the minimum and maximum password lengths?”, and the only way to answer the question is to examine the variables and count “1-2-3-4-5-6″ etc. That said, there is a trade-off in my proposal, insofar as a reader would have to infer from looking at the tests that minimum and maximum password length are being tested. In my opinion, that’s not as error-prone or tedious as counting string lengths–but as with any trade-off, opinions may vary.

Comment by Cristiano — December 8, 2009 at 8:22 pm

I think the text writted in BDD style and automated within jbehave (www.jbehave.org) is more easy and intuitive for java systems.

Pingback by Conversations with Dale » Tradeoffs — December 9, 2009 at 12:15 pm

[...] a thoughful comment on my blog post about writing maintainable automated acceptance tests, Chris Falter suggested a different way to [...]

Pingback by Maintainable Automated Acceptance Tests « Test Infected — December 11, 2009 at 4:22 am

[...] have led to companies abandoning test automation initiatives, according to Dale Emery. In a newly published paper, Dale shares some practical ways to avoid common problems with test automation. He starts with some [...]

Comment by Paul Leclerc — December 19, 2009 at 3:03 pm

I deeply respect both Dale and Uncle Bob but I’m concerned that the true message of both articles might be lost in “proving that one tool is better than another”. The point that I think that both are making is that you want to make your tests in WHATEVER TOOL YOU’RE USING be as maintainable as possible. Tests, like code, can be fragile, have duplication, can smell and they can also be refactored. Gerard Meszaros, author of “xUnit Test Patterns: Refactoring Test Code” gave a presentation this week at the company that I work for. He used the very same techniques that Dale and Uncle Bob used but used an xUnit based tool for his examples. His presentation, like Dale and Bob’s, did not describe how to use their tools per se, but focused on removing the cruft, verboseness, etc from the tests to make them more maintainable.
All three of these presentations made me a better developer! Thanks Dale, Bob and Gerard!

Comment by Dale — December 19, 2009 at 4:24 pm

Good point, Paul. I hope people aren’t seeing any of this as a tool war. I certainly didn’t mean it that way.

I’m delighted that the developers of Robot Framework, FitNesse, and Cucumber (and perhaps other tools I’m not aware of) are rapidly learning from each other, each incorporating the good ideas from other tools into their own. I look forward to the day when people can represent tests in whatever way seems most natural and expressive to them, and each tool can directly execute tests written in a variety of those “natural” styles. The tool developers are clearly moving in that direction.

Comment by Neil Thorne — February 14, 2010 at 11:45 am

Hi Dale,

I think your paper is on the money. Testing is engineering. Maintaining any large body of code requires discipline. And maintainability is the enemy of most of the ATDD suites out there.

Just to add to the list of great tools in this area – I’m using Concordion. I really love this tool because the website author really understands this problem of maintainability and really tries to address it in the design of the framework. I also think Concordion doesn’t get the attention it deserves and I’ve been on mission to improve that.

It could be argued that this tool requires more programming skill than others mentioned. But in my opinion that is a very good thing indeed.

I have written a blog entry with a full example of using Concordion here. http://bit.ly/cZhlVn. Concordion specifically enforces abstraction to enhance maintainability. Hopefully these ideas will catch on.

Neil

Pingback by ParkCalc automation – Refactoring a data-driven test | Markus Gärtner's blog — June 24, 2010 at 2:25 pm

[...] we can improve it. Before I do this, I will point you to two great articles from Dale Emery. The first is tenish pages piece where he walks through a login screen. Uncle Bob showed the same example [...]

Pingback by ParkCalc automation – Refactoring a keyword-driven test | Markus Gärtner's blog — June 26, 2010 at 3:31 am

[...] Today, we’re going to continue our ParkCalc automation excursion. We will take a closer look on the second test in the provided test examples, the keyword-driven format, and see how we can improve it. Please note that I added an update to the previous blog entry showing that we can improve the test even more by extracting the date ranger into meaningfully named variables – just as Dale Emery did in his article Writing Maintainable Automated Acceptance Tests. [...]

Pingback by ParkCalc automation – Refactoring a data-driven test : Clevertester — June 30, 2010 at 5:07 am

[...] we can improve it. Before I do this, I will point you to two great articles from Dale Emery. The first is tenish pages piece where he walks through a login screen. Uncle Bob showed the same example [...]

Comment by Xu Yi — July 13, 2010 at 8:08 pm

I feel so lucky that I was there in the first place when Robot hasn’t been open sourced yet. I totally fit my style in testing/test automation, and whatever until mindset.

If you haven’t tried Robot, you should try.

Trackback by blog.codecentric.de — July 28, 2010 at 2:11 pm

How to Structure a Scalable And Maintainable Acceptance Test Suite…

You started to write automated acceptance tests, so that you don’t need to retest all the results from earlier sprints at the end of every sprint. Greate, we too. After a while of successful test automation, tests start to look like a big ball of…

Comment by lorak_kash — August 10, 2010 at 5:02 am

Hi,

I just want to say thank you for providing such good paper on automation. I’m just starting my work in this area and will try to write maintanable tests ;-)

and just small noticed from my observations:
on “Listing 1″ 2nd “Account created” example should fail due to “user already exist” issue ;-)

regards,
lorak_kash

Comment by dale — August 13, 2010 at 1:13 am

Good eye, Lorak.

It turns out that the problem you spotted is actually a feature. ;-) There’s currently no “change password” command, so we change passwords by creating an account with an existing username and a new password. You can no doubt identify numerous horrible consequences of this feature. Fixing those problems, and adding a “change password” command, are jobs for a future iteration.

And there’s a subtler problem here, hinted at by your insight: Each test leaves side effects that might confuse the next test. Perhaps we should rewrite the tests so that each starts with an empty account list.

Leave a comment