I liked your paper. It reminded me on the tutorial session with Elisabeth Hendrickson I joined at the Agile Testing Days in Berlin this year. She had the same example and it was very easy for me to follow. Maybe I’ll be able to take some time and give Robot Framework a try in a week or so, when I’ve dealt with the last conference for me this year.

Markus, yes, you’ll see lots of similarities between what I do and what Elisabeth does. It was Elisabeth who turned me on to Robot Framework, and we often co-teach her wonderful Acceptance Test Driven Development class. And of course I stole the “login” example from her.

EP의 생각…

Writing Maintainable Automated Acceptance Tests…

Bob Martin posted a great video showing how to apply these ideas using FitNesse.

Someone will translate this to Cucumber any day now…

I became aware of this article by Uncle Bob’s blog and just want to express appreciation for this gap-filling work!

Assuming a project will be implemented using Acceptance Test-Driven Development (ATDD), then analogous to Test-Driven Development (TDD) the steps of development are the following:
* Write a failing acceptance test
* Develop production code using TDD (test, code, refactor) satisfying the test
* Refactor the acceptance test.

So basically, the missing step was the last one, namely the refactoring of the acceptance test, which includes techniques and patterns similar to those applied during the development of production code.
I just wanted to point this out, because the word refactor or its derivatives aren’t present in the paper. Since “test automation is software development”, therefore improving the design and structure of the tests is refactoring.

Great article, Dale! I would change just one small thing: I think the tests might be a bit more expressive of system behavior if you changed the variable names “${aTooShortPassword}, ${aMinimumLengthPassword}, ${aMaximumLengthPassword}, and ${aTooLongPassword}” to ${aPasswordWith5Chars}, ${aPasswordWith6Chars}, ${aPasswordWith16Chars}, and ${aPasswordWith17Chars}”. The reason is that as I looked at the tests you presented, I found myself wondering, “so just what are the minimum and maximum password lengths?”, and the only way to answer the question is to examine the variables and count “1-2-3-4-5-6″ etc. That said, there is a trade-off in my proposal, insofar as a reader would have to infer from looking at the tests that minimum and maximum password length are being tested. In my opinion, that’s not as error-prone or tedious as counting string lengths–but as with any trade-off, opinions may vary.

I think the text writted in BDD style and automated within jbehave (www.jbehave.org) is more easy and intuitive for java systems.

[...] a thoughful comment on my blog post about writing maintainable automated acceptance tests, Chris Falter suggested a different way to [...]

[...] have led to companies abandoning test automation initiatives, according to Dale Emery. In a newly published paper, Dale shares some practical ways to avoid common problems with test automation. He starts with some [...]

I deeply respect both Dale and Uncle Bob but I’m concerned that the true message of both articles might be lost in “proving that one tool is better than another”. The point that I think that both are making is that you want to make your tests in WHATEVER TOOL YOU’RE USING be as maintainable as possible. Tests, like code, can be fragile, have duplication, can smell and they can also be refactored. Gerard Meszaros, author of “xUnit Test Patterns: Refactoring Test Code” gave a presentation this week at the company that I work for. He used the very same techniques that Dale and Uncle Bob used but used an xUnit based tool for his examples. His presentation, like Dale and Bob’s, did not describe how to use their tools per se, but focused on removing the cruft, verboseness, etc from the tests to make them more maintainable.
All three of these presentations made me a better developer! Thanks Dale, Bob and Gerard!

Good point, Paul. I hope people aren’t seeing any of this as a tool war. I certainly didn’t mean it that way.

I’m delighted that the developers of Robot Framework, FitNesse, and Cucumber (and perhaps other tools I’m not aware of) are rapidly learning from each other, each incorporating the good ideas from other tools into their own. I look forward to the day when people can represent tests in whatever way seems most natural and expressive to them, and each tool can directly execute tests written in a variety of those “natural” styles. The tool developers are clearly moving in that direction.

Hi Dale,

I think your paper is on the money. Testing is engineering. Maintaining any large body of code requires discipline. And maintainability is the enemy of most of the ATDD suites out there.

Just to add to the list of great tools in this area – I’m using Concordion. I really love this tool because the website author really understands this problem of maintainability and really tries to address it in the design of the framework. I also think Concordion doesn’t get the attention it deserves and I’ve been on mission to improve that.

It could be argued that this tool requires more programming skill than others mentioned. But in my opinion that is a very good thing indeed.

I have written a blog entry with a full example of using Concordion here. http://bit.ly/cZhlVn. Concordion specifically enforces abstraction to enhance maintainability. Hopefully these ideas will catch on.

Neil

[...] we can improve it. Before I do this, I will point you to two great articles from Dale Emery. The first is tenish pages piece where he walks through a login screen. Uncle Bob showed the same example [...]

[...] Today, we’re going to continue our ParkCalc automation excursion. We will take a closer look on the second test in the provided test examples, the keyword-driven format, and see how we can improve it. Please note that I added an update to the previous blog entry showing that we can improve the test even more by extracting the date ranger into meaningfully named variables – just as Dale Emery did in his article Writing Maintainable Automated Acceptance Tests. [...]

[...] we can improve it. Before I do this, I will point you to two great articles from Dale Emery. The first is tenish pages piece where he walks through a login screen. Uncle Bob showed the same example [...]

I feel so lucky that I was there in the first place when Robot hasn’t been open sourced yet. I totally fit my style in testing/test automation, and whatever until mindset.

If you haven’t tried Robot, you should try.

How to Structure a Scalable And Maintainable Acceptance Test Suite…

You started to write automated acceptance tests, so that you don’t need to retest all the results from earlier sprints at the end of every sprint. Greate, we too. After a while of successful test automation, tests start to look like a big ball of…

Hi,

I just want to say thank you for providing such good paper on automation. I’m just starting my work in this area and will try to write maintanable tests

and just small noticed from my observations:
on “Listing 1″ 2nd “Account created” example should fail due to “user already exist” issue

regards,
lorak_kash

Good eye, Lorak.

It turns out that the problem you spotted is actually a feature. There’s currently no “change password” command, so we change passwords by creating an account with an existing username and a new password. You can no doubt identify numerous horrible consequences of this feature. Fixing those problems, and adding a “change password” command, are jobs for a future iteration.

And there’s a subtler problem here, hinted at by your insight: Each test leaves side effects that might confuse the next test. Perhaps we should rewrite the tests so that each starts with an empty account list.