User Acceptance Testing Patterns and Processes

by Randy Rice

March 2011

What is a Pattern?

I do woodworking as a hobby. Woodworking uses patterns and plans quite a bit. I find that the more complex the project, the more I need a good pattern!

Patterns are helpful because:

• They provide repeatable ways of doing something

• They can be adapted to meet your situation

• You can have different patterns for different situations

• They help estimate the amount of resources needed

What the Best Way to Perform UAT?

There are many ways of performing UAT. The key is to find the way(s) that best meets your needs. In many cases, the patterns and processes for UAT can be divided into formal and informal. I don’t have a magic formula for knowing how much process is right for you. However, consider the following:

• The level of documentation needed from the process
The more documentation needed from UAT, the less you will be able to use informal methods. You can get documentation from informal approaches, but if you must show test scripts and test plans for independent testing then it will be hard to reconcile the need for documentation with informal UAT methods. Also consider that UAT scripts can have uses in training, documentation and other project areas.

• The level of test repeatability needed

A benefit of test scripts is they provide a repeatable path to follow. This is good when you want to plan in advance what will be tested at a scenario level. If you lack business process knowledge or system knowledge, or want to discover defects instead of confirming correctness, informal tests may be better.

Repeatability is needed for three reasons: 1) To be able to confirm the software failure you are reporting, 2) to tell the developer what you did in a test, and 3) to be able to re-test a fix.

• The time available for test design, test performance and test evaluation

Formal test processes can take time to perform, especially the first time. If you have only a few days for UAT, a less formal approach may be the best option. There are techniques such as session-based testing, where each test objective is performed in a one to two-hour timeframe.

However, consider that good test planning can save time in the performance of UAT by reducing or eliminating the need to research the observed results from a test. Think about all the times as a tester you may have asked after a test, “Is this a bug or a feature?”

• The degree of guidance needed by the user testers

A key rule is to write tests based on the expected audience. If your audience will need to know exact keystrokes (such as in the case on a new application), you will probably need detailed test scripts and/or test cases. Less formal approaches depend more on the testers’ knowledge and experience.

• The amount of knowledge you have (or can obtain) about the system you plan to test

You need system and business knowledge to plan acceptance tests. If you lack either of these, you will likely to either gain the knowledge quickly to create documented tests, or gain knowledge as you test. Exploratory testing is a good way to learn and test at the same time.

• The level of business, technical and project risk

Opinions vary on this point, but my view is that the higher the risk, the more definition you will want in your testing processes. Imagine yourself using a medical device that is safety critical. Which would you prefer the testers have used – a formal test process with test cases and scripts that had been reviewed and traced back to requirements, or an informal testing process?

• The ability of the user testers to think of productive tests on their own

I believe the key to being a good tester for any type or phase of software testing is to be able to think critically and creatively. This is an absolute requirement for exploratory testing and a good requirement for formal test processes as well.

• The scope of testing

Formal processes may have difficulty scaling to large numbers of tests, especially if they must be designed and performed in a short period of time. Informal processes are often used when there’s a lot of testing and not much time to perform the tests.

Informal Approaches

Remember that UAT is concerned with determining acceptance of software. An important decision must be made at the conclusion of UAT – “Should we accept the system or should some things be fixed or changed?” It’s one thing to test for the purpose of finding defects but another thing to test for the purpose of acceptance.

• Exploratory Testing

This is designing tests as you perform them. It is a rapid form of testing that is good for when you don’t know much about the application you are testing. Exploratory testing is not ad-hoc or random testing, based on reasoned hypotheses and assumptions. Exploratory testing can be used for UAT if it is managed well and the users performing the testing are able to think of good tests as they are testing.

• Agile Testing

While exploratory testing is often used on agile projects, there are other methods of acceptance testing aspect of agile projects. Agile acceptance testing is based on user needs, but may not always be performed by users. Sometimes agile UAT is automated. True UAT in the agile context should be designed to be more than just functional testing performed by testers or developers.

• Beta Testing

Beta testing is essentially asking end-users to try “pre-production” software in their environment to find problems. While defects can be found in this kind of testing, there are problems with the approach: 1) There is no assurance that the users will test anything, 2) The tests may be weak, and 3) The users may not be able to describe how a particular failure occurred, and 4) you live with the bugs and problems.

Formal Processes

Formal processes are designed to address the major challenges of UAT, such as complexity, large scope, little time and steep learning curves. It may appear that these processes are heavy and time-consuming, but the opposite is often true. The process I apply has been designed to plan, perform and evaluate a rigorous UAT effort quickly. Using this process in many projects, I have proven it can be applied quickly and effectively. The structured UAT pattern and process:

• Is based on what happens in the business or organization, not requirements or specifications,
• Decomposes major business processes, not system processes, into real-world scenarios
• Can test a person or thing from the beginning to the end of its life in the system.

Summary

The UAT pattern you select should match your project and organizational needs. The structured UAT pattern is an organized and effective way to design an acceptance test based on business processes. This pattern allows a non-technical user to start with an initial understanding of the business and the system to design a rigorous acceptance test.