This section describes the recommendation developed by TFU that have been disseminated for the WP leaders and WK partners for adaptation of the actual evaluations and testing in particular context of the uses of the tools. First, the recommendations are described. Second a list of links for further studying of the different methods is enclosed.


The following figure (see figure 1) is a picture that gives an example of how different methods could be situated in the development cycle. The cycle was created to help planning the evaluation processes for a particular tool and context. Some of the methods mentioned in the cycle can be used in many phases of the development cycles some other are more tied for example, to a functioning tool. The evaluation/testing cycle and the templates referred to at the end of the section are meant to help with the planning, executing and reporting of the evaluation(s) and test(s). Most of the methods and the templates fit more to the laboratory kind of testing while field trial evaluation and testing needs more adaptation of the methods. The research plans of the WP8, 9 and 10 are not presented here although the data acquired from the former mentioned research also often contains usability related data. The usability related data can be “tagged” so that it is available for the technical partners (see below the feedback loop).
Figure 2 also presents points where the evaluation cycle overlaps with the process-model of technical working knots
Figure 1 Test plan recommendation and process model for technical working knots

Basic guidelines for planning usability testing were provided in the form of “steps to take. Below the three basic steps to be taken for usability testing are presented.

  1. The organization of the test, e.g. making the test plan.
  2. Execution of the test.
  3. Analysis of the data and writing the test report.

[1]More detailed steps in the end of the document.

Below are links to websites that describe the different methods.

1. Work-shops:
Requirements/ functions meeting /GUI design
Claims-analysis (- use search for an example of claims analysis)

2. Heuristics:
Using heuristics

3. Prototyping:
Rapid prototyping

4. Pluralistic walkthroughs:
Riihiaho walkthrough
Groupware walkthroughs

5. User testing ("lab")
Using users

6. Field experiment (Close co-operation with WP8 and WP3 - "evaluation of the practices").
General methods:

Task force on usability also provided templates to enable easier note taking and reporting. The templates should be adapted in order to fit better to the laboratory kind of usability testing.

Link to the note takers template (more fitted to lab-testing)

Link to the usability report template (meant to be modified according to the needs)

[#1]Steps for the planner and executer of the test to take
(use if needed. This is a guideline not an obligation)

Step 1: Organising and creating the test plan
One should be familiar with the product/tool/service that is going to be tested. Therefore, it is important to get to know the product/tool/service. It is also essential to think about what issues need testing, as well as why test them. After this step, it is easier to sketch the test plan and how the process will be conducted.

We would move “Step 5” in second position here. The entry point shall be the description of a scenario, split into small pieces (elementary tasks with precise objectives). Then it is easier to insert test cases in this “roadmap” and follow the different steps below for each test case that is “inserted” in the script.

Step 2: Setting the goals of the test
One should have a clear idea of the goal of the test. In order to do so, it is imperative to list and define the goals of the test and the type of data that is expected from the test. The aforementioned list and definitions of goals influence the choice of the method(s) employed.

Examples of some general goals are:

  • Common usability
  • Suitability for the purpose of use
  • Ease of use in the first time
  • Longitudinal use experiences (learnability)
  • Etc

Step 3: Defining the usability requirements
The usability requirements that are set for the tool/service/product should be known and seen that there is no contradiction with those criteria.

Step 4: Choosing appropriate test group
The number of users depends on the resources, goals and type of the test. It is also important to see that the users are representation of the actual users of the tool/service/product and they should have been involved in the development process.

Step 5: Defining the test cases or tasks and the context (situation)
There are two essential insights into this:
  1. If it is a field trial, then context is already defined, but the situation that is suitable for the test has to be selected.
  2. If there is no possibility for an actual situation, then the test case or task has to be described so that the task is tied to some meaningful goal that the user might have. Usually it is a story that describes the background of the situation and what the user should drive for to reach their goal.

Step 6: Choosing the method of testing
What method is chosen depends on the data that is wanted, what is the phase (e.g. actual use surroundings are possible), what are the recourses, and availability of end-users. (Sample of test methods can be found below).

Step 7: Piloting the test
The test itself has to be tested with a small group of end users. The same persons should not be used in the actual test. Most often some miss thinking is found in the pilot test, e.g., video cameras not in the right place, the task description is not good, the data that is gotten is not the right kind of data.

Step 8: Knowing the tool/product/service

It is imperative that people who are planning the test know the tool/service/product, its features and functions and envisioned use.

Step 9: Writing the report
The data of the test has to be analysed after which a report of the test has to be written. It is also good to evaluate and prioritise the findings with respect to which ones are “severe” and have to be corrected as soon as possible and which ones are not so problematic. A brief list of what should be in the report (for more detailed description see the template in the appendix 5): a description of the user interface, a short description of the task (story) that was used, a description of the test group, an explanation of the test method(s) used, the reasons why that particular method was chosen should be presented, the results of the test and finally the conclusions of the test.
(Sinkkonen I, Kuoppala H, Parkkinen J, Vastamäki R. 2002).

Sinkkonen I, Kuoppala H, Parkkinen J and Vastamäki R. (2002). Käytettävyyden psykologia. Edita: Publishing Oy/IT Press.

This page is a category under: usability and under Category Of Recommendations


recommendationM12.jpg Info on recommendationM12.jpg 355691 bytes
  Page Info My Prefs Log in
This page (revision-3) last changed on 18:24 25-Mar-2017 by merja.

Referenced by

JSPWiki v2.4.102