Usability Testing

This lab will make students learn the technique of evaluating interactive system’s design through user participation where the actual users are asked to interact with the system.

After completing this lab, students will be able to apply this technique to evaluate interactive system’s design. They will have a good insight about:

• How to evaluate design with
• What are the different steps involved?
• When to evaluate design with
• Benefits and drawbacks of evaluation with users

Introduction

A popular and effective way to gather information, about actual use of a system, for evaluation purposes is to observe users interacting with it. In this lab, you are going to evaluate an interactive system’s design through user participation where the actual users are asked to complete a set of predetermined tasks and the evaluator watches and records the users’ actions. Such an approach is termed as empirical in which evaluation is based on observation or experience rather than theory or pure logic. In this approach, a real environment is simulated to conduct user tests. Before conducting them, a careful planning is required to make sure that conducted tests will be free of biases and representative of future interactions.

When to Evaluate with users

It is very common to perform user testing after evaluation with experts because that helps us not waste our users on stuff that’s just going to get picked up automatically. We want to focus the valuable resources of user testing on stuff that other people wouldn’t be able to pick up on.

Benefits and Drawbacks of Evaluation with Users

Experts walking through your system can generate false positives that might not actually happen in a real environment while this does not happen in evaluation with users and it is going to be more accurate.

One of the things that we see is that heuristic evaluation can often be a lot faster — It takes just an hour or two for an evaluator — while the mechanics of getting a user test up and running can take longer.

The heuristic evaluation results come pre-interpreted because evaluators are directly providing us with problems and things to fix while it takes more time to infer from the user tests what might be the problem or solution.

To explain how the evaluation takes place, let us consider Google map application and apply this technique to evaluate it. Since google map offers a lot of feature so it would not be possible in this lab to apply this technique to evaluate all of them. We, thus, focus on following three features:

• Street View
• Finding a place of interest and its related information near some location
• Adding an intermediate destination between source and destination and the “Avoid tolls” option.

Activity 1: Tasks design

To evaluate the ergonomics of the Google Maps for the aforementioned features, we need to design tasks for each of them. These tasks (given below) would eventually be given to actual users.

• Center the Eiffel Tower in Street View
• Find out the color of the signage of any Chinese restaurant in/near Orsay, France
• Find a path from Orsay to Orléans, via Chartres, without paying tolls

Activity 2: Determining Criteria

Criteria refer to those characteristics of the population that can influence the way product is used. We need to figure out different criteria for evaluating the Google Maps.

Solution

We figure out five different criteria for evaluating the Google Maps. These are sex, age, expertise in Google Maps, expertise in Computers, and expertise in Paper Maps.

Google Map Expertise: Expert users will be more familiar with Google Maps features and be able to respond quickly to the unseen/seen scenario.

Computer Expertise: People using computers regularly would be able to adapt quickly to the Google Maps application than ones who use them rarely.

Paper Maps Expertise: People who are experts with the paper map would be more comfortable in visualizing the maps as compared to others. They may be tempted to use the Google Maps in a different way compared to others.

Activity 3: Determining Context

Context refers to those characteristics of the environment that can influence the use of product. We need to come up different contexts that can affect the use of Google Map.

Solution

We come up with three different contexts that can affect the use of Google Map. These are:

Screen size: Bigger screen size makes navigation more efficient.

Time stress: Stressing users by limiting the time can affect the way they perform the task. Some may become more efficient and perform it quickly while other may panic.

Mouse with scroll wheel: Scroll wheel permits fast zooming in and zooming out and external mouse is usually considered more efficient than the laptop’s touch pad.

Activity 4: Choosing Participants

A rule of thumb is to have 6 persons per criteria and per context. We must choose our users carefully. They must be minimal in numbers and at the same time satisfy the rule.

Ideally 18 carefully chosen persons can satisfy that rule for the criteria. The following table shows the sample of population. We can see that carefully chosen 18 users do satisfy the rule.

Table 5.1: Sample of Population (“X” indicates that the user satisfies the criterion)

User ID	Sex		Age			Expertise in Google Map /Computers			Expertise in Paper Maps
	M	F	<30	30-60	>60	Yes/Yes	No/Yes	No/No
1		X	20				X
2		X	22			X
3	X		27					X	X
4	X		29			X			X
5	X		29			X
6		X	24				X
7	X			30		X
8	X			32				X
9	X			35				X	X
10		X		43			X
11		X		51			X
12	X			33			X		X
13	X				77	X
14	X				80	X			X
15		X			61			X

16		X			65			X	X
17	X				62			X
18		X			61		X		X
Total	10	8	6	6	6	6	6	6	7

In the case of Contexts, it is easy to satisfy that rule because a context is not a characteristic of the population and it can be controlled by evaluators during the tests.

Table 5.2: Contexts: Sample of Population (“X” indicates that the user satisfies the criterion)

User ID	Screen Size		Time Stress	Mouse with Scroll Wheel
	14” Laptop Screen	5” Mobile Screen
1	X			X
2	X
3	X		X	X
4	X
5		X	X
6		X
7		X	X
8	X			X
9		X	X
10	X		X	X
11	X		X
12	X			X
13		X
14		X	X
15	X		X
16		X
17	X			X
18		X	X
Total	10	8	9	6

Activity 5: Usability Experiment

Once we have chosen the users, we are ready to perform usability experiment with them. During the experiment, we should observe the participant, and take notes of interesting events, such as difficulties encountered. We may also perform video recording so that we can view it later in case some events get missed by the evaluator.

Activity 6: Post Experiment Investigation

Once the experiment has finished, we can ask user to fill out a questionnaire inquiring about user’s experience with the system. It also often happens that we want to know about why user performed a specific action. So, this is the time to inquire this.

Given below is an example of questionnaire to inquire about user’s experience.

Questionnaire

Task 1:

• How many different ways do you know to see the Google maps in street view? Which ones? Which one do you prefer all the time?
• How would you rate Google map in terms of its “ease of use” for achieving this task? (1-very easy ~ 5-very difficult)

Task 2:

• Do you know any other way to do this task?
• How would you rate Google map in terms of its “ease of use” for achieving this task? (1-very easy ~ 5-very difficult)

Task 3:

• Did you know about the “Avoid toll” option in Google maps? What is your opinion on the placement of “Avoid toll” option in Google maps?
• How would you rate Google map in terms of its “ease of use” for achieving this task? (1-very easy ~ 5-very difficult)

Activity 7: Analysis based on Observation and User’s suggestions

Street View Option

There is no explicit mentioning of Street View option on the Google Map interface. There is a Pegman icon for a Street View but the problem with Pegman is its very low affordance. It does not express what action it supports. Users may also get to see the street view option when they search for a location and after that it requires some discovering (i.e., you need to click on location and hover over the related photo shown normally on the left of the screen).

Adding a destination/stopover: Users want “Add a destination” option to be readily visible.

Avoid toll option: The “Avoid toll” option is not directly accessible. To explore it, one needs to dig into the “Show Options”. Users want this option to be directly accessible on the interface at least when they find an itinerary for car.

Recommendations

There can be two contradictory opinions about the placement of “Avoid Tolls” option. Some would argue because people use it rarely, it is well placed. However, for someone using it for the first time, it is very difficult to figure out where it is. Our proposal is to make the avoid tolls option available near the text “The route has tolls”. It can be either in the form of link placed near the text or the togglable text. Users, already familiar with the placement of “Avoid Tolls” option, would still be able to quickly find the route by checking the check box under the “Show Options” while at the same time the option will be readily available and directly accessible for the novice user.

Figure 5.1 Avoid tolls option

Home Activities

Figure 5.1 Avoid tolls option

Activity 1: Choice of System to be Evaluated

The application, system, or device should be similar, or at least related in some sense, to the topic your group selected as Final Year Project. This will help in identifying design issues a similar system possesses and lead you to design better product compared to the competitor.

Activity 2: Tasks Design

Since the selected system may offer a lot of feature so would not be possible to apply this technique to evaluate all of them. You would therefore identify 3 to 5 features of the system and design tasks for them. These tasks would eventually be given to the actual users.

Activity 3: Determining Criteria

The users should be representative of the “typical” users of the system or device. You will need to figure out different criteria for user selection.

Activity 4: Determining Contexts

You will need identify different characteristics of the environment that can influence the use of product.

Activity 5: Choosing Participants

A rule of thumb is to have 6 persons per criteria and per context. You must choose your users carefully. They must be minimal in numbers and at the same time satisfy the rule of thumb.

Activity 6: Usability Experiment

During the experiment, you should observe the participant, and take notes of interesting events, such as difficulties encountered, frustration and annoyance, or surprise. For complicated experiments, it is advisable to have an observer and a facilitator; the role of the latter is to assist the participant if necessary. After the experiment, take some time to write down observations about the behavior of the participant and properties of the interaction between participant and device. After the experiment, you can also ask the participants questions about their experience, or give them a brief survey.

Activity 7: Analysis

Finally, you would recommend the possible improvements based on the observations you record during experiment and user’s suggestions.

Assignment Deliverables

You need to submit a report containing the following items:

• Small description of the chosen system
• Chosen features and the corresponding tasks
• Criteria, along with sample of population table
• Contexts along with sample of population table
• Observations recorded for each user
• User’s Feedback/Suggestions obtained through interviews/questionnaires
• Recommendations to improve the usability of the system