TeamF::CGT	Main Page | About | Help | FAQ | Special pages | Log in
	Category: Documentation

Usability Testing

From TeamF::CGT

Contents 1 Abstract 2 Introduction 3 Usability Test Design 3.1 The Participants of Usability Test 3.1.1 Characteristics of Users 3.1.2 Number of Users 3.2 The Critera for the features to be tested 3.3 Measurement and Metrics 3.4 Scenarios 3.4.1 Characteristics of Scenarios 3.5 Questionaires 3.6 Data Collection 4 Usability Testing Process 5 Usability Tests 5.1 Toolkit Scenarios 5.1.1 Creating New Database 5.1.2 Creating Tables 5.1.3 Creating “Project” Table 5.1.4 Creating “Cost” Table 5.1.5 Adding “Name” as an attribute of “Project” Tables 5.1.6 Adding “Description” as an attribute of “Project” Tables 5.1.7 Editing an Attribute 5.1.8 Deleting an Attribute 5.1.9 Creating “Risk” Table 5.1.10 Adding “Exposure” attribute as a “linked attribute” 5.1.11 Adding “Project Name” attribute as a foreign key to “Risk” Table 5.1.12 Editing Table 5.1.13 Deleting Table 5.1.14 Creating Trees 5.1.15 Saving the information 5.1.16 Generating SQL Script to create the database 5.2 User Application Scenarios 5.2.1 Launching the software 5.2.2 Viewing Risks 5.2.3 Viewing the properties of Risks 5.2.4 Adding a Risk 5.2.5 Editing a Risk 5.2.6 Using the filter 5.2.7 Changing Appearances 6 Results of Usability Testing 6.1 Toolkit 6.1.1 Detailed Analysis of the Features 6.1.2 Major Usability Findings 6.2 User Application 6.2.1 Detailed Analysis of the Features 6.2.2 Major Usability Findings

Abstract

---

This document describes the measurement, test cases and results of analysis of usability tests.

Introduction

---

Usability Testing is conducted to study how the users feel about using the features of the software. Team F has designed and conducted usability to measure the user-friendliness of both Toolkit and the User Application to improve the features of the software.

Usability Test Design

---

The Participants of Usability Test

Characteristics of Users

The users undertaking the usability test should be completely unfamiliar with the software to be tested. It would be very desirable if they come from different background. The participants should be the typical users of the software. One of the participants of the software must be the client.

Number of Users

The number of users should be five. According to the claim of "Five users is enough", it is the most efficient number (Virzi, R.A., Refining the Test Phase of Usability Evaluation: How Many Subjects is Enough? Human Factors, 1992. 34(4): p. 457-468)

The Critera for the features to be tested

The usability test aims to test functions and features of the software that satisfiy the following critera:

• Important
• Frequently used.
• Considered hard for the users to master

Measurement and Metrics

The four main aspects to be tested are:

1. Time
2. Accuracy
3. Recall
4. Emotional Response

The measurements are subjective.

Time:

• How long did it take the user to complete the set task?
• Metrics:

1. Quicker: The user completed tasks quicker than expected.
2. Acceptable: The user completed tasks within the expected time. Time is acceptable when the user follow the menus and finish a task in staright forward manner.
3. Poor: The user took more time than expected to complete the task.

Accuracy:

• How many mistakes did the user make?
• How serious were the mistakes?
• Was the user able to recover from the mistake?
• Metrics:

1. Very High: The users did not make any mistakes.
2. High: The users made one or two mistakes.
3. Medium: The users made few mistakes.
4. Low: The users made many mistakes.

Recall:

• Was the user be able to repeat the tasks without referrint to the instuctions again?
• Were they able to recall how to perform the tasks after a period of non-use?
• Metrics:

1. Excellent: The users were able to recall without having to look at instructions.
2. Acceptable: The users were generally able to remember how to repeat the task.
3. Poor: The users did not remember how to do the tasks.

Emotional response:

1. How did the user generally feel while undertaking the task?
2. Was the user

• Comfortable
• Confident
• Indifferent
• Confused
• Stressed

Scenarios

Scenairos describe the instructions that participants of usability test are required to perform to use a particular feature of the software.

Characteristics of Scenarios

Short: The user shouldn’t be reading long instructions. They would lose interest. Long instructions makes it harder for them to follow.

Specific: The instructions must be stated precisely and unambiguously. Each scenario must have an aim, set of actions to follow and result in output that achieves that aim.

Realistic: The scenario must be the tasks that the users will be expected during the course of using the software.

Questionaires

Questionnaires are used to elicit information from the participants about themselves and their views on the usability of the software.

There are three types of questionaires:

The pre-test Questionnaire: The purpose of this is to collect information about qualification and background of the participant.

Post-Scenario Questionnaire: The purpose of the post-task questionnaire is to elicit the participant’s views on the task’s difficulty and to gather any other feedback.

Post-test Questionnaire: The purpose of this to understand participant’s overall perception of the system’s usability and understand any particular issues they have in relation to usability of the software.

Data Collection

• Video recording of the client
• Completed Questionaires

Usability Testing Process

---

1. Use the IDEA lab located in ICT building
2. Ask the participant to fill out the pre-test questionnaire
3. Make them feel comfortable.
4. Start Recording
5. Ask the participant to complete the tasks for each scenario
6. Ask the participant to complete the post-scenario questionnaire
7. Request the participant to complete the post-test questionnaire.
8. Thank the participant for doing the usability test and give them chocolate as a reward!
9. End Recording

Some processes to follow during the usability test:

• Do not tell the participant how to do some thing
• Do not explain to them the how the interface works
• Let them make mistakes.
• When the participant asks questions, do not provide direct answers. Let him/her try to figure out the answer by asking the participants questions.

Usability Tests

---

Toolkit Scenarios

Toolkit is software that is used to generate schema for a database in simple manner by adding tables and their attributes.

The following steps will guide you through how to create a basic database.

Creating New Database

1. Launch the Toolkit.
2. Create a New Database by selecting “New Blank Template”
3. Choose the Database system
4. Input the connection string as “DSN=CGT”

Creating Tables

Each database consists of one or more tables. Each table consists of many attributes. Each attribute describe a property of information in the table. Each attribute has a particular data type. It may reference an attribute in another table.

• All the options to create and edit tables and attributes are available in the “Objects” Menu.
• Alternatively, you could use the icons in shortcuts bar below the menu bar to add and edit tables and attributes.

Let’s create the first table.

Creating “Project” Table

1. The name of the table is “Project”
2. The table has two attributes:
   1. "Name”: The data type for this is “nvarchar (50)".
   2. "Description": The data type for this is “text".

1. Go to “Object” Menu and select “Add Table” to create a table.
2. Enter the name for table and set its permission as “User”.

Creating “Cost” Table

Add another table called “Cost”. This has no attributes at this stage.

Adding “Name” as an attribute of “Project” Tables

1. Click on “Project” table in the side bar to highlight it.
2. Click on “Add attribute”.
3. Enter the name of attribute.
4. Tick “Display Attribute”.
5. Select “Normal” and click on “Modify”.
6. Select the data type.
7. Add “Name” an attribute.

Adding “Description” as an attribute of “Project” Tables

1. Follow the above steps.

Editing an Attribute

“Description” should actually be called “Descriptions”. Click on “Edit Attribute” and change the name to “Descriptions.

Deleting an Attribute

It was decided “Descriptions” is no longer needed. Click on “Delete Attribute” and delete this attribute.

Creating “Risk” Table

Let's create another table called "Risks".

It has the following attributes:

• Name:
  • Data type: varnchar(250)
• Probability:
  • Data type: bigint
• Impact
  • Data type: bigint
• Exposure:
  • This attribute is a calculated value.
  • The values for this attribute are obtained by multiplying Probability and *Impact.
• Project Name:
  • This is a foreign key in “Risk” Table.
  • It references “Name” attribute in “Project” table.

Add the following attributes as done previously:

• Name
• Probability
• Impact

Adding “Exposure” attribute as a “linked attribute”

The values for this attribute are obtained by multiplying Probability and Impact. In other words, "Exposure = Probability x Impact.

1. Click on “Add Attribute”
2. Enter its name
3. Select “Calculated” and click on “Modify”
4. Click on “Row Operation”
5. Enter the appropriate “operation parameters”.

Adding “Project Name” attribute as a foreign key to “Risk” Table

This is a foreign key in “Risk” Table. It references “Name” attribute in “Project” table.

1. Click on “Add Attribute”.
2. Enter the name as “Project Name”.
3. Select “Linked” and click on “Modify”.
4. Select the values for “Table” and “Attribute”.

Editing Table

The marketing manager did not like the word “Project”. He wants you to call it “Cool Projects”

1. Click “Project”
2. Click on “Edit Table”
3. Change the name.

Deleting Table

1. Select the “Cost” table.
2. Click on “Delete Table”.

Creating Trees

Trees provide a way to organise information by defining the relationships between different tables.

1. Goto View -> View Constraint
2. Click Add
3. Give tree a name.

1. You need to create a tree, where "Cool Project" is the parent and "Risk" is the child.
2. Drag "Cool Project" on to "Tree Structure" Pane.
3. Drag "Risk" on to "Cool Projects"

Almost done!

Saving the information

1. Go to “File” Menu
2. Select “Save as” and save the file.

Generating SQL Script to create the database

1. Go to “File” Menu
2. Select “Export SQL”

That is it!

Thank you for testing our product. We appreciate your help and your input will help us improve the usability of our product. Thanks once again!

User Application Scenarios

This is web-based Risk Management Software used in a large corporation to manage the risks. This system contains information relating to the projects, risks and business processes to mitigate the risks.

Launching the software

1. Type in the following URL in a browser: http://spanky.riskwizard.com/spanky/UserApp.aspx
2. Enter username as “admin”
3. Enter the password as “test”

Viewing Risks

1. In the “Tree” panel, select “Risk”
2. Find out the Likelihood of the risk “Missing Deadline” occurring
3. What business process does the risk “Bad Design” relate to?

Viewing the properties of Risks

The properties of the risks are listed under a risk in the “Tree”

1. What are the cause(s) of “Bad Design” is?
2. What are the control(s) put in place for “Bad Design”? Which employee is responsible for it?

Adding a Risk

You have been asked to add a new risk. A new risk is added using “Add” option in the Menu panel. Cause, Control and Consequence properties are added by using “Add Child…” in “Add option”.

If the values for Cause, Control or Consequence property do not exist in the drop-down list, you need to add them by using the “Generic Cause” or “Generic Consequence” option in the Tree panel.

The details of the risks are as follows:

• Risk: Missing Deadline
• Consequence: 5
• Revised Consequence: 5
• Revised Likelihood: 5
• Likelihood: 5
• Business Process: Development
• Cause: Poor Implementation
• Consequence: Late Delivery
• Control Weekly Project Planning:

Editing a Risk

The risk manager conducted the weekly risk review. She has asked you to change the following values of “Missing Deadline” risk:

• Revised Likelihood: 2
• Revised Consequence: 2
• Causes: Poor Planning

Using the filter

The risk manger is analyzing the risks, she has asked to you to find out all the risks that have Revised Likelihood of 2.

This achieved by following steps:

1. In the Filter Panel, Click on “Risk”
2. Click on “Add”
3. Enter the appropriate values for the inputs
4. Save
5. Click on the created filter
6. Click on Apply

What are the risks that have “Revised Likelihood of 2?”

Changing Appearances

1. Go to “Appearances” Menu
2. Change the skin to “Longhorn”

That is all, folks! Thank you for completing this usability test.

Results of Usability Testing

---

Toolkit

The users described the overall perception of the Toolkit as “Good”. They generally found the Toolkit to be very user-friendly.

They found the software simple and easy to use. The users were very impressed with the "look and feel" and found interface to be consistent with standard Windows GUI.

Generally, the Toolkit performance on the four criteria as follows:

1. Time: Acceptable
2. Accuracy: Very High
3. Recall: Excellent
4. Emotional Response: Comfortable

Detailed Analysis of the Features

Feature	Time	Accuracy	Recall	Emotional Response
Creating New Database	Quicker	Very High	Excellent	At Ease
Creating a Table	Poor	Low	Excellent	Confused
Editing a Table	Quicker	Very High	Excellent	At Ease
Deleting a Table	Quicker	Very High	Excellent	Confused
Creating a Normal Attribute	Acceptable	Medium	Excellent	Confused
Creating a Linked Attribute	Acceptable	Very High	Excellent	Comfortable
Creating a Calculated Attribute	Acceptable	Very High	Excellent	Comfortable
Editing an Attribute	Quicker	Very High	Excellent	Comfortable
Deleting an Attribute	Quicker	Very High	Excellent	Confused
Creating a Tree	Acceptable	High	Excellent	Confused
Saving XML file	Quicker	Very High	Excellent	At Ease
Generating SQL Script	Acceptable	Low	Excellent	Confused

Major Usability Findings

The strengths of Toolkit

• Intuitive features
• User-friendly and well-structured user interface
• Very Colourful and eye-catchy splash screen and icons
• Ability to access via menu/toolbar/content menu and keyboard
• Easy to recall how to use features

The weakness of Toolkit

The following table explains the major usability issues and the recommendations to correct these issues.

Feature	Issue	User Action	Recommendation
Creating a Table	The users were confused when they tried to create a table for the first time.	The users did not know they had to click on the root item to activate menus for creating table. On average, it took a user two minutes to figure this out. Most users ended up clicking on "New" icon and trying a database again. Some users clicked on "Objects" Menu. It crashed the system.	The options for creating a table should be obvious. One way is to automatically highlight the root item in the panel, thereby activating the menus to create table. The software should not crash, when users click the menu items in "Object" menu. Fix this bug.
Creating a Table	The order of tables in the left hand side panel.	After a user created a table, the tables get rearranged in random order in the left hand side panel. It made it difficult to look for a particular table	Order the table in an alphabetical order
Creating a Normal Attribute	Selecting an attribute as a normal attribute	The users did not know they needed to click on "Modify" button to select datatype.	Rearrange the form such that "Attribute Type" is above "datatype" text"
Creating an Attribute	Selecting the datatype	The users spent some time looking through the datatypes as they are not in an alphabetical order. Most users did not realise they had to add edit the textbox below after selection when the datatype is nvarchar(250).	Sort the datatypes in alphabetical order Re-arrange the form for selection of datatype to make it easier. Put in an error message when the users do not put in numerical values when datatypes such as nvarchar is selected
Deleting a Table	No alerts when a user deletes a table	The users were surprised that there were no warnings when they deleted a table.	Warn the user when they perform an undesirable activity.
Generating SQL Script	Incorrect extension	When the users clicked on "Export to SQL", the default file name had .xml extension.	Fix the bug
Creating Table Editing attributes	Lack of alerts	When the users clicked on "New" and proceeded to create new database when they did not know how to create a table for the first time, there was no alerts about discarding the old copy. There is also no alerts when users modify attributes.	Revise the alert and warning messages.
Creating a Table Creating an Attribute	Navigation	Often the users found the tab order navigation difficult. They could not tab from one control to another in a logical order	Ensure a logical tab order is implemented across the controls.
Creating a Tree	Menu Item	The users were quite confused about the trees. The features were hidden from them before and after defining the trees	Perhaps have another panel that shows the tree structure
Toolbar	Navigation	Users found the dynamic toolbar quite confusing. Most did not realise that they had to highlight a table or an attribute in the left-hand side panel in order to access the options to edit these. They assumed since an attribute or a table was highlighted in the main panel(right hand side), they had access to the options to edit them. Most users stopped using the main tool bar and instead right-clicked on the items in the left hand side panel to edit the settings.	Review the menus and toolbar ordering and activations.

User Application

The users described the overall perception of the User Application as “Good”. They generally found the User Application to be very "cool" and quite be very user-friendly.

They found the software simple and easy to use. The users were very impressed with the "look and feel". They liked the colourful skins.

Generally, the User Application performance on the four criteria as follows:

1. Time: Acceptable
2. Accuracy: Very High
3. Recall: Excellent
4. Emotional Response: Comfortable

Detailed Analysis of the Features

Feature	Time	Accuracy	Recall	Emotional Response
Logging In	Quicker	Very High	Excellent	At Ease
Viewing the Information of Parent Table	Acceptable	Very High	Excellent	At Ease
Viewing Information of a Child Table	Quicker	Medium	Excellent	Comfortable
Adding Data to Parent Table	Poor	Very High	Very Good	Confused
Adding Data to Child Table	Acceptable	Medium	Excellent	Comfortable
Editing Information of Parent Table	Quicker	Very High	Excellent	Comfortable
Editing Information of Child Table	Acceptable	Very High	Excellent	Comfortable
Adding a Linked Attribute	Acceptable	Very High	Excellent	At Ease
Creating a Linked Attribute	Poor	Medium	Excellent	Confused
Creating a Filter	Acceptable	Low	Excellent	Confused
Using a Filter	Acceptable	Very High	Very Good	Comfortable
Removing a Filter	Quicker	Very High	Excellent	Confused
Changing skins	Quicker	Very High	Excellent	At Ease

Major Usability Findings

The strengths of User Application

1. "Look and Feel" of the User Application: In users words, "very professional looking" and "pretty damn awesome!"
2. Use of Icons
3. Use of tree strcture to represent information

The weakness of User Application

1. Orientation of Interface
2. Navigation
3. Functionality Grouping
4. Non-Intuitive Filter System

The following table explains the major usability issues and the recommendations to correct these issues.

User Application Example	Feature	Issue	User Action	Recommendation
Creating a new Risk	Adding data to parent table	The users were totally confused as to how to how to create new risk. They spent more than two minutes trying to figure how to add a new risk. They had to be shown to how to do it.	After the user had viewed one of the children of the parent table, in this case, "Weekly Design Review" under Risk, "Bad Design", the user did not know how to create new risk. The main reason for this, "Risk" did not appear under the "Add" Menu. They did not know that they had to click on one of the risks in Tree Panel, in order for the "Risk" to show under "Add" menu.	Re-organise the menus. Rather than accessing every thing through "Add" menu, perhaps one menu that provides all functions relating to the parent table, in this case, "Risk"
Viewing the Consequences, Causes and Controls relating to Risk, “Bad Design”	Viewing Information of the child tables of a parent table	When asked to find out about the consequences, causes and controls, the users did not realise they could click on the nodes of the children of the risk to view more information	The users did not click on the children of a parent in the tree to find out more information. They used the colour and perceived meanings of the icons to guess which one were the consequence, cause and control of a risk.	The colour and the representation of icons play a major part in user interaction and perceiving the meaning of attributes. Ensure the icons do reflect meaning of the attributes. The icons have significant impact on user perception.
Adding a "Cause" to "Risk"	Adding information relating to a child table of a parent	When user tried to add information relating to "Cause" by selecting "Possible Cause" in the menu, the list of "Consequences" were displayed instead of "Causes".	Could not add the cause to the risk.	Fix the bug
Adding a "Cause" to "Risk"	Navigation	When a "cause" did not show up in the drop-down list, the user had to go back and select "Causes" from the tree and create it. Then come back and add them.	The users became confused when they could not find it in the drop list. Once told, they need to add it, they were quite uneasy. It broke up their advances in progression. The users found this cumbersome.	The users tend to like advancing in progression. When that the progression is broken, they become annoyed. Perhaps construct menu option in the content page that allows them to add these values without the need for going back to add these values.
Saving, editing, viewing information	Navigation	It took about 0.5 second for the page to reload each time a value was added or to move one page to another.	Some viewers found it annoying.	In the future, when extending the software, only update the information that were changed rather reloading everything to speed up refresh rate. For example, AJAX.
Adding "Risk"	Using "Add" Menu	When clicked on "Add" Menu, the system crashes	Users accidently clicked on "add" menu number of times. The system crashed.	Do not allow "add" to be "clickable"
Using Filter	Filter	The orientation of filter on the screen	When asked to use the filter, the users did not know how to access. It was below the user's viewer. Eventually the users scrolled down the screen and found the filter panel.	The filter panel should up moved up.
Saving a newly created filter to view risks	Filter	The users did not save the filter	After the users created the filter, they clicked on apply rather than save	De-activate the "Apply" button when creating a fliter
Removing an already created filter	Use of Filter	Expected behaviour when a filter is removed	After a filter was removed, the users expected the tree to automatically update itself	Update the tree to include the effect of removing the filter
Viewing attributes of "Risk"	Orientation of “Risk Name”	The users disliked searching for information	The attributes should be positioned to make it easier depending on their importance.	Move up "Risk Name" to the top.
Orientation of Menu, Tree and Filter	Navigation	The interface is not coherent.	The users disliked scrolling down to see things.	Reduce the height of tree and move the filter up. All the important features must fit in a single view.

This page has been accessed 2,246 times. This page was last modified 07:35, 29 October 2006. Content is available under Attribution-NonCommercial-ShareAlike 2.5.