Usability Testing (M33121)
Level 6
Coursework Specification
As part of this assignment, you will design, run, and report on a usability evaluation study of a popular application/website. A list of popular apps/websites is available on Moodle. Please select one app/website that has not been selected by more than two students. This will be the app/website your study will focus on.
Your submission should include 3 parts:
1. The design of the study
2. A discussion of the results and their implications
3. An analysis of the threats to validity of the study.
Part 1. Study design
Part 1 should report on the design of your usability study, and it should be structured to cover all the topics below:
1.1. Usability goals. Identify 3 possible usability testing goals for the application/website you have chosen. Present these goals as succinct statements. Below, a few examples:
● Decide whether this app will be perceived similarly efficient by novice users and advanced users.
● Determine whether this app will encourage elderly users to book flight tickets online.
● Determine the extent to which this app helps students manage their finances.
1.2. Methodology. Identify the usability concepts you need to measure to be able to meet the goals defined for the study.
For each of the concepts identified, define metrics that can measure that concept.
Identify and discuss the methods and techniques that will allow you to capture the data you need to calculate these metrics. For each method, identify the data you will be collecting and the ways in which you will derive the metrics based on this data.
1.3. User profiles. Define your participants’ user profiles. Discuss the rationale behind every user characteristic chosen with reference to the goals defined for the study.
1.4. Procedure. Provide a step-by-step workflow of the testing session. For each step, identify who is involved, what is achieved, how much time is required, what possible risks can you identify and what are the plans to mitigate them.
Define one or more scenarios (maximum five) to be used during the testing session.
Discuss any requirements for material or equipment used.
Part 2. Results and implications
Part 2 should report on the data analysis, the results of the study, and the implications of the results. A minimum of 8 participants is required, but for statistically significant results, please aim to include 10-12 participants. Part 2 should discuss all the topics below.
2.1. Data analysis. Possible data collected may include video recordings, audio recordings, survey answers, interview data, etc. For each data type collected, define an analysis protocol. At least two data types need to be considered in the data analysis. At least one data type must be qualitative.
For each data type collected, apply the analysis protocol. Your submission should include a sample of each of the types of data collected and the specific analysis done on that sample. At least two data types need to be considered in the data analysis. At least one data type must be qualitative.
2.2. Results interpretation and implications. Discuss the results obtained in relation to each of the metrics defined for the study. Discuss the meaning of the results. Focus on any patterns identified in the data, and the possible reasons for them.
Discuss the implications of the results in terms of the goals set for the study and the design of future iterations of the app.
Part 3. Limitations and recommendations
Part 3 should report on the limitations of the study (i.e. threats to validity)) and their possible implications on the results, as well as any recommendations for improving the usability of the app/system evaluated. It should discuss the topic below.
3.1. Limitations and recommendations. Discuss any biases you think might have influenced your study. Focus the discussion on each design decision, and analyse what biases each decision had the potential to lead to.
Discuss alternative designs in answer to the usability issues identified as part of the study.