Client Project - 2022
Focus: Advanced Usability Evaluation
Duration: 6 Weeks
Project Overview:
This project was the outcome of a collaborative effort between Loto-Quebec and HEC Montreal. Loto-Quebec entrusted us with the task of evaluating the user experience of their mobile lottery app. In response, I, alongside a team of master's students, came together to create an experimental design aimed at addressing usability issues and helping enhance their platform. The information shared below will be restricted to ensure the confidentiality of Loto-Quebec's proprietary details.
Project Timeline
Project Mandate
Purpose: Loto-Québec’s lottery mobile app is the hub from which users can validate tickets and purchase a multitude of online games. Their goal is to promote digital adoption, so users convert from buying paper tickets to purchasing them through the app.
Objective: Evaluate the UX dimensions of learnability, effectiveness, utility, and satisfaction across the core elements of the typical customer journey when using the app.
Research Question & Hypothesis
Following a meeting with the Loto-Quebec team, we conducted a comprehensive group analysis that complemented the insights shared by Loto-Quebec, while also conducting our own evaluation. Subsequently, we formulated a test plan consisting of four research questions and a central hypothesis centered on the pivotal interactive steps users undertake throughout their customer journey.
Ticket Scanning:
Q1: What is the impact of displaying a different message (after scanning the barcode of a physical ticket) on users’ intention to convert to buying tickets on the app next time, compared to the status quo message?
Adding Balance:
Q2: What are the users’ next intuitive steps after (supposedly) completing their app registration?
Q3: What obstacles are preventing users from successfully and efficiently adding funds to their account credit?
Ticket purchasing:
Q4: How easily and successfully can users purchase an online lottery ticket on the app?
Hypothesis:
Displaying a different, more incentivizing message (after scanning a physical ticket) has a positive effect on users’ intention to repurchase their next lottery ticket through the app (instead of buying a traditional paper ticket in-store), compared to the current message in the app.
Recruitment of Paticipants
For our recruitment process, we opted for convenience sampling, a method chosen for its practicality and ease of access. To minimize the impact of social desirability bias, we took an additional precautionary measure. During the research sessions where a participant had been recruited by a particular moderator or notetaker, that moderator or notetaker purposefully refrained from attending. This separation aimed to create a more impartial and unbiased environment for data collection and analysis.
Inclusion Criteria:
Exclusion Criteria:
18+ years
Proficient in English
Located in Québec
Have access to the internet
Have a microphone and webcam
Have screensharing software (Zoom)
Not passing the screening questionnaire provided by Loto-Québec.
Demographic Summary: We had a total of 16 participants, 9 men, and 7 women ranging from the ages of 22 to 57.
Experimental Design
Once we had completed the finalization of our test plan, we embarked on the development of an experimental design, meticulously crafted to facilitate the testing of our research questions and hypothesis. Given the challenges posed by the COVID-19 pandemic, we made a collective decision to implement a remote experiment. This experiment comprised three distinct tasks that simulated the natural actions users would undertake when interacting with the mobile app. Employing a within-subject design, all 16 participants were exposed to each of the experimental conditions to ensure comprehensive data collection and analysis.
Scenarios & Task Measurement Tools
The design experiment encompassed a comprehensive scenario involving a Loto-Quebec user's journey. In this scenario, the user's first task was to scan a previously purchased ticket using the app to determine its status (winner or loser). Subsequently, the user was prompted to add funds to their app balance and proceed to purchase a new 6/49 lottery ticket. To facilitate a detailed analysis of the user experience, we deconstructed this scenario into three distinct tasks.
We employed various scientific measurement tools to gauge the overall scenario's effectiveness, including the assessment of participants' mental models. This evaluation was conducted using the Customer Satisfaction Score (CSAT), Customer Effort Score (CES), and the Net Promoter Score (NPS) to provide a comprehensive understanding of the user experience and to identify areas for potential improvement.
Final Report
Below is an excerpt from the final report that was presented to Loto-Quebec, encompassing the analysis and recommendations. Please note that certain details have been withheld for professional confidentiality reasons.
