POKEMON GO USABILITY TEST

Testing Pokemon GO’s UX and Its Effect On Players

TIMELINE

Nov - Dec 2023

TOOLS

Microsoft Office Suite, Figma

ROLE

User Experience Researcher & Test Moderator

Overview

For my final capstone project in my Usability Testing class, I teamed up with three other designers to analyze the usability of the Pokémon GO interface. It was actually Christine Taylor's brilliant idea (check out her portfolio), and being a massive Pokémon and Pokémon GO fan, I couldn't resist joining in. Our main goal was to dive into why there's been a consistent decline in the number of active and new users. We wanted to explore if any UX-related issues could be contributing to this trend.

Could UX-Related Issues Be to Blame For Pokemon GO’s Downfall?

The first thing we did was create a plan for our process:

  1. Discover the main pain points of the app

  2. Outline a usability test outline consisting of a task list and script

  3. Conduct usability tests with at least five different participants

  4. Create a formal report and presentation detailing our findings

Process

Heuristic Evaluation

Where Does the Interface Fail?

Once we had our process in place, it was time to jump into the research phase. We fired up the Pokémon GO app and started exploring for any potential hiccups. After tinkering with the app for a while, our team decided to conduct a heuristic evaluation using Jakob Nielsen's 10 Usability Heuristics. Our aim was to uncover the major pain points of the app and pinpoint any interface features or processes that may have contributed to its decline over time. This way, we could create a solid task list for our usability test.

Based on our initial run-throughs we found that Pokémon GO breaks these heuristics frequently:

  • Heuristic #6: Recognition Over Recall

  • Heuristic #9: Helping Users Diagnose and Recover From Errors

Discoveries

Task List

What Will We Have Our Particpants Do?

Once we wrapped up our initial research and heuristic analysis, we moved on to creating a task list for our live usability test. We came up with a total of fifteen tasks, with eleven tasks for the first part of the test where users started fresh with a new account, and four tasks for the second part of the test, which focused on a more advanced/higher-level account. This way we would be able to test the app on both basic and advanced levels.

Usability Test Task List

Participants

While we are coming up with the task list we also were recruiting participants. Since we were under a time contraint and we had a lot of cool friends, we decided to ask people we knew to participate in our test.

Who Would Be Taking Our Test?

To gather crucial participant information, we put together a pre-test questionnaire that covered important aspects like age range, familiarity with the Pokémon franchise, familiarity with Pokémon GO, tech savviness, and more. This allowed us to ensure we had a diverse pool of participants with varying levels of experience. It was a great way to gather all the necessary details upfront.

Pre-Test Questionnaire

Now that we had a good group of participants, I decided it would be a good idea to assign each participant with a Pokémon that we will use as their alias. Below is our “Particidex”, which shows the participants and their corresponding Pokémon GO skill levels.

Test Participant Information (“Particidex”)

Moderation

Who Will Be Moderating the Test? Me.

Before we could begin testing we had to prepare by assigning everyone in the group a role for what we would be doing during the test. The group elected me to be the moderator during the tests. 

During a usability test the moderator acts as a facilitator and guide, creating a supportive environment for participants to provide their feedback and insights on the usability of the product being tested. This meant that I would be the one handling all of the interactions with the participants. To ensure that everything would run smoothly, I created a script of what I would say and do while facilitating the usability test.

Test Script

Moderator Script Pages 1 & 2

Usability Test

Lack of Error Feedback, Navigational Issues, and Unintuitive Shortcuts.

With our task list finalized, our participants selected, and other preparation completed, we were all set to dive into the live usability testing phase. With the university lab rented out, our tripod set up, and five freshly made Pokemon GO accounts made, we began testing.

After one round of pretesting with our groupmate Spencer and five successful usability tests, we were ready to see what Pokemon Go was doing wrong.

Here is what we found:

For a game that relies on a GPS signal to play, you would think it would tell the user when their signal is bad. Turns out, this isn’t the case. During the initial phase of the test, several participants encountered difficulties in catching Pokémon due to their avatars being inconsistent with their actual location. Participant 01 (Bulbasaur) and Participant 05 (Cyndaquil) experienced avatars that were wandering aimlessly on the screen instead of remaining stationary. Instead of informing the user that their signal was bad, it asked them if they were driving. This caused confusion and frustration among these participants.

Lack Of GPS Error Feedback

“It thinks I’m a passenger, but I’m definitely not. Not even moving”

- Bulbasaur

“It’s just running and there’s no pokémon spawning”

- Cyndaquil

Interestingly, both of the beginner participants heavily relied on the PokéBall menu to accomplish almost all of the assigned tasks. They tended to visit the profile and nearby buttons much less frequently, usually acting as a last resort to complete specific tasks such as favoriting a postcard, checking nearby raids, or reviewing the time they caught their first Pokémon. They believed that it would be easier to navigate if more specific tasks were put under the PokéBall menu along with the more common tasks.

Navigational Problems

“I wish they put more things in the Poké Ball button, as it’s the easiest way to interact with the interface”

- Charmander

“I think raid should be under the Poké Ball button, because I didn't really know about the nearby button so I would have just looked around on the screen to see them”

- Squirtle

Unintuitive shortcuts: healing & transferring

We found that while it is possible to heal and transfer multiple Pokémon, the process is rather challenging. The advanced and intermediate players were able to take advantage of the press and hold shortcut for healings and transferring multiple Pokémon, but the beginner players had to resort to the strenuous task of hopping back and forth between menus to achieve the same results.

While gifts were not part of our assigned tasks, Participant 04 (Chikorita) expressed their frustration with the friend menu and highlighted the inefficiency of sending and opening gifts, particularly for players with a large number of friends. Unlike the transfer and healing shortcuts we mentioned earlier, there is currently no option to send or open multiple gifts simultaneously. Participant Chikorita expressed that this was a huge time sink for long-time players.

Bulk Gift Sending

“Opening and sending gifts is hard. I have like a hundred friends, so opening and sending gifts to a lot of people will take me over ten minutes.”

- Chikorita

SUS Scale

The Huge Disparity Between Beginner and Advanced Players

At the very end of the live testing, after the participants had a thorough interaction with the app, we had them rate their experience on a SUS Scale. A System Usability Scale (SUS), is a ten-item questionnaire based on the Likert Scale, that is used gather feedback on interfaces. Participants were asked to rate each statement, ranging from Strongly Disagree to Strongly Agree. While the SUS may not be the ultimate tool for assessing interface quality, we were intrigued to see how the scores would vary between the participants.

After tallying up the scores we found that our participants had diverse scores when it came to rating Pokémon GO. The experienced players gave the game exceptionally high scores, finding the interface intuitive, easy to learn, with only a small amount of issues. On the other hand, the beginners found the app more challenging and acknowledged that they would need to invest far more time to grasp its functionality. This discrepancy in scores highlighted the differing perspectives and experiences of players with varying levels of expertise.

Expert player mean score: 92.5

Beginner player mean score: 55

System Usability Scale (SUS)

The stark contrast between the ratings of expert and beginner players highlighted the distinct perspectives through which the app is perceived. The high ratings from expert players can be attributed to their extensive experience of playing the game almost daily for years, resulting in a strong familiarity with the interface. On the other hand, beginner players found the app overwhelming and lacked the enthusiasm to continue playing due to the initial issues they encountered. This disparity underscores the importance of considering the needs and experiences of different user groups to enhance the overall user experience of the app.

This could also explain why the new user player base for Pokémon Go has been steadily declining over the years. Since the interface is built around pleasing long-time players, it frustrates new players and keeps them from being able to get into the game.

SUS Scale Analysis

Recommendations

Okay, But How Do We Fix These Issues?

As previously mentioned, during our research we uncovered various issues with the Pokémon GO interface. Being able to find an issue is one thing, but being able to fix them is another. So here are some of our recommendations to help solve these issues:

Players were given little direction when starting the game causing beginners to struggle. Let players choose how much help they get.

#1 Add an option for in-depth onboarding

If the GPS signal is bad, tell the users. This lets players know they need to find better signal to keep playing.

#2 Give better error feedback

More features should be found in the PokéBall menu as well as their original locations.

#3 Include features in multiple locations

#4 Allow users to heal Pokémon on its individual page

Reduce the amount of taps it takes to heal Pokémon by adding a button that allows you to heal a Pokémon on its individual page

Recommendation #4

Reflection

What Did I Takeaway From This Process?

Throughout the process of conducting usability testing on Pokémon GO, I've had some major personal takeaways. Seeing the stark contrast in ratings and feedback between expert and beginner players has made me reflect on the importance of interface familiarity and onboarding support. It's clear that long-time players find the app intuitive and seamless, while newcomers often feel overwhelmed. This has reinforced the need for clear guidance and a smooth learning curve for new users. I've also learned the value of effective evaluation methods and participant feedback in driving design improvements. By addressing pain points and simplifying complex features, we can create a more inclusive and enjoyable user experience for all players.

Being the moderator for the usability test was also a major eye-opener for me. It came with a lot of responsibility as I had to ensure everything ran smoothly and that my instructions didn't unintentionally influence the participants' responses. It made me realize the importance of maintaining objectivity and creating a comfortable testing environment for unbiased feedback.

~ fin ~

Like what you see? Check out more of my work ↓