5 Tips - How to set-up a successful remote user testing
5 Tips - How to set-up a successful remote user testing
One of the frequently highlighted tips for a successful user testing is to do it in person. It has a lot of advantages to do user testing face-to-face. Such as that it is easier to connect to your participants.
A personal connection reduces the stress level and makes your participant more willing to share their thoughts and opinions on the test subject. Also, you clearly can see facial expressions and reactions, which can be helpful to evaluate if what a person is verbalizing really matches their feelings. During our testing sessions we regularly run into users that say that the system was easy to use, but during the test they struggled for 3 minutes to finish a certain task. It is because humans don’t want to seem incapable and feel the user test is also about testing them.
Anyway, there are many reasons why sometimes it is not possible to set up a face-to-face testing and we need to go for a remote testing session e.g. the spread of the corona virus or simply that your target group isn’t available in your surroundings. Especially in our globalized world it is beneficial to have an alternative to F2F. Also, there are some advantages to remote testing e.g. that your participants test your product in their real-life environment including the device they would normally use and distractions they could probably face. We found that some users really struggle first time they are using our “Testing Mac”, and remote testing erases this issue completely.
But remote user testing requires some additional thoughts upfront to make it successful and generate outcomes as insightful as those of a face-to-face testing session. That’s why we collected our top 5 tips on how to set up successful remote user testing sessions.
1. Deciding on the right tools
Especially in remote testing it is important to have a sufficient technical set-up that allows you to
Build a strong connection with your user
Document your findings
To build a strong personal connection we recommend using Google Hangouts or Zoom for remote sessions. Seeing your counterpart instead of only hearing the voice helps to get on a more personal level and also allows you to see their facial reactions. Further, you can easily invite them to a meeting and explain them how they can share their screen. Anyway, be aware that not all participants are able to use it. Especially, when you have the testing during worktime some of your participant might face software restrictions. For this case, it is important to clarify upfront if this solution is suitable for everyone. If not Google Hangouts or Zoom there are plenty of other tools and your participant probably, they already has an alternative in mind.
Remote testing offers perfect solutions for documentation. We highly recommend recording both screen and audio off each session. You can use both to analyze your session in detail later on. Of course, with the written permission of the participant upfront. To record the screen, you can use QuickTime. It also allows you to record a limited part of the screen. Unfortunately, QuickTime alone is enough to record the audio and screen at the same time. But there is program to help with that: SoundFlower. It allows you to record audio via multiple channels.
2. Setting up the testing sessions
As remote participants do not visit our office in person it is easy to neglect the planning part. Still, it is super important to professionally set up the meeting. You need to follow certain steps to ensure everything is going as planned.
You need to follow certain steps to ensure everything is going as planned.
Set a date & time with your participant & note taker (mind different time zones)
Check with your participant if they can use the tools you are planning to use.
Get their written agreement for screen & voice recording as well as data usage. Depending on client and project also an NDA will be needed.
Send your participant a calendar invite including the link to the tool you are going to use.
If you plan the meeting more than a week ago do a friendly follow up the day before. This avoids that you participant forgets about the meeting.
Book a quite meeting room and book it a least 5 Minutes before the meeting starts and 15 minutes after the meeting is supposed to end. So you have enough time to set everything up and do not need to quite the meeting before everything is said. You will realize quickly that some participants have to say more than others.
Have a way to reach the participant via email or phone in case there are issues, or you run late.
Be careful which information you give the users upfront. Less is probably more. If the system is already live and you want to test with new users, it would be a pity if they check out the system upfront out of curiosity and already know their way around.
If you mind all those points you are ready for take-off!
3. Prepare for technical fails
Nobody is perfect. Not you, not technology. Therefore, you should always have a plan B for technical fails. Nothing is worse than coming out of a successful testing and realizing that the screen recording failed. Similar suffering is to realize that your PC does not start 1 Minute before the meeting.
One way to avoid that those issues result in 0 testing outcomes and frustrated clients is to have an alternative tool for every tool you use. For example, you can also record the audio with your phone or switch from Google Hangouts to Skype.
Having a note taker in your session can also be a lifesaver in case your PC is deciding for a 30 Minutes update shortly before a call. Plus, in case all recordings fail he has the key findings already written down.
4. Set the right atmosphere.
To have the same quality as a f2f-testing one of the most important things is to create a warm atmosphere. It can be difficult to empathize with people over the web. Besides creating a digital f2f connection it is also important to take the time to thank them for their time, do some small talk upfront and get to know your counterpart a little bit better. Besides that it is especially important to let them know that everything that doesn’t work out is the systems fault – not theirs. Including this in your test on-boarding will help the participant to talk freely about their experience and thoughts during the testing.
5. Know when a remote testing does not make any sense
As mentioned above remote user testings can be a really good and time efficient alternative to physical testing. In some cases, however it is important to insist on face-2-face testings. This is the case if:
You need to observe the user’s environment/the testing is highly depended on a specific situation that cannot be recreated sitting in front of a laptop
You need to see how they interact with others or physical objects during the testing session
Your users are not comfortable with technology and will run into problems setting up the testing session on their side
For most products, apps and other software solutions these points are not applicable. But if you realize that those will influence the results of your sessions it makes sense to go back to good old f2f-testing and wait until the crisis is over.
Started by define problems
We reviewed the existing process to define improvement as the starting point of the lecture and setup 3 specific goals for our team to achieve:
Creating a common value/ standard for the design outcome.
Creating agreements to clarify the goal and what is included in the project.
Creating measurement to evaluate if the outcome is good enough.
We learn from other existing business model, which is similar with us as examples to do analysis. We tried to reference some of the similarity process to fit into our team situation and test if that would help us. By applying one of the working process mentioned by Tim Brown (IDEO) to measure if there is any missing parts would lead us to reduce mistakes and errors in our existing process.
Develop the process
Based on the existing situation, we develop 2 processes to fit into the design and development journey. Start from asking a specific question in order to find out the direction of the project as specifications to define what element needed to be measured, after that creating criteria to measure if the result is meeting the requirements.
Design Specifications - Provide explicit information about the requirements for a product and how the product is to be put together as a solution to reduce the misunderstanding between designer, product manager, and the client.
Design Criteria - Create measurements for evaluating goals that a function must achieve in order to be successful in order to improve the design and reduce the risk as much as possible.
Learn by practicing
We did a reflection of some previous projects by applying the new process and see if that would help us to improve the defined issues as an exercise. Therefore we would have a clearer understanding of the process by trying it and testing it.
After we defined the new process, the team will start applying it to every future project/task in order to improve the performance. It might be hard and need extra time to get used to it at the beginning, but we are happy to have a try after we spot the value from the exercise in order to provide a better product to our clients.