4 Tips for the ultimate remote UX test
We've probably all experienced it in recent months: prototype fantastically designed and animated, UX test scripts prepared to perfection, recruited respondents for your in-person interviews, test location and technology ready, and then ... Corona!
Remote moderated UX testing is of course nothing new, but in today's low touch reality this form of user research is more relevant and valuable than ever. We also experienced this in recent months and are therefore happy to share our pitfalls, success factors and our own learnings.
The first dilemma: testing on smartphones vs larger screens
At M2mobi we are mobile natives, so we always think mobile first. With remote UX tests, this is immediately your first challenge: Do you let your respondents view your mobile prototype on a large screen at home, or do you let them do this on their smartphone? And which device and camera setup provides the best insights?
After all, the question is: how can you really watch what is happening on their smartphone and how they respond to it? Of course you cannot install eye tracking tools with them or place multiple cameras to record their eye and finger movements, facial expressions and interactions.
Essential for your test to be successful is the right setup. In the image below we have visualized our favorite stack. We work with 1 interviewer and 1 observer / data logger as standard, so that one person can focus all attention on the test and test persons and the other can immediately record all findings and give feedback, if necessary. Our tip: use Slack (or any messaging or communication tool you prefer, like Microsoft Teams) to communicate possible points of interest to the interviewer during the interview so that you don't miss any of the test persons expressions and interactions.
Why which tool?
You can do your remote research with a set of free and relatively standard tools. Our favorites:
- Our favorite video conferencing tool is Google Meet. Why? It is browser based and can therefore be used by anyone (even without registering), it allows you to easily share screens, you can easily manage the composition of your windows, the video and sound quality are good and crashes are rare.
- InVision is one of our favorite prototyping tools for these kinds of tests, but choose your own favorite. It is essential that you can share your prototype in-browser and - if it turns out that you have made a mistake - you can adjust and sync it very quickly.
- Google Docs is the tool in which we maintain our test scripts and research notes.
- Slack is useful for keeping a line with your observer and potential audience during your interview.
- Via Google Forms you can - if desired - easily have test persons fill in additional information such as a System Usability Scale afterwards.
- QuickTime Player is a great free tool to record your sessions.
- A tool like uxcam can help you, for example, to record your interview and create heatmaps of the tests. THere are many alternatives as well.
Our positive experiences
It is important to report in advance: in the past period we mainly carried out explorative and comparative research. Our experiences are based on that. During the tests that we conducted, larger screens appeared to be sufficient to give a user a sufficient smartphone experience. After all, your prototype is visualized in a mock-up, making the user seems to make the mental switch from desktop to smartphone experience quickly. An additional advantage: when you ask your test persons to indicate with their cursor what they are looking at any given moment, you create a semi-eye tracking insight. Because their webcams are usually mounted directly above their screen, you also have a perfect view of their facial expressions.
Before performing unmoderated tests, especially if they are primarily aimed at, for example, conversion optimization, the question is whether the insights concerning smaller and larger screens are comparable.
In our first remote tests, we asked test subjects to use the prototype on their smartphones and share the screen directly in the Google Meet. After all, we expected that this purest user experience would also provide the purest insights. As it turned out, we had more trouble tracking what the user was doing, but in particular tracking their eye movements and seeing facial expressions. An extra device also turned out to be a considerable extra hurdle for many participants (feedback from multiple microphones and speakers, problems with sharing the screen and test subjects reluctance to share their smartphone screen).
- Always go for a setup with 2 screens
The large screen to watch your test persons, the small screen for your script and possibly Slack
- Create separate meetings per test person, instead of 1 long meeting
You want to prevent the early respondent from having to wait or refuse, because you still have to complete a delayed test with that verbose other respondent. Planning and timeboxing are of course essential, but make sure to create separate meetings, so that in case of emergency your colleague will do the introduction while you complete another test.
GDPR and the retention of personal data
It is of course quite standard to record your sessions. But why actually? Especially if you work with an observer or data logger, you will usually not look back at any interview. Do you still want to record your sessions? Keep in mind that you inform your respondents about this in advance, make sure that they give explicit permission and take the GDPR guidelines into account. After all, organizations may not keep data longer than necessary. Although there is no specific retention period for personal data under the General Data Protection Regulation (GDPR), you want to avoid having to tell your respondents after a data breach, for example, that you have handled their data and recordings carelessly. Our opinion: only record interviews when strictly necessary, ensure secure storage and delete the data (both the personal data of the respondents and their interview) immediately after you no longer need it.
Our 4 indispensable tips
1. Test and double check the technology
Subject your complete setup to a stress test at least once and try to empathize with your test persons. Believe us, you will encounter it all: faltering Wi-Fi connections, out-of-date browsers, a screen capture tool that captures the wrong screen and no sound, users who completely crash your prototype, et cetera. Our tip: opt for compatibility, make sure your test persons don't have to install any additional tools or create accounts and make sure you are prepared for the worst case scenario.
2. Prepare your respondents well
As easy as it is to sense, inform, coach and reassure a test taker in real life, it can be that difficult in a remote test. Start by emailing them - after the recruitment - with a clear explanation of what they can expect from the test and how they can prepare. Send them a final reminder a few hours in advance with the most important need-to-knows. Reassure them at the start of the test and go through the most important technical matters step by step: using techniques that are new to them may make them even more nervous than usual. It's up to you to make sure that everything runs smoothly on their end.
3. Assume that you are a remote help desk for your users
The best way to prepare is by keeping Murphy's Law in mind: "Anything that can go wrong, will go wrong". Faulty webcams, unstable WiFi connections, PCs that start installing updates halfway through a test, test persons who do not understand how to share a screen, you name it. Allow enough time for this in your script, make sure you are prepared for possible issues and be as clear, understanding and patient as possible.
4. Sharpen your senses
From behind your laptop, it is even more important to get a good feel for your test persons. Obviously you asked them to think out loud beforehand, but what if they suddenly stop? Do you notice that one frown, or did you just look at your other screen? As we said before, ask your test person to use his cursor as his "eyes" and to always think out loud, but above all, stay focused on all the small nuances in test persons' behaviors. This will make your life a lot easier.
Our conclusion: from now on only remote?
Believe us: Remote moderated testing - and specifically eplorative and comparative research - is easier and more valuable than you might think. In our opinion, it is an excellent alternative or addition to on-site testing. Why?
- The insights are usually of comparable quality.
- It is easier to recruit respondents: respondents currently prefer to stay at home.
- Lower costs, due to the lack of travel costs.
Do you have any questions or comments about this blog? Let us know! For now: have fun testing!