21 Jun How pilot testing can dramatically improve your user research
Today’s put up is all about how to make sure that your user research, a crucial element of any design toolkit, is as dependable as attainable.
Quality user research lets you generate deep, significant user insights. It is a key data supply inside WiderFunnel’s Explore section, and can be used to generate strong experiment hypotheses.
Unfortunately, conducting user research isn’t at all times as simple because it sounds…
Do any of the next sound acquainted?
- During research classes, your contributors don’t perceive what they’ve been requested to do,
- The phrasing of your questions has given away the reply or has brought on bias in your outcomes,
- During your checks, your contributors are unable to finish the assigned duties within the time supplied,
- After conducting contributors classes, you spend extra time analyzing the research design slightly than the precise outcomes.
If you’ve got skilled any of those, don’t fear. You’re not alone.
Even essentially the most seasoned researchers expertise “oh-shoot” moments, after they understand that there are flaws of their research strategy.
Fortunately, there’s a solution to considerably scale back these moments: It’s known as pilot testing.
Pilot testing is a rehearsal of your research examine, permitting you to check your research strategy with a small variety of check contributors earlier than you conduct your predominant examine. Although that is a further step, it might be the time finest spent on any research challenge.
Just like correct experiment design is a necessity, you will need to take the time to critique, check, and iteratively improve your research design, earlier than the research execution section. By doing so, you can be sure that your user research runs easily, and dramatically improve the output from your examine.
And one of the best half? Pilot testing can be utilized to all sorts of research approaches, from fundamental surveys to extra complicated diary research.
Start with course of.
At WiderFunnel, our research strategy is exclusive for each challenge, however at all times follows an outlined course of:
- Developing an outlined research strategy (Methodology, Tools, Participant Target Profile)
- Pilot testing of research design
- Recruiting certified research contributors
- Execution of research
- Analyzing the outputs
- Reporting on research findings
Each a part of this course of can be mentioned at size, however, our focus right this moment is on pilot testing.
You ought to at all times start any research with the high-level query: “What are we aiming to learn through this research?”. You can use this query to information the event of your research methodology, choose research instruments, and decide the participant goal profile. Pilot testing lets you rapidly check and improve this strategy.
WiderFunnel’s pilot testing course of consists of two phases: 1) an inside research design overview, and a pair of) participant pilot testing.
During the design overview, members from our research and technique groups sit down as a bunch and spend time critically fascinated about the research strategy. This entails reviewing:
- Our high-level objectives for what we’re aiming to be taught
- The instruments we’re going to use
- The duties contributors shall be requested to carry out
- Participant questions
- The research participant pattern measurement, and
- The participant goal profile
Our workforce usually spends a whole lot of time discussing the questions we plan to ask contributors. It can be tempting to ask contributors quite a few questions over a broad vary of matters. This inclination is usually as a consequence of a worry of lacking the invention of an perception. Or, in some instances, is the results of working with a big group of stakeholders throughout completely different departments, every attempting to push their very own distinctive agenda.
However, making use of a broad, unfocused strategy to participant questions can be harmful. It can trigger a research workforce to lose sight of its authentic objectives and produce research knowledge that’s troublesome to interpret, thus limiting the variety of actionable insights generated.
To overcome this, we use the next strategy when creating research questions:
Phase 1: To begin, the research workforce creates an inventory of potential questions. These questions are then reviewed in the course of the design overview. The objective is to create a concise set of questions which can be clearly written, don’t bias the participant, and praise one another. Often, this entails eradicating a lot of the questions from our preliminary listing and remodeling people who stay.
Phase 2: The second section of WiderFunnel’s research pilot testing consists of participant pilot testing.
This follows a fast and iterative strategy, the place we pilot our outlined research strategy on an preliminary 1 to 2 contributors. Based on how these contributors reply, the research strategy is evaluated, improved, after which examined on 1 to 2 new contributors.
Researchers repeat this course of till all the research design ‘bugs’ have been ironed out, very like QA-ing a brand new experiment. There are completely different standards you can use to check the research expertise, however we concentrate on testing three predominant areas: readability of directions, participant duties and questions, and the research timing.
- Clarity of directions: This entails ensuring that the directions usually are not deceptive or complicated to the contributors
- Testing of the duties and questions: This entails testing the precise research workflow
- Research timing: We consider the timing of every process and the general experiment
Let’s have a look at an instance.
Recently, a consumer approached us to do research on a brand new space of their web site that they have been growing for a brand new service providing. Specifically, the consumer needed to conduct an eye fixed monitoring examine on a brand new touchdown web page and supporting content material web page.
With the consumer, we co-created a design temporary that outlined the important thing studying objectives, goal contributors, the consumer’s challenge funds, and a research timeline. The predominant studying objectives for the examine included growing an understanding of buyer engagement (eye monitoring) on each the touchdown and content material web page and exploring buyer understanding of the brand new service.
Using the outlined studying objectives and research funds, we developed a research strategy for the challenge. Due to the consumer’s funds and request for eye monitoring we determined to make use of Sticky, a distant eye monitoring device to conduct the research.
We selected Sticky as a result of it lets you conduct unmoderated distant eye monitoring experiments, and observe them up with a survey if wanted.
In addition, we have been additionally ready to make use of Sticky’s present participant pool, Sticky Crowd, to outline our goal contributors. In this case, the factors for the goal contributors have been decided primarily based on previous research that had been performed by the consumer.
Leveraging the capabilities of Sticky, we have been capable of outline our research methodology and develop an preliminary workflow for our research contributors. We then created an preliminary listing of potential survey inquiries to complement the attention monitoring check.
At this level, our research and technique workforce performed an inside research design overview. We examined each the research process and move, the related timing, and finalized the survey questions.
In this case, we used open-ended questions as a way to not bias the contributors, and restricted the overall variety of questions to 5. Questions have been reworked from the proposed lists to improve the wording, be sure that questions complimented one another, and have been centered on attaining the training objectives: exploring buyer understanding of the brand new service.
To assist with query readability, we used Grammarly to check the construction of every query.
Following the interior design overview, we started participant pilot testing.
Unfortunately, piloting an eye fixed monitoring check on 1 to 2 customers isn’t an reasonably priced choice when utilizing the Sticky platform. So, we received artistic and used some free instruments to check the research design.
We selected to make use of Keynote presentation (timed transitions) and its Keynote Live function to remotely check the research workflow, and Google Forms to check the survey questions. GoToAssembly was used to watch contributors through video chat in the course of the participant pilot testing. Using these instruments we have been capable of conduct a fast and reasonably priced pilot check.
The preliminary pilot check was performed with two particular person contributors, each of whom match the goal participant standards. The pilot check instantly identified flaws within the research design, which included confusion relating to the check directions and points with the timing for every process.
In this case, the preliminary directions didn’t present our contributors with sufficient context round what they have been searching for, leading to confusion about what they have been really imagined to do. We had additionally assumed that 5 seconds could be sufficient time for every participant to view and comprehend every web page. However, the supporting content material web page was very context wealthy and 5 seconds didn’t present contributors sufficient time to view all of the content material on the web page.
With these insights, we adjusted our research design to deal with these points, after which performed a further pilot with two new particular person contributors. All of the changes appeared to resolve the earlier ‘bugs’.
In this case, pilot testing not solely gave us the boldness to maneuver ahead with the primary examine, it really supplied its personal “A-ha” second.
Through our preliminary pilot checks, we realized that contributors anticipated a set perform for every web page. Participants anticipated the touchdown web page to seize their consideration and appeal to them to the service, whereas, they anticipated the supporting content material web page to offer extra particulars on the service and educate them on the way it labored. Insights from these pilot checks reshaped our strategic strategy to each pages.
The seemingly ‘failed’ results of the pilot check really gave us an enormous A-ha second on how customers perceived these two pages, which not solely modified the solutions we needed to get from the user research check, but in addition drastically shifted our strategic strategy to the A/B variations themselves.– Nick So, Director of Strategy, WiderFunnel
In some situations, pilot testing can really present its personal distinctive insights. It is a pleasant bonus when this occurs, however you will need to keep in mind to at all times validate these insights via extra research and testing.
Still not satisfied concerning the worth of pilot testing? Here’s one remaining thought:
By conducting pilot testing, you not solely improve the insights generated from a single challenge, but in addition the method your workforce makes use of to conduct research. The reflective and iterative nature of pilot testing will really speed up the event of your abilities as a researcher.
Yes, pilot testing would require an funding of each effort and time. But belief me, that small funding will ship important returns on your subsequent research challenge and past.
Do you agree that pilot testing is a necessary a part of all research initiatives?
Have you had an “oh-shoot” research second that would have been prevented by pilot testing? Let us know within the feedback!