“Piloting is a very important part of the research process.”
That’s what University of Colorado College of Nursing at Anschutz Medical Campus Professor and Chair of the Behavioral, Family, and Population Health Division Paul Cook, PhD, told students during the college’s PhD Summer Intensives Week.
“You may have this grand plan and an intervention you want to implement and one of the first things someone is likely to say to you is ‘You have to have a pilot study’,” he says. “If you haven’t been down this road yet, you likely will be sent there soon.”
What is a Pilot Study?
A pilot study is a small, preliminary study conducted before a full-scale research project. It’s designed to test research methods and procedures and identify any problems that could happen in the main study.
“A pilot study establishes what we want to do,” Cook says. “As soon as you propose a major study, you get reviewers and they ask you all kinds of questions, and if they’re giving you all this money, they’re going to want to see results. The main question is does your intervention work? But they will also have questions about your methods, and your measures, and whether you are actually able to do the study that you propose to do. If there are technical flaws, your study won’t actually answer the research question that it is designed for.”
Purposes of a Pilot Study
Cook explained six potential purposes of a pilot study and went into detail about what PhD students need to consider when conducting research.
Feasibility is showing that methods are practical and that the study can be conducted. This includes figuring out if the right people can participate, and if you can retain them, checking demographics against a general patient population, and making sure the study’s intervention causes no adverse effects.
“This is showing that you’re capable of doing the study you’re proposing. You’ll explain in your grant application that you’ve shown you can recruit people and enroll them in this type of research,” Cook says.
Acceptability focuses more on participants in a study. Researchers need to ask themselves if participants will be able to follow through and not drop out of the study. Participants need to be reliable and willing to take part.
“This is not thinking ‘Can I do it as the researcher?’. Instead, it’s ’What do the participants think of it?’” he says. “So this is more about the people in the intervention, and this is a good place for qualitative research.”
Conducting qualitative research with focus groups can give researchers insights and feedback from participants to help them understand what they liked, what they didn’t, or if there were any barriers.
Cook says researchers also need to test psychometric properties of their measurement tools. Psychometric properties are aspects of a test that indicate any strengths or weaknesses. They reveal if a measure is reliable, which means free of error, and valid, which means relevant and useful.
“Testing psychometric proprieties is something researchers need to do to make sure our measures used in the study are good measures and are feasible,” he says. “Developing and refining measures is something you should always be working on.”
The fourth potential purpose of a pilot study is demonstrating treatment fidelity – measuring the consistency and reliability of an intervention.
“Is this intervention (or treatment) sound? If different people are delivering this treatment, is it all done in the same way? Did I train people appropriately and am I getting the outcomes I want after that training? Those are questions you should ask,” Cook says.
Researchers also need to iterate and refine their interventions. This is exactly what it sounds like – look at an intervention and decide if any changes should be made.
“Can we make a treatment better? Can we build a better mousetrap? This is a dismantling of your treatment and refining it over time,” Cook says. “Let’s say you were working on a 12-week treatment. Was week six really necessary? Maybe we could drop that and go right from week five to week seven.”
The last aspect is one that most people think of when they consider a pilot study, but that is actually quite challenging: showing preliminary efficacy.
This gives researchers the potential to set a minimum-improvement benchmark and demonstrate the clinical significance of the study’s results. But researchers need to be cautious in interpreting their results because a pilot study only answers the question “Can it work?” not “Does it work”?
“I think this is the hardest one to justify because it leaves you open to criticism,” Cook says. “You’re going to get challenges from reviewers, from your committee, and people at the journal where you do your study and submit the article. They may see your pilot as nothing but an underpowered outcome study that should have been much larger.”
Research Advice
Cook advised students to avoid conducting certain types of studies for their PhD dissertation, including full-scale intervention studies. He says they’re complex and difficult. Pilot studies may, however, be a step in the right direction.
“Do something that's kind of piecemeal and developmental,” he says. “Refine measures and do some feasibility work to see if people will sign up. It’s something you could probably bite off without adding years to your life and dollars to your debt while you finish your PhD.”
He also told students that the scientific community will guide students (and new researchers) in the right direction when pilot data are needed.
“Your peers will guide you,” he says. “You’ll put an idea out there and colleagues may say “No, you can’t do that’. It has taken me years to get to a space of gratitude for that peer review feedback. It’s actually a gift they’re giving you because they’re telling you how to make your science better and how to improve things you want to study."
