Making pilots public and improving developmental science

So why aren’t we reporting pilots?

Why do we pilot?

Most psychology researchers pilot their studies. Piloting involves testing the method and technical aspects of a study before the main data collection begins. We do this to ensure everything works as expected before inviting participants to the main study. Skipping piloting risks methodological or technical issues emerging after significant progress has already been made.

Depending on the study, different types of piloting might be preferred or prioritized. For example, some focus on testing feasibility by checking whether participants understand and follow the procedures. Others use pilots to develop materials, like refining survey questions or designing a new experimental paradigm, and some to evaluate the effectiveness of a method. Finally, many run technical pilots to ensure software, equipment, and the general setup function correctly.

Why is piloting critical for developmental research?

In developmental science, piloting is especially important. Studying babies and children often requires extensive preliminary work to create age-appropriate paradigms. From designing videos that babies will actually watch to building tasks toddlers can understand and tolerate, learning what works across developmental stages takes time and effort. Perfecting study designs before main data collection is also essential because recruiting families is resource-intensive, and finding volunteers willing to complete surveys or come to the lab, amid other childcare responsibilities, is not easy.

This pretesting produces valuable knowledge that can benefit the broader research community. For example, discovering that a particular set of instructions doesn’t work for 4-year-olds, or that 20 half-minute videos are too long for 10-month-olds (but that 15 videos works well, as long as there are sound effects throughout), can be incredibly helpful for others developing paradigms for the same age groups. Still, insights about what worked (and what didn’t) during piloting rarely makes the public record and often ends up in a “file drawer” of unreported pilots, both in developmental science and psychological research more generally.

Working towards more transparency in piloting

Because piloting provides important insights that can easily get lost if not reported, together with several other psychology researchers, we formed the Pilot Reporting Task Force. The Task Force aims to facilitate change regarding pilot reporting practices (or lack thereof) in psychology.

We first sought to understand how researchers pilot their studies and feel about transparently reporting them. In an international online survey with 135 researchers, we found that the majority piloted their studies; however, the methods of piloting varied, and most disclosed that they rarely reported this information in publications. Despite this, many agreed that basic information about pilot studies should be included in final publications (see the full report here).

Given that researchers largely support pilot study reporting but lack guidance on how to do it, we’re now developing resources, including templates and adaptable guidelines. This is not an easy task, and we hope to encourage broader discussion around transparent piloting practices. For those particularly interested in this topic, we invite you to visit our project page or reach out if you’re curious about joining a project or pitching a new one.

What developmental researchers can gain from pilot reporting

It is no coincidence that we as researchers with a background in studying child development initiated a task force on pilot reporting. The resource-intensive nature of our discipline makes methodological efficiency especially important. Sharing insights about what works across different age groups can help others avoid time-consuming dead ends and wasted resources.

Given how novel our paradigms often are and how much trial-and-error goes into making experiments work with babies and children, the process of method development is likely as valuable as the final result (or at the very least, it provides important context for evaluating the result). Beyond evaluation alone, sharing this kind of information could also accelerate methodological innovation or even foster collaboration. Many researchers frequently discover interesting effects of stimuli or contextual factors during piloting that must be controlled for but cannot be fully explored due to resource constraints. By including more context about these effects in our publications, we could build a more collective knowledge base and generate momentum for collaborative initiatives to investigate them further. Of course, such momentum depends on efforts beyond reporting alone, but none of these additional efforts are possible without that initial transparency.

To kickstart this process, developmental researchers could begin by documenting key decisions made during piloting: For example, why were certain stimuli selected? How were instructions modified for different age groups? What procedural changes were made based on pilot feedback? Even basic documentation of these decisions could provide valuable context for other researchers.

In the interim, an easy first step would be to compile this information in a repository and reference it in methods sections with a simple sentence acknowledging that the study was piloted.

Looking ahead

Currently, there are no general recommendations for reporting pilot studies, and it’s likely that pilots are primarily viewed as part of researchers’ methodological due diligence rather than as essential components of the main argument. Particularly, given that these preliminary studies can be smaller in scale and that publications are often evaluated on their specific findings rather than the thoroughness of methodological development. Without an established reporting culture, pilot transparency isn’t likely to be top of mind. However, whenever we discuss this topic with colleagues, many quickly realize the importance of these preliminary studies and the benefits that can be gained from reporting them.

We also foresee some clear barriers to more transparent piloting, which are also likely to contribute to the lack of reporting culture. In addition to lack of reporting guidelines, journal space constraints might sometimes preclude more detailed descriptions. The iterative nature of piloting also makes summarizing it concisely challenging. Lastly, disagreement over what constitutes piloting (or a legitimate use of it) may make researchers hesitant to discuss their practices for fear of scrutiny.

Nevertheless, we believe that fostering a transparent culture around preliminary research has many opportunities. We hope that the standards and templates we develop can serve as a helpful resource. However, the goal is not to create one uniform standard for piloting (or its reporting) across all research areas, but rather to encourage researchers to uncover the process behind method development and testing. The field has long favored clean and polished research narratives, while overlooking the complex and sometimes messy process behind each study. Pilot study reporting could be a window into a more transparent scientific process in psychology in general, and developmental science in particular.

If you’re interested in reading more about this topic, a comprehensive discussion is available in our preprint (Handley-Miner et al., 2025).

About the Author