What a pilot study can and cannot show

This guide is general information, not medical advice. It explains how to read pilot study results; it does not say whether anyone should join a study or take a treatment.

A pilot study is an early, usually small trial. Its job is to find out whether a larger study can be run and how, not to answer the main question. This guide draws on two methods papers: the CONSORT extension for randomised pilot and feasibility trials [1] and a tutorial on pilot studies [2].

What a pilot can show

Whether the full study can be done. The CONSORT extension says the primary aim is to assess feasibility, and frames the question as whether the future trial can be done [1].
Whether the steps of the main study are workable in practice. The tutorial says the feasibility of the steps that need to take place as part of the main study is assessed [2].
Safety, and the dose levels and response of the intervention. The tutorial lists assessment of treatment safety, determination of dose levels and response among the reasons for a pilot [2].

What a pilot cannot show

A reliable effect size. The tutorial says it can be dangerous to use pilot studies to estimate treatment effects, as such estimates may be unrealistic or biased [2]. It advises that pilot results should be used cautiously for treatment effects and variance estimates [2].
Whether a treatment works. The CONSORT extension says formal hypothesis testing of effectiveness or efficacy is not recommended in a pilot, and that a pilot's aim is not to assess effectiveness or efficacy and it will usually be underpowered to do this [1].
A basis for power calculations. The tutorial says that if not used cautiously, results of pilot studies can potentially mislead sample size or power calculations [2].
A conclusion from a p-value. The tutorial says emphasis should not be placed on statistical significance when pilot studies are not powered [2].

A pilot is designed to support the development of a future definitive trial [1]. Its results are not designed to stand in for that trial.

Five checks a reader can make

Does the paper call itself a pilot or feasibility study, and say it is designed to support a larger trial? The CONSORT extension describes this purpose [1].
Does it report a formal test of whether the treatment works? Formal hypothesis testing of effectiveness is not recommended in a pilot [1].
Does it present effect sizes as estimates for a future trial? Treat them with caution, because they may be unrealistic or biased [2].
Does it report feasibility outcomes, such as whether the main-study steps worked? The tutorial says the focus of a pilot should be on assessment of feasibility [2].
Does it say how safety was assessed? Safety is one of the reasons a pilot is run [2].

Sources

[1] Eldridge SM and others, "CONSORT 2010 statement: extension to randomised pilot and feasibility trials", BMJ 2016, doi 10.1136/bmj.i5239. https://pmc.ncbi.nlm.nih.gov/articles/PMC5076380/

[2] Thabane L and others, "A tutorial on pilot studies: the what, why and how", BMC Medical Research Methodology 2010, doi 10.1186/1471-2288-10-1. https://pmc.ncbi.nlm.nih.gov/articles/PMC2824145/