How to read error bars and confidence intervals

A chart or results table often shows a range around an estimate. That range may be a confidence interval, or a bar drawn for another measure. This guide explains what such a range can and cannot tell you. The statistical points below come from four pages of the NIST/SEMATECH e-Handbook of Statistical Methods and from a 2007 review of error bars by Cumming, Fidler and Vaux in the Journal of Cell Biology, each cited after the point it supports.

First, what kind of bar is it

Cumming and colleagues note that error bars may show confidence intervals, standard errors, standard deviations or other quantities, and that different types give quite different information (https://doi.org/10.1083/jcb.200611141). A range or standard deviation bar describes how spread out the data are. A standard error or confidence interval bar describes how precisely the mean has been estimated (same source). Their first rule is that a figure legend should always say which kind of bar is shown. Without that, the bars cannot be read.

What a confidence interval says

A confidence interval gives a lower and an upper limit for a quantity such as a mean, rather than a single estimate (https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm). For the mean, the limits are the sample mean plus or minus a t-multiplier times the sample standard deviation divided by the square root of the sample size (same source).

NIST notes that 90%, 95% and 99% intervals are often used, with 95% the most commonly used (same source). The confidence level does not give the probability that one particular interval contains the true mean. NIST says that "a 95 % confidence interval does not mean that there is a 95 % probability that the interval contains the true mean." The level describes the method: if many samples were collected and an interval computed from each, "in the long run about 95 % of these intervals would contain the true mean" (same source).

What the width depends on

NIST says that "As N increases, the interval gets narrower" (https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm). It also says that "The larger the sample standard deviation, the larger the confidence interval" (same source).

What overlapping bars do and do not show

Overlap is not a test of whether two results differ, and it can mislead in both directions.

Overlapping 95% confidence intervals do not by themselves mean there is no statistically significant difference. Cumming and colleagues give a rule of thumb for two independent groups with a sample size of 10 or more each: when the 95% intervals overlap by about half of one arm, P is about 0.05, and when their tips just touch, P is about 0.01 (https://doi.org/10.1083/jcb.200611141).
Standard error bars that do not overlap do not by themselves show a statistically significant difference. For the same case of 10 or more per group, the authors say a gap of about one standard error between the bars indicates P of about 0.05 (same source).
These rules apply only when the two means are independent. When the same group is measured more than once, for example the same animals at different times, the authors say the bars do not give the information needed to judge the difference (same source).

The rules above are approximations. For a definite answer, look for a direct comparison: the NIST two-sample t-test, for example, builds its test statistic from the difference between the two sample means and a term based on the two sample variances and sizes (https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm).

Four checks a reader can make

Check what the bar is. Look in the figure legend or methods for whether it shows a standard deviation, a standard error or a confidence interval (https://doi.org/10.1083/jcb.200611141). If it is a confidence interval, check the level: NIST says 90%, 95% and 99% intervals are often used (https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm). Compare bars only when they show the same measure at the same level.
Check the sample size. The width of a confidence interval shrinks as N increases (https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm). Cumming and colleagues also ask readers to check that n counts independent experiments or individuals rather than repeat measurements of one sample (https://doi.org/10.1083/jcb.200611141).
Separate statistical and practical significance. NIST says that the ability of a test "to detect differences that lead to rejection of the null hypothesis depends on the sample size". With a particularly large sample, a test may reject the hypothesis that two process means are equal even when the difference is so small that it has "no real engineering significance". With a small sample, a difference that is large in engineering terms may not lead to rejection of the null hypothesis (https://www.itl.nist.gov/div898/handbook/eda/section3/eda35.htm).
Check for a direct comparison. If the question is whether two results differ, look for a test or interval for the difference itself rather than judging from overlap (https://www.itl.nist.gov/div898/handbook/eda/section3/eda353.htm).

Small samples and proportions

For a proportion, such as the share of units that fail, NIST says that "If the number of failures is very small or if the sample size N is very small", symmetric confidence limits approximated using the normal distribution "may not be accurate enough for some applications". In those cases the page gives an exact method based on the binomial distribution, which produces an interval that is not symmetric about the observed proportion (https://www.itl.nist.gov/div898/handbook/prc/section2/prc241.htm). A reader who sees a symmetric interval around a proportion from a very small study should check which method was used.

Limits of this guide

This guide uses four NIST handbook pages and one review article as the source for the statistical points. The overlap rules are approximations for simple cases. It does not give a number for any particular trial or technology result. Read the paper's methods section for the type of bar, the confidence level, the method and the sample size used.