Internal and External Validity

Not all scientific studies are created equal. Some studies are well-designed, with results that stand up to the intense scrutiny, analysis, and replication demanded by the scientific method. On the other hand, some studies are designed poorly, resulting in conclusions that can be called into question or that are not supported by the data. In this post, we explore two of the major ways that scientific studies are evaluated, giving you some tools to help in your own evaluation of the caliber of research studies.

The two aspects of research quality we will discuss today are internal validity and external validity. First, let’s consider the word validity. A study is considered valid - from the Latin word for 'strong' -  if it is strongly supported by facts and logic. In terms of scientific research, to have valid conclusions, a study must have a valid design. This brings us to internal validity.

Cause and effect

Internal validity measures how well a study’s design supports a cause-and-effect relationship between variables. To be as certain as possible that X variable causes Y effect, the study must be designed to eliminate as many confounders as possible. Confounders are any variable that could impact the Y effect, other than X.

Let’s consider a hospital that wants to test how effective a handwashing training program is on handwashing compliance. Threats to internal validity in this study would be how the participants were selected (did they volunteer or was it random?), how the data was tracked (was the data tracked consistently?), or how the training was delivered (did everyone get the same trainer and materials?). The study needs to be designed carefully to avoid these and other biases.

Generalization

The scientific method demands that results found in one study be able to be replicated by other researchers. If a study has strong external validity, it means that the same results should be attained if the same study were to be conducted in another setting. It also means that no unique characteristic of the study setting, participants, timing, or other element played a significant role in the results.

Using the same scenario as above, a study measuring the impact of a handwashing training program on handwashing compliance would be externally valid if that same training program could be counted on to achieve the same results in a different hospital. Aspects of the study or training program itself would have to be designed in such a way that the participants and conditions are as representative of other settings as possible. The external validity of a particular study changes as more research is conducted in the same area; other studies may demonstrate that it was not very generalizable (that is, its external validity was weak) or that it proved to be very consistent in different settings (making its external validity strong).

The way a study is designed has an enormous impact on its resulting internal and external validity. Stay tuned next week as we discuss the lesser-known member of this research quality triumvirate, fidelity.

Editor's Note: This post was originally published in July 2016 and has been updated for freshness, accuracy and comprehensiveness.