Chapter 2 A Brief History of Experimental Design

Researchers in the pre-statistics days lacked the statistical framework that today’s researchers take for granted. Our ancestor scientists were remarkably adept at the scientific method, in making observations, and in collecting data with great care. However, they struggled with designing experiments, in summarizing the data, and in drawing unbiased inference from it.

The statistical approach to experimental design we use today was first enumerated about a century ago, largely by Sir RA Fisher. His story is interesting in part because it is just so classically accidental.

Figure 2.1: RA Fisher in 1913, from the Adelaide Digital Archive

At the outset of his career Fisher did not foresee authoring the foundational principles of experimental design and statistics practiced by most of us today. He took that trajectory by accident.

For about five years after graduating from Cambridge, Fisher worked as a census bureaucrat and part time math teacher.

He was smitten by Darwin’s theory of evolution, which was the hot discovery of the day, of course. Fisher’s side hustle was to work on mathematical problems related to evolutionary genetics. Today, we would probably recognize him as a hobbyist quantitative geneticist or perhaps even as one of the first bioinformaticians. That’s certainly where his career ambitions seem laid. He never lost an interest in evolution and would go on to become, unfortunately, a prominent eugenicist. The take-away from that, alone, is that statistics is not a fool-proof antibias framework.

Still, one big contribution he made during this early stage was no small feat. He defined variance and its relationship to the mean of a population. He proposed that variance is useful as a descriptive statistic for the variability within a population. Further developed, it would soon become the foundation of the powerful multigroup experimental designs that are called ANOVA, the analysis of variance, which are widely used today in the biomedical sciences.

In 1919 Fisher was hired as a temporary statistician by Sir John Russell, the new director of the Rothamsted Experimental Research center in England.

After decades of underfunding Rothamsted had become a bit rundown. Russell, an agricultural chemist who today we would probably categorize as a biochemist, was hired to beef up postwar (WWI) agricultural research in the UK. Upon arrival he realized the station had a large repository of data. Fully expecting to create even more under his leadership. Russell believed bringing a mathematician on board could help him make sense of this data they already had on hand.

Thus, Russell hired Fisher to take a temporary position. Today, we would recognize Fisher early in his Rothamsted role as a freelance data scientist charged with conjuring meaning from reams of the station’s data, some of which represented serial agricultural experiments that had been running for decades.

As he dug in Fisher saw a lot of flaws in the Rothamsted dataset. He had difficulty making sense of much of it. Mostly because the experiments were, in his view, so poorly designed the results were uninterpretable. If that sounds at all familiar then I’ve achieved my objective for mentioning it. There’s nothing worse than spending months of painstaking effort collecting data that can’t be analyzed.

Here’s when the paradigm shifted. Fisher began to think about the process by which experimental data should be collected. Almost immediately after digging into his Rothamsted work he invented concepts like confounding, randomization, replication, blocking, the latin square and other factorial designs. As I mentioned above, his invention of the analysis of variance extended his prior work on variance. The procedure of maximum likelihood estimation soon followed, as well.

It was a truly remarkable period. In 1925 Fisher published a small book, Statistical Methods for Research Workers. In 1934 he published its extension, Design of Experiments. In these works lay the foundations of how researchers today approach their experiments. His statistical procedures, developed with agricultural science in mind, would soon cross oceans…and then disciplines.

It is telling that the first academic statistics department was at Iowa State University, in the heart of the corn belt, in the 1920’s. George Snedecor started it. George Snedecor is also the author of the F-distribution and the F-test, which is used to make decisions on ANOVA experimental designs. By the 1950’s statistical methods were spreading rapidly through the medical literature.

Today, experiments that we would recognize as statistically rigorous are those in which Fisher’s early principles operate as procedures. We know today that randomization and pre-planned levels of replication are essential for doing unbiased research. The block ANOVA designs he mapped out then are among the most common experimental designs that we see in the biological and biomedical literature today.

There’s much more to this history, including many additional players and plenty of controversy that remains unsettled to this day. I emphasize Fisher mostly because his experimental design and analysis procedures remain the standard for prospective experiments today.