Statistics as a science

Statistics is sometimes described as a science. I think what people typically mean by this is that the discipline of statistics has a body of theory that is stated in precise terms and evaluated on the basis of logic and agreement of the theory with experience; that there is some effort towards making this theory unified and widely applicable; that the discipline is gradually building a solid body of knowledge.

Statistics does have these features, but I’m not really interested in the broad and vague question of whether statistics is correctly described as a science. Rather, I think it’s interesting to take the statement, “Statistics is a science,” and run with it. I think there is one aspect of science that seems to be missing and represents a major gap and opportunity in the theory of statistics.

One feature (perhaps the central feature) of science is observation of certain natural phenomena. Granted, this is a feature of the natural sciences. When people describe statistics as a science, they are generally putting it alongside mathematics or engineering disciplines — which appear to be described sometimes as “formal sciences” — rather than physics or biology. But that is really my point. Statistics has been viewed as a matter of inventing techniques or concepts to help people work with data. It has not traditionally been concerned with studying and developing a theory of how data are more or less productively put to work in practice or how people reason about data in the real world.

I think statisticians could learn a lot from observing data and scientific thinking in the wild. There are a few different ideas and scenarios that I have in mind which I’ll try to describe here (in no particular order).

1. My experience from almost 10 years as a statistician has been that both statisticians and non-statisticians have ways of reasoning about data that mainly come from intuition or experience rather than formal statistical training. I think I have learned as much about how to approach data from experienced medical scientists as I have learned from statisticians and formal statistical training. Of course, in any discipline there are knowledge and skills that come naturally with experience, and it’s not necessarily productive to try studying and describing every little thing in an expert’s explicit and implicit knowledge. However, I think that in statistics, when we consider all of the really useful knowledge or mental tools that an experienced statistician or scientist uses to reason about data, the balance is too heavily on the side of knowledge that comes exclusively from experience/intuition and has no formal treatment in the theory of statistics. More formal study and development of vocabulary for this type of reasoning could reveal interesting and valuable concepts to incorporate into statistical training and improve scientific collaboration.

2. In particular, there is a phenomenon in statistical analysis that I think most experienced statisticians would attest to, though as far as I know, there isn’t a term for it. As one spends time working with a data set and using various models or other approaches to investigate a question with the data, one often gradually develops a feeling of either trust or distrust towards the data set in general, or towards a specific finding in the data. It arises partly from observing consistency or inconsistencies within the data. Sometimes the comparison of this data set with findings from other data sets, or what is expected for other reasons, has a role. For one project, I remember when a faculty statistician said that she felt confident about a certain finding and wasn’t worried about issues related to multiple comparisons or a lack of pre-specified hypotheses because the finding just felt reliable; I was skeptical towards that attitude, which easily could be based on post-hoc reasoning or all kinds of ulterior motivations, but at the same time I understand and think there’s something real to the notion that reliable findings or characterizations of a data set have a certain, difficult-to-define feeling when you arrive at them after looking at the data from many angles. Meanwhile, some data sets produce a feeling that something isn’t right and that you shouldn’t trust whatever you find there.

3. When the earliest statistical concepts and methods were being developed, quantitative data were relatively rare. Mathematicians and scientists were working in a kind of foreign, unexplored territory where they needed to develop some rudimentary tools for navigating and communicating. Now data are everywhere in science, in engineering, in business, in everyday personal devices. Data are fully present and ubiquitous in the ecosystem, which means there are countless opportunities to observe what happens when human minds try to understand and solve problems with data. And in fact, in all kinds of scientific disciplines (and probably in business-oriented “data science,” though I don’t have experience there), there are scientists developing their own ways of working with data without necessarily consulting statisticians or statistical theory. Even in previous eras, many developments in statistical methodology originated from scientists rather than formally trained statisticians. I have personally witnessed on multiple occasions a non-statistician showing more insight and savvy about data than a professor of statistics. Insight comes from hands-on experience, and the sad fact is that many academic statisticians spend very little time handling data. I would guess that a large majority of the work collecting and using data in the world is being done without the direct involvement of a statistician. It stands to reason that statisticians should be primarily observing and learning from people outside the discipline.

4. One example of an area where statisticians could be studying and developing theory from current and past practices is data quality. Within university statistics departments, data quality has hardly any place in the curriculum or research efforts. When it does receive attention, statisticians tend to have a simplistic idea of data quality — that it is a matter of how many missing values there are or how consistent the formatting is in a spreadsheet. A common view of a statistician, particularly one whose thinking is mainly shaped by graduate school training rather than experience, towards a variable in a data set is that all one needs to know about it is the definition that one would typically find in a data dictionary. In reality, many variables have no simple, concise definition, and understanding what they represent requires understanding a variety of processes for collection, quality control, and transformation behind that number you see in the data table. The “quality” of the data depends on all those processes. In modern science, there are constantly new, complex technologies which produce new types of measurements. How does anyone know what those measurements represent? (One question is, in a broad sense, what does the technology or process measure? Another is, for a specific data set, can we trust that we measured what we intended to? For example, was there contamination in the laboratory sample or assay process?) Despite the importance of that question, in my experience the discipline of statistics provides hardly any formalized, systematic answer, or even much vocabulary for approaching the problem. Clearly, people and organizations manage to address these questions. I think it would be valuable for statisticians to turn this into a formal area of study.

5. Another area that deserves attention is how to create and maintain productive systems for data collection and analysis and for communication of the results. If statistics as a discipline is about learning from data, then it should be interested in the practical question of how an organization or society can learn from data. One example I have in mind is in the design of large clinical research studies. The typical statistician would consider the design as a matter of deciding on the hypotheses to test and selecting appropriate procedures and outcome measures that address those hypotheses. I’ve observed that investigators often see the study as a resource being built for future uses that will probably go beyond any particular questions that the study team has at the design stage. The study procedures and data collection are chosen partly with that long-term view. Some long-run studies prove to be such valuable resources that they continue for decades and almost become an ecosystem unto themselves, supporting entire careers and spawning secondary studies. Statisticians could study the factors that make certain studies or databases more valuable than others. That could include government statistics; what does it take to establish and maintain trustworthy data on the state of the economy, for example? What are the systems that make data more accurate, durable (useful for a long time), and productive (useful for a variety of purposes)?

I believe statistics could encompass so much more, and provide so much more value, than it does now. Unfortunately, to make that happen, statisticians would have to have greater curiosity and engagement with people and problems outside their discipline. Statisticians tend to be timid, introverted personalities who would rather work on clearly defined problems in solitude. People with greater ambitions in science or elsewhere tend to be drawn into careers other than statistics, and statistics holds to a relatively narrow focus.