Is replication necessary?

Hi everyone,

I thought I would post a discussion thread on the topic that came up near the end of class on Friday: does science require replication?

Here are my thoughts on the matter:

Personally, my thinking on this is evolving, partly because my background does not include any kind of formal training on how case studies are conducted. I insisted that science requires replication because I was trying to represent the importance of understanding how inference works when you are using statistics to account for natural variation in your system. I think my instinct was to make sure people understood that you should try to design studies to measure that variability and not to draw conclusions from not enough data (because people do this all the time). I apologize if I sounded really strident about it! I fully acknowledge that in many cases replication isn't possible, and we still need to be able to draw some conclusions based on that information. I also definitely believe that there is information in case studies, which are only one data point!

One of my ecology professors suggested that in 'hard' sciences like physics, chemistry, etc, the objects they study are nearly identical (atoms, molecules, etc) and when you get to ecology, the objects you study (plants, animals, etc) are not all identical to each other in important ways, which makes assessment of the variability really important. When you get to people, you have even more variation! Maybe none of your study objects are the same!

So, when you do a statistical analysis, the conceptual reason you want replication is that if you take a small sample and your sample happens to be pretty unusual, you could make a conclusion that doesn't really reflect what the rest of the 'population' in question is like. So I think this issue comes up when trying to extrapolate from your current study to other systems.

I do believe social science is science. But what I'm curious about is how one draws conclusions from one potentially idiosynchratic example. Maybe some of you with more experience with case studies could offer me some guidance? Or really, if anyone has ideas on this, please share! How do you apply information from a case study to other situations?

2 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. The case study question is an important question that social (and ecological) science has dealt with since their inception in the academy. Can you learn more about a process with a series of replicates across systems, or with one intensive study? The same question is often raised about ethnography in anthropology, whether an "idiosyncratic" case study can tell us anything about the general (this is often phrased in terms of local/global causality). Your question comes down to certain issues that ecologists, and most scientists shy away from, namely philosophical questions of abstraction, epistemology and ontology. Each of these questions in inherent in any practice whether they are scientific or not, and each crucially important for questions about processes. What can replicates across systems tell us that a case study can't? What does reliance on replicates imply about how we construct our categories, whether those categories are useful, and what we consider the nature of being to be? Is the data from replicates any more useful that the data from a case study? Is this an issues of quantitative data verses qualitative? What sorts of practices does science legitimate? If in the end it comes down to how explanatory our theories are, does it matter if they can be quantified?

    One of the main problem that ecologists face is that the systems we study are intermediate in size, not subject to only Newtonian or quantum mechanics (as really really small or really really big stuff is). We study systems that are at the intersection of a multitude of weakly determining forces from evolution and themodynamics to foreign direct investment and colonialism.

    So then the question is can we learn anything about a general process from studying how a number of processes come together in a particular environment. Does a case study on the way nitrogen cycles (using replicates or pseudoreplicates) in a forest in Amazonia say anything about the way nitrogen cycles in a temperate forest. Can we learn anything about general processes from a case study?

    I think so, but again, only if we view a case study as an interaction of processes, not simply an "idiosyncratic" and static picture. For more, please see philosophy and social science

    ReplyDelete