Tuesday, October 29, 2013

Premature Experimentation: Revaluing the Role of Essays and Thought Experiments


Some years ago I served as an outside member on a dissertation committee in a linguistics department. The graduate student in question wanted to conduct experiments, an unusual idea for a linguist. When the idea was discussed during the initial committee meeting, a colleague from the linguistics department sighed and said dismissively Ah, experiments. Psychologists always want to do experiments because they don’t know what’s going on. The years had not yet mellowed me (ahem), so I had to bite back a snide comment.

But now I’m starting to wonder if that linguist didn’t have a point after all. Isn’t our field suffering from premature experimentation? Don’t we all have a tendency to design and run experiments before research questions have been really thought through?

I see four major sets of reasons why this might be the case.
  1. Institutional. Empirical articles are the principal currency of our field, so there exists an incentive structure to design and run experiments. Graduate students need experiments to be awarded their degree, postdocs need them to secure a tenure-track job, junior faculty need them to receive tenure, senior faculty need them to procure grants. 
  2. Educational. Students need to learn the trade. Designing and running experiments is a complex set of skills that take years to master.
  3. Cognitive. Experiments are used as mental crutches. It’s hard to mentally manipulate abstract concepts. It’s easier to think about designs, variables, conditions, counterbalancing, and randomization because much of this can be grounded in perception and action (e.g., in the form of an imaginary 2 X 2 table), or offloaded on the environment (e.g., sketched out on the back of an envelope).
  4. Temperamental. We are eager to “get our hands dirty” and curious to see results early on. When I get a new piece of software or some home appliance, I’m usually too impatient to carefully read the manual and/or do the tutorial. I take a quick look and then play it by ear. Running experiments before having things thought through is like starting without having perused the manual.
The last two reasons may not apply to everyone (they all apply to me, I’m afraid) but it is clear that there is a pressure and a drive to produce experiments.

How do we counter this pressure? I believe we should re-evaluate the importance of speculative articles, essays in which authors try out and develop thoughts about a specific topic. The essay is reviewed much like a philosophy or linguistic article would be reviewed (e.g., based on theoretical relevance, soundness of argumentation and clarity of exposition) and is then introduced to the field, whereupon it may receive post-publication feedback, feedback that might give rise to further theory development. And then, at some point, the moment for experimentation has arrived.

Some topics are just not (yet) amenable to empirical research. But this does not mean they aren't interesting or worthwhile to discuss... (I had drafted this post up to this point about two weeks ago. Today someone on Twitter referred to an article that illustrates my point and helps me conclude the post.)
 
In an article that is currently in press in Perspectives on Psychological Science, the social psychologist Joseph Cesario takes a critical look at what he calls the literature on behavioral priming, which has been the focus of several posts in this blog. Cesario observes that the field is lacking theories that specify the precise contingencies under which particular priming effects can be obtained. He then asserts that therefore failures [to replicate priming effects] are uninformative at the current time.

I concur with Cesario’s theoretical criticism of priming research but I disagree with his statement that replications are uninformative. Moreover, if a researcher cannot state the conditions under which an effect is expected to replicate, then the study itself is uninformative. The experiment was conducted prematurely.

It is much better in such cases to just be upfront and present the idea as a thought experiment. The Gedankenexperiment has a venerable history in science. Moreover, it does not carry with it the pretense that there is empirical support for the idea.

There is more to say about the Cesario article but I’ll limit myself here to the conclusion that it nicely illustrates my main point: we should elevate essays to a higher status in our field and at the same time become warier of premature experimentation.

Just imagine what would (not) have happened if Diederik Stapel had not felt the need to produce “evidence” and had just described thought experiments in a series of essays.

16 comments:

  1. Sorry if this is a tangent, but do you suppose the tendency to rush into experimentation has anything to do with a lack of emphasis on observation?

    There are a couple of different models for how to do science. One is based on physics--try to put everything into a few rules, best represented as equations. Another is based on Aristotle or Darwin--observe the world around you and look for patterns.

    It seems to me that the more observations you collect, the more precise a hypothesis you can formulate--and since your observations will already be in terms of human behavior, it's a little easier to figure out how to translate from theory to experimental prediction.

    Observations of real-world behavior might suggest, say, that priming is more likely to have an effect where people don't have a strong opinion, or don't have an emotional stake in the outcome, or are under social pressure, or have a particularly wide attentional window due to positive mood or some other factor. This is just off the top of my head as a non-expert in the field; somebody who thinks about priming regularly and observes the world through that lens would have a better set of predictions.

    So, isn't it telling that at dozens of high-ranked psychology graduate programs, not a single one claims to be looking for students with keen observational skills? Or that naturalistic/observational studies seem to be the province of social psychologists and anthropologists? Surely experiments don't tell us anything if we don't already have some idea what we're looking for?

    ReplyDelete
    Replies
    1. I agree that researchers often lose sight of real world behavior when they start experimenting and doing experiments about experiments, something my own area, cognitive psychology, is more guilty of than social psychology.

      Delete
  2. ~ Excellent post Rolf! Having read across the disciplines for years, I've often had the impression that "data" (and significant findings) are often used as a replacement for critical or deep thinking on the topic at hand. So findings are promoted rather than clear elaborations as to what might be possible mechanisms behind the data. Of course, this is what the system in place rewards, so it's not too surprising.

    ReplyDelete
    Replies
    1. Thanks, Steve! I agree. It's also the case that "significant" effects probably create a hindsight bias that causes people to overlook methodological weaknesses: "It looks like a crappy experiment but because there was an effect, it must have been good after all." This form of hindsight bias (of which we probably all are guilty) will disappear with preregistration.

      Delete
  3. I agree with some of your points here Rolf, and perhaps it's my youthful exuberance, but I think it's fine to run the studies and learn on the fly. As long as you're not overstating your findings, or are willing to pare them back when it's called for, I don't see any necessary disconnect between thinking and experimenting. The problem is when you run one study and pretend like you've bottled a phenomenon. That's understandably and rightly frustrating for many people. Certainly we should stay away from words like uninformative that (I think unintentionally) devalue the just-as-useful efforts to better understand the phenomenon (or lack thereof) in question.

    I definitely agree with you that there should be a whole lot more emphasis on, and credit given for, thought experiments and essays and exploratory work the likes of which Paul Rozin bemoans the lack of.

    ReplyDelete
    Replies
    1. Thanks Dave. Good to hear we are in agreement. The findings of these type of studies are generally overstated. If Cesario is right that the experiments can only be replicated by the authors themselves then it seems impossible to understate the findings: they only apply to this researchers' subject population, lab space, experimenters, equipment, campus, floor of the building, side of the building, time zone, presence or absence of a professional sports team and of a zoo, and so on.

      Delete
    2. As with any finding, you should explain: here's what I did and here's what I found. Where you need to be cautious is in speculating what that means. When you don't know what parameters affected your results you should say so -- in priming and other research as well. Of course we often don't know what we don't know until later, but that's a good reason to be conservative.

      I don't think Joe is saying that nobody else should replicate other than the original authors. Only that when others try to replicate and fail to get the same results, we should not automatically assume that this implies that the original effect was a false positive. Again, caution.

      I think progress in science can be a messy and time consuming process. We are often overly eager to make firm conclusions. In the back and forth we should be careful not to lose sight that our goal is to figure out what's actually going on, not just argue back and forth about whether one side jumped to a conclusion to quickly or the other side dismissed it prematurely.

      Experimental data isn't the only way to resolve these questions, but it certainly can inform us (that's why I liked Joe's use of the term "ambiguous" but did not like "uninformative"). We just have to recognize that we need more than one or two data points (experiments) to say "this is true" or "no it's not."

      Delete
    3. To decide whether something is "true" or not is an appraisal of the truth likeness of a prediction by a theory. There is always a theory, we are not explorers looking for a new continent.

      A failed replication must have consequences for the theory making the original claim. Therefore: "Only that when others try to replicate and fail to get the same results, we should not automatically assume that this implies that the original effect was a false positive. Again, caution."

      No, we should maybe not conclude a false positive, but there must be a consequence for the theory that produced the prediction and I believe the claim by Cesaro, that there basically is no theoretical account that is good enough to produce empirically precise and accurate predictions, is correct. I prefer Rolf's conclusion, let's stop publishing "facts" that are basically just signs of correlations.

      Failed replications mean we should hurry back to to the drawing board and improve on those theories or discard them altogether.

      Measurement contextuality and individual differences may be at work here, but such problems are solvable, theoretically, mathematically, as they have been solved in many other fields of science. So blaming such confounders to me is premature settling for weak theories. Tackling such problems requires investing a lot of time and energy in a discussion about measurement in psychological science, complexity and physics at the ecological scale, thought experiments included!

      Delete
  4. Could some of the aversion to thought experiments in psychology be due to the fact that reviewers will always (at least in my experience!) punish any speculative sections toward the end of journal articles? Obviously I see why they do this- it's never good to go too far beyond one's data. But could this have the unintended consequence of engendering a somewhat thoughtless and atheoretical attitude among researchers?

    ReplyDelete
    Replies
    1. This is true. The current incentive structure works against thought experiment. I'm hoping this will change. I'm already seeing more essayistic work in open access journals like Frontiers.

      Delete
  5. I agree with one of the above writers that psychology (especially during university training) takes a quite narrow selection from the toolbox that could be associated with experimentation and the scientific method in general. to put it coarsely: if it isn't a randomized controlled trial (or some non-clinical look-a-like) it isn't worth looking at for an aspiring or arrived experimental psychologists. I find these type of experiments to be the capstone of the scientific method, but of course not the whole story. It would be great if we could expand our view and pick up other rigorous scientific tools associated with experiments "below" the RCT (or cognitive psychology experiment from which we generalize to the population). For example learning our students how to do statistics when you are doing an intervention with just one person. How do you determine whether your treatment or advice resulted in some better outcome (above random variation) for THIS patient? Or indeed learning them the value of observational studies (by for example showing what is done in neighbouring fields such as biology). By taking this more broader grasp from the experimentation toolbox we would actually be able to arrive better prepared at the moment when we do our big-sample/ hypothesis testing study. And with better prepared I mean with much better intuitions, theory and predictions.

    ReplyDelete
    Replies
    1. I agree Matthijs. We should better identify the different stages of the scientific process and realize they each have their role to play. At the top are confirmatory experiments performed in multiple labs. However, thought experiments, observations, exploratory studies are also important but should not be presented as confirmatory studies. I could cite Wagenmakers et al. again, but I've cited them so often in various posts already that I feel I'd have to charge them for it.

      Delete
  6. I am obsessed with physics (predisposed to become disenchanted with the social sciences). I dropped out of grad school for reasons that many of your articles adress. I am a faithful, albeit, primitive follower, Rolf. I didn’t know guys like you knew how to write (didn’t see much evidence of that at the University of Pennsylvania). “Beauty is truth, truth beauty”—John Keats. Your essays prove that Keats was right.

    ReplyDelete
    Replies
    1. Thanks Angel. I'm glad you like the writing and a faithful follower of the blog. I very much enjoy not having to write in an academic style. I hope you will stay in the social sciences. If anything, we need more people with a natural sciences mindset, not fewer.

      Delete
    2. Angel, I'm curious here. What group did you mean with "you guys"? Psychologists? Dutch people? Professors? Persons of the male gender? Bloggers?

      Delete