Wednesday, May 28, 2014

Trying to Understand both Sides of the Replication Discussion

I missed most of the recent discussion on replication because I’m on vacation. However, the weather’s not very inviting this morning in southern Spain, so I thought I’d try to catch up a bit on the fracas, and try to see where both sides are coming from. My current environment induces me to take a few steps back from it all. Let’s see where this goes. Rather than helping the discussion move forward, I might, in fact, inadvertently succeed in offending everyone involved.

Basically, the discussion is between what I’ll call the Replicators and the Replication Critics Reactionaries. I realize that the Replicators care about more than just replication. The Reactionaries are reactionary in the sense that they Critics are opposing the replication movement. The Replicators and the Reactionaries Critics are the endpoints of what probably is close to a continuum.

Who are the Replicators? As best as I can tell, they are a ragtag group of (1) mid-career-to-senior methodologists, (2) early-to-mid-career social psychologists and social-psychologists-turned-methodologists, (3) mid-career-to-senior psychologists from other areas than social psychology.

Who are the Reactionaries Critics? As best as I can tell they are mid-career-to-senior social psychologists. (If there are Reactionaries Critics who don’t fit this bill, I’d like to hear who they are, so I can expand the category.)

What motivates the Replicators? They are primarily motivated by a concern about the state of our field. However, purely looking at the composition of the group, it is possible that career advancement is at least a small part of the motivation as well. The Replicators are generally not the senior people in their field (social psychology) or are in an area (methodology) where they previously did not have the level of exposure (who reads the Journal of Mathematical Psychology?) that they’re enjoying now. And maybe the people from other areas, who seem to have little extra to gain from taking part in the discussion, just enjoy making snarky comments once in a while.

What motivates the Reactionaries Critics? It is clear that senior social psychologists are often the target of high-profile replication efforts. They are also rattled by recent (alleged and proven) fraud cases among their ranks (Stapel, Sanna, Smeesters, Fรถrster). So it is not surprising that they feel they are under attack and react rather defensively. Given the composition of the group, they have something to lose. They have a reputation. Not only that, they have always been able to publish in high-profile outlets and have received a great deal of positive media attention. All of this is threatened by the replication movement. But there is something else as well, the Reactionaries Critics value creativity in research, maybe above anything else.

How do Replicators view original studies? They view them as public property. The data, the procedure, everything should be available to anyone who wants to scrutinize it. This leads them to be suspicious of anyone who doesn’t want to share.

How do Reactionaries Critics view original studies? They seem to (implicitly) view them a bit like works of art. They are the author’s intellectual property and the process that has led to the results requires a certain artistry that one has to be “initiated in” and cannot easily be verbalized.

How do Replicators view replications? There is no single view. Some replication attempts are clearly efforts to show that particular (high-profile) findings are not reproducible. Other attempts are motivated because someone initially liked a finding and wanted to build on it but was unable to do so. Yet other replication efforts are conducted to examine the reproducibility of the research in an entire area. And there are other motivations as well. The bottom line, however, is that Replicators view replicability as an essential part of science.

How do Reactionaries Critics view replications? Given their emphasis on creativity, they are likely to have a low opinion of replications, which are, by definition, uncreative. Furthermore, because the process that has led to the published results cannot be verbalized easily in their view, replications are by definition flawed because there is always some je-ne-sais-quoi missing.

How do Replicators view Reactionaries Critics? Reactionaries Critics are apparently against open science and therefore probably have something to hide.

How do Reactionaries Critics view Replicators? A good researcher is creative. Replications are, by definition, uncreative, ergo replicators are unimaginative third-rate researchers who are only using replication to try to advance their own careers.

Of course these are caricatures (except in some very prominent cases). My take is that I understand why some Reactionaries feel they are under siege and that it is unfair that the spectre of Stapel is frequently raised when their research is involved. I agree that part of being a good researcher is being creative. However, the most important part of the job is to produce knowledge (which has to be based on reproducible findings). I agree that someone who only does replications, while useful, is not the most impressive researcher. On the other hand, I know that Replicators do their own original and creative research in addition to performing replications (and I see no reasons why Reactionaries couldn't do the same). There are no fulltime replicators outside of Star Trek.

It won’t be a surprise to the readers of this blog that I’m on the side of the Replicators. I think the EXPERIMENT-IS-WORK-OF-ART metaphor is untenable and at odds with what science is all about, which is openness and reproducibility, or EXPERIMENT-IS-PUBLIC-PROPERTY (I’m going all Lakoff on you today). Having said this, my sense is that the notion of replication conflicts with the Reactionaries’ Critics' (implicit) ideas about conducting experiments. To bring the Replicators and Reactionaries Critics closer together it might be useful to have a discussion about what are experiments? and what are experiments for? For now, it would help the discussion if members of both groups abandoned the useless REPLICATION-IS-TREBUCHET metaphor and instead adopt the, admittedly less dramatic, REPLICATION-IS-STRUCTURAL-INTEGRITY-CHECK metaphor (which I tried to promote in my very first post).

“Our house is on fire!” exclaimed E.J. Wagenmakers recently on Facebook. In a similar vein, but with less theatrical flair, I’d put it like this: “our foundation is not as sturdy as we might have thought. Everyone, let’s check!”


Now back to the pool.


26 comments:

  1. Nice post, except for where you diss the Journal of Mathematical Psychology! ;-)
    E.J.

    ReplyDelete
    Replies
    1. Thanks. And no dis intended, of course.

      Delete
  2. I think all of us should join forces against the Dichotomizers ;)

    ReplyDelete
    Replies
    1. Can't have a debate without sides of course. That's why it is a discussion and not a concussion. ;)

      Delete
  3. There are a lot of grad students among the Replicators. I think it is important not to overlook them, for several reasons:

    1. They're doing a lot of the actual intellectual work and scientific labor. David Johnson and Felix Cheung, 2 of the 3 authors of the replication of Schnall, are grad students; Johnson was the lead author. It was their idea to run the study, and they invited Donnellan to join them. Donnellan has taken a lot of heat on social media because he is the most senior and because of his blog, but we should not diminish Johnson and Cheung's very important scientific contribution.

    2. A lot of the invective in social media this weekend was directed at the "replicators" without specifying who. Perhaps they were thinking of the tenured folks, but they were targeting everybody. As Yoel Inbar has commented (http://bit.ly/1hz7YH4), it is a very big problem when senior people are saying nasty things about grad students in public.

    3. A lot of other grad students are watching this whole thing and wondering what's going to happen next. As I said, I suspect a lot of their sympathies are with the Replicators. But they're probably wondering what's going to happen if they enter the fray. We need to make sure they feel like they can safely contribute to the future of their field.

    ReplyDelete
    Replies
    1. Very good points, Sanjay. Students, including some of my own, are really crucial in the effort. I'd like to add this to the post but don't know how I can do this best while still showing that it's an addition.

      Delete
  4. I assume you are referring to direct replication, specifically? I am not aware of anyone I would characterize as a "reactionary" who is against replication, per se. To be sure, there are differences in the perceived value of direct and conceptual replication, but I haven't seen anyone actively trying to shut down the direct replication endeavor. In contrast, I have seen a number of calls to curtail conceptual replication.

    Many of the people you refer to as "reactionaries" are merely trying to place direct replication in a broader context of ways to improve science and expand knowledge, and to point out important limitations, where they exist.

    Moreover, the people you refer to as "reactionaries" have a greater appreciation for conceptual replication than do the "replicators." So, who gets called a "replicator" and a "reactionary" depends on what kind of replication you are talking about.

    I would guess that the reason the "reactionaries" are older relates to Dan Gilbert's recent post. At this point, there is real fear among younger academics about appearing "reactionary." The zeitgeist is very strong, and has a moral bent to it. If I were a junior faculty member, post-doc, or grad student, I'd feel much more comfortable expressing pro-direct replication views. In fact, it is junior, not senior academics who have the most to lose by appearing to not sufficiently support the new emphasis on direct replication.

    The suggestion that all of this has to do with media attention is a bit ironic. At this point, the best way to get media attention (if you care about such things) is to report a failed replication of a famous study.

    I think your suggestion that "reactionaries" view data as art and not public property is just plain wrong.

    In my opinion, it mostly boils down to this: There are some people who believe we can't make progress unless and until every finding is verified by robust, direct replication, and that conceptual replication doesn't help much (and is even a major problem). There are others who believe that, for theory building, conceptual replication is much more important than direct replication (which is not unimportant).

    ReplyDelete
    Replies
    1. I changed "reactionary" to "replication critic." I think both direct and conceptual replication are needed, see earlier posts starting here: http://rolfzwaan.blogspot.com/2013/06/how-valid-are-our-replication-attempts.html.

      Delete
    2. Would be better to change "replicator" to "direct replicator" and "replication critic" to "direct replication skeptic"

      Delete
  5. I think you have succeeded in offending everyone -- the replicators, the reactionaries, and the vast majority of people who fall into neither of those categories, including myself. This response deserves a blog-length treatment, but with a new arrival in the family about two weeks away this is going to have to suffice.

    My guess is that the vast majority of people in social psychology (and beyond) could probably be characterized as believing the following things, to vary degrees.

    1. There are shortcomings in the current research practices in our field. We are not necessarily unique in this way, but if our goal is to try to get a better understanding of what's true, we have to improve. This is true in several areas, including, but not restricted to reducing p-hacking and other questionable research practices, publishing null results, and increasing the amount of direct replication that we do.

    2. There is a collective action problem whereby it's difficult for many individuals, particularly at early stages in their careers, to unilaterally decide that they will forego methods that will improve their productivity, particularly when it can be tantalizingly easy to rationalize one's behavior in various ways.

    However, we are grateful to those who are putting a lot of time, effort, and sometimes career security on the line to overcome this collective action problem. This is far from an exhaustive list, but Brian Nosek's work with the OSF, Bobbie Spellman, and Simmons, Nelson, and Simonsohn's p-hacking and related work come immediately to mind.

    3. We want the "revolution" to be peaceful and fair. Our assumption is, perhaps somewhat naively, that everyone has been acting in good faith. Improving research practices should not come at any individual's expense (although there may be inevitable collateral damage, it should be minimized whenever possible). That means that we shouldn't single out a single person who is one of many adhering to norms that should be changed.

    For me, a perfect example of this is that it's not fair to call someone out for dropping a failed study from an article (particularly when asked to do so by an editor!) when that's the accepted norm, even if that's a problematic practice. Changes have to happen at a collective level -- in this case, starting with reviewers and editors changing their behavior.

    (Continued…)

    ReplyDelete
  6. (Continued from previous comment)

    In this regard, most of us are neither Replicators nor Reactionaries, we are for lack of a better term, Moderates. When you scratch past the surface, I would wager that a good number of Replicators and Reactionaries are Moderates as well, they've just been caught up in a debate that has an us vs. them mentality that is, in my opinion, counterproductive. (And this is a teasing jab at you, Rolf, but dividing people into two camps, and labeling one "reactionaries" probably isn't helping).

    And to the extent that there are people at the extremes, I'll add only this. For so-called reactionaries, I believe most of them are coming from a place where they're trying to protect the innocent. Certainly there's defensiveness going on, but I think that's a relatively natural reaction, and the way to reduce that defensiveness is not with hostility.

    For the replicators, I'll repeat what I wrote in support of Kahneman's "etiquette" proposal. Science is ll about ideals, but scientists are humans. I know there are those who disagree, but I believe that pragmatic solutions mean taking that fact into account. If replication were perfect, then you could ignore people's human sentiments and just do the science. But replication is not perfect. There are a lot of ways to get replications wrong, and we're a long way from perfecting the process. We have to make allowances for that and not pretend like we're just doing science -- we are affecting people's lives. The Moderates are much more likely to go along with the replication agenda the less dogmatic it is.

    And in the end, I strongly believe that getting the Moderates on board is what will make or break the current replication movement. It's all about the norms -- what people believe is the right way to behave, and how they think everyone else is behaving -- is what will guide behavior. Top down rules may make some difference, but really it's a question of people choosing to behave differently. I think the Replicators, even at the extremes, have good intentions and have already done a lot of good. But for their project to ultimately succeed, it's going to take more than repeating the replication mantra. They have to behave in a way -- and I think there are many great examples of people who already do -- that convinces the vast block of Moderates to come along.

    p.s. So much for not writing a blog length treatment -- I have to break this into two posts because it's too long!

    ReplyDelete
    Replies
    1. Nice comments, Dave. Just a brief reply. I'm not dividing people into two camps. I'm just trying to make sense of a debate that has been unfolding in the previous days, in which I was not involved. Debates usually have (at least) two sides and they were clearly visible in the debate. The point of the post is that we should all be on the same side and check the structural integrity of the field.

      Delete
  7. I assume you are referring to direct replication, specifically? I am not aware of anyone I would characterize as a "reactionary" who is against replication, per se. To be sure, there are differences in the perceived value of direct and conceptual replication, but I haven't seen anyone actively trying to shut down the direct replication endeavor. In contrast, I have seen a number of calls to curtail conceptual replication.

    Many of the people you refer to as "reactionaries" are merely trying to place direct replication in a broader context of ways to improve science and expand knowledge, and to point out important limitations, where they exist.

    Moreover, the people you refer to as "reactionaries" have a greater appreciation for conceptual replication than do the "replicators." So, who gets called a "replicator" and a "reactionary" depends on what kind of replication you are talking about.

    I would guess that the reason the "reactionaries" are older relates to Dan Gilbert's recent post. At this point, there is real fear among younger academics about appearing "reactionary." The zeitgeist is very strong, and has a moral bent to it. If I were a junior faculty member, post-doc, or grad student, I'd feel much more comfortable expressing pro-direct replication views. In fact, it is junior, not senior academics who have the most to lose by appearing to not sufficiently support the new emphasis on direct replication.

    The suggestion that all of this has to do with media attention is a bit ironic. At this point, the best way to get media attention (if you care about such things) is to report a failed replication of a famous study.

    I think your suggestion that "reactionaries" view data as art and not public property is just plain wrong.

    In my opinion, it mostly boils down to this: There are some people who believe we can't make progress unless and until every finding is verified by robust, direct replication, and that conceptual replication doesn't help much (and is even a major problem). There are others who believe that, for theory building, conceptual replication is much more important than direct replication (which is not unimportant).

    ReplyDelete
  8. While a disagree with a few minor points from Dave Nussbaum's comment, I wholeheartedly agree with him that "there are a lot of ways to get replications wrong... [and] we have to make allowances for that...[because] we are affecting people's lives"!

    Indeed, this is why we should view "unsuccessful" replications as beginnings rather than ends and avoid black and white thinking that findings exist or don't exist. Scientific beliefs involve carefully calibrating our beliefs according to all of the available evidence (for more details, see my latest blog post about this: http://proveyourselfwrong.wordpress.com/2014/05/27/unsuccessful-replications-are-beginnings-not-ends-part-ii/).

    From this perspective, however, it is clear that we also need to change the modal way psychologists interpret *ANY* empirical findings, whether these are original results or replication results. This can vividly be demonstrated by simply replacing the word "replications" with the word "studies" in Nussbaum's original quote:

    "there are a lot of ways to get [*STUDIES*] wrong... [and] we have to make allowances for that...[because] we are affecting people's lives"!

    Indeed! Any research finding -- whether original or replication findings -- should be seen as a beginning rather than an end. We should demand a lot more evidence supporting a certain claim (via repeated risky tests), before increasing our confidence in a claim before running to the press!

    ReplyDelete
    Replies
    1. Along these lines, it seems to me that one conclusion we can draw from the reactions to the replication findings is that many of these replication studies were not needed in the first place. That is, scrutinising the original studies reveals that they are unconvincing, so a replication check is not necessary; the impetus is still on the proponents to show that there is anything worth considering. If the replicators cared about the specific topic then it is fine for them to run further investigations, but I think many of these replication studies were a waste of effort for the replicators and the field.

      I am more bothered that many scientists considered the original studies to be convincing than that so many replications were unsuccessful.

      Delete
    2. Observing some ongoing replication projects from the side it has also struck me as somewhat ironic that compared replicators are putting in a lot more work (and sometimes thought) into the replication than must have gone in to the original study even though they profess to have no confidence whatsoever in the original finding. This is maybe why I prefer "replicating up" (see next post).

      Delete
  9. This comment has been removed by a blog administrator.

    ReplyDelete
  10. This comment has been removed by the author.

    ReplyDelete
  11. Nice post Rolf. I like how you are being very evenhanded. I will add a comment from my personal experience. When first reading about social priming I was amazed and interested. I tried to build on these finding, but wasted a lot of time, since I couldn't even replicate the original results!
    So, yes, I understand that the original authors can become defensive, or even hostile, when people fail to replicate them, but the goal of science is to further knowledge, not egos. Or, in other words, keeping failed replications under the lid will lead other researchers to waste a lot of time, and give us a wrong understanding of human nature.

    ReplyDelete
    Replies
    1. Thanks Yair, I'm trying to be evenhanded. Not that everyone recognizes this of course.;) I think social priming (however one wants to define it) is probably an area that has more problems than some other areas. However, it has also attracted more replication attention than other areas, in part because its experiments are so straightforward to implement (e.g., as compared to cognitive neuroscience) that have hardly received any scrutiny at all might prove at least as problematic.

      Delete
  12. Hi Rolf-

    I am a replicator and I liked the post a lot. I was not offended probably because it is hard to be offended after you have been called a shameless bully and an a****le on twitter. In fairness, I brought most of this on myself because of a few poorly chosen sentences in a blog post. So be it. Mostly, I would like to clarify my own motivations (to the extent that I can even be consciously aware of them) to illustrate your point that the “direct replicators” are a fairly rag-tag group. I also want to piggy-back on a point raised by Sanjay to protect a few innocents in this whole mess. I can take the hits, I am not sure they can.

    I got involved in replicating studies by accident. I had worked with a colleague on a paper about reporting practices in social/personality psychology. This forced me to read broadly across different subfields and thus exposed me to research outside of my own specialized niche. It was an eye opening experience in terms of statistical power and the level of detail in many studies. This process gave me a sinking feeling that all was not well in the field. So I thought I would “donate” some of my time to making things better by replicating studies that I found interesting. The root motivations are a mixture of curiosity and what I would term “healthy” skepticism.
    This has never been about career advancement or media attention in my case. I think this is true for most of the replicators. There are more direct and less controversial ways to accomplish those goals rather than replicating studies.

    Anyways, I think we have started to develop a culture in my area of my department that emphasizes transparent reporting, statistical power, data sharing, and a general openness to conducting replication studies (whether it is within lab or more independent replications). This probably motivated David and Felix to want to conduct the direct replications that were featured in this recent special issue of Social Psychology. (Don’t worry -- we also want our graduate students to be able to make their own positive contributions to the field so we also emphasize the importance of developing independent lines of research.) But to the extent that I can get inside their heads – I fully believe they were motivated by curiosity and an enthusiasm for science. Hopefully that can be inspirational to all views in the current debate.
    -brent donnellan

    ReplyDelete
    Replies
    1. Thanks for your kind and trenchant comments Brent. It looks to me like the "Replicators" are indeed not a group with a single agenda but folks with different interests and different motivations to conduct replication studies. As I said in the post, I don't think career advancement is a major motivational factor to do replication studies at all. It is, however, at least in theory a factor and in a heated debate, Critics seem to suggest it is a major factor. Likewise, in a heated debate, some Replicators might come to think, say, or believe some things that are unfair about the Critics. This is what I was trying to point out. The way I got interested in replication is very similar to yours. In my next post, I'll talk a little more about this.

      Delete
  13. I think that a neutral, professional language in a replication study, in conversations via email, twitter and blogs should do it - both for critics and replicators.

    More specifically, it should be possible to reproduce work without their help. For that purpose, all journals should require authors to upload all data (not just ask authors to send them on request), full descriptions of the data and procedures etc. in an online supplement. Ideally, anyone should be able to reproduce the work independently without ever contacting the author.

    In addition, it is not clear to me why an original author should have the right to analyze the replication and review the manuscript before publication. Once the original paper (and data) is published and enters the scientific dialogue, it is not the ‘property’ of the original author, but belongs to all. In the past, authors have answered to replications of their work, which I call a replication chain – but after publication.

    ReplyDelete