What a difference a day makes: How social media is transforming scientific debate
Postpublication peer review of "Sex differences in brain connectivity" PNAS paper by Ingalhalikar
- I woke up Tuesday morning to find a tweet from David Colquhoun asking my opinion of a new paper on sex differences in brain connectivity, asking if it was neurobollocks. I found the paper here:
and took a quick look.
The immediate reaction was that this was typical PNAS paper: extremely eye-catching result with only scanty description of methods. I replied:
- It seemed odd to me that the authors wanted to generalise to males and females in general, without telling us anything about where these people came from, how they were selected, and what level of cognitive functioning they had.
- I added that if an effect was only evident when N = 900, then it would be a very small effect; in other words, while there may be a mean difference between males and females, the variation within each of those groups would be much larger than any difference between them. This was not conveyed at all in the media coverage.
- I then received an email from Radio5Live asking if I'd talk about the study. Agreed to do so and reorganised my day to fit in with their schedule. Ed Yong warned me that they might keep me hanging around a long time, and so it proved. Only then they decided not to feature the story after all. According to Ed this is typical - they are rather like airlines and overbook interviewees so they can always have someone to hand.
- Fortunately, my failure to be interviewed didn't matter, because several other people were on to the study, and noting a whole range of problems with it. I particularly liked this blogpost by Matthew Thomas, who picked up on the failure to report effect sizes.
- And then someone actually estimated the effect sizes in the study (using sample size and reported stats I assume);
- Plus a nice piece by Tom Stafford:
- Another issue that was mentioned on Twitter was the way that the authors talked about 'hardwiring' of the brain, when we know connections are modifiable:
- Then a new criticism: could movement artifact be a factor? I have to say I thought this was implausible - why should males and females differ systematically in how they move in the scanner? But as the day went on, people started to produce evidence to suggest this might be important.
- By Weds morning, Australia had woken up to the story, with neurosexism expert Cordelia Fine on the case:
- This story raises a whole host of interesting issues:
1. it's an interesting case where the story plays into a lot of stereotypes about gender differences. Some of the comments on the story were by people who seemed unwilling to accept any notion that there might be gender differences in the brain. I don't agree. I think it's quite likely that there are gender differences in the brain, just as there are in other bits of the body - see http://deevybee.blogspot.co.uk/2011/05/x-and-y-of-sex-differences.html …. My criticisms of the study are not because I dislike its message, but because it is badly reported and over-hyped. Throughout the day, Kevin Mitchell provided an important corrective to those who wanted to dismiss the study because it did not fit their preconceptions:
- 2. The behaviour of the scientists doing the study is hard to understand. When they talk of hardwired differences in brains of males and females, and link their results to putative behavioural differences that fit stereotypes but which they haven't measured, they lose credibility among their scientific peers, while at the same time attracting a great deal of publicity. I can't imagine why any serious scientist would do this. Do they really believe what they are saying - in which case they are bad scientists? Or are they so swept up in a brief moment of media fame that they don't care about their reputations?
3. I wanted to Storify this because I think it's a great example of how Twitter and blogging can allow for rapid discussion of topical papers. It's sometimes argued that the unconstrained nature of the internet is not conducive to rational debate, but in this case, I think it has worked well. I learned things that I had not been aware of, and was impressed at the speed with which key issues in this paper were identified.
The use of PubPeer for postpublication discussion seems an important new development, which allowed the movement artefact issue to be aired right away.
PS. 14:15 4th December. I have now added a comment on PubPeer, which you can find here:Also another blogpost on this story:https://sites.google.com/site/speechskscott/SpeakingOut/askingquestionsaboutmenandwomenbylookingatteenagers …
5th December: and another ace piece on this story by @Psych_Writer
and also this more informal take on it by @Echidne
8/12/13: Additional link from an early critique that seems spot-on to me:
PS. You used to be able to create a Story on Storify much more easily, by finding relevant tweets on Twitter and sending to your story. As far as I can make out, that functionality has been lost, and Google suggests this is a deliberate attempt by Twitter to squeeze out Storify. That's a shame, and has meant that I have omitted some material that I'd have liked to include here.
Did you find this story interesting? like or comment as 6 already did!