November 5th, 2010

Questionable research: Don’t genuflect at Cochrane

Above is an evidence-based practice pyramid (Creative Commons licensed) from the University of Washington Health Sciences Libraries page about evidence-based practice resources.

For those not familiar with the EBP pyramid, it is a tool used to teach about research articles in medicine with the most relevant to clinical care towards the top (Cochrane Library and then other systematic reviews of the literature) and least clinically relevant towards the bottom (clinical reference texts, case studies etc). I learned about this approach to research when I was studying health informatics in school and didn’t really give it a second thought. This Is The Way It Is and makes logical sense.

Lately I have really been rethinking that gold Cochrane capstone after the Pacific Northwest Chapter of the Medical Library Association (PNC/MLA) meeting in Portland last month.

Dr. Erick Turner’s presentation during the meeting of Selective Publication and Drug Efficacy: Don’t Believe Everything You (Don’t) Read (My Notes, His Slides) is a must-read for medical librarians and others concerned about pharmaceutical research and evidence-b(i!)ased medicine. The presentation gives much more background (charts in the slides) than what was published in the book Unhinged with the information about Turner and his work. The review documents for eight of the twelve antidepressants discussed are housed online in the Oregon Health & Science University Digital Resources Library.

At the end of his presentation one of the first questions asked was about the significance of this information for systematic reviews. He replied along the lines of (from my notes)

We use Cochrane as a search method, people genuflect at that name, but all the unpublished trials were missed. Starting point is published literature, FDA reviews are grey literature. Instead, he goes to the grey literature first to identify a cohort of pre-marketing trials then goes to published results to find them which misses post-marketing trials but questions spun factor.

Definitely read my notes or the presentation slides to get the full context of the spin factor since it’s pretty extreme.

Another Medical Librarian Perspective Along the Same Lines

Why Health Sciences Librarians Should Challenge the Flaws of the Peer Review System by Marcus Banks.

We seem to be on a similar wavelength questioning that which is currently seen as sacrosanct:

Yet, despite the recent provenance and pedestrian function of “peer review,” those two words have become holy totems among researchers and librarians.  In our classes we make sure to point students to the checkbox in various databases that limits to peer reviewed journals.  We extol peer reviewed research and excoriate Wikipedia.  And so on and so forth, all in the belief that “peer review” leads people to wonderfully validated, scientifically rigorous, and profoundly accurate research.

Oh, how it doesn’t. But we’re librarians so what can we do about it? Plenty as he suggests:

If we assume that the peer review system is structurally flawed, what should librarians do about it? Here are some options, arranged from least to most attractive:

  • Worst: Continue to glorify the peer review system
  • Better but still bad: Stop the glorification but go no further
  • Better and now good: Vocally discuss the challenges of peer review system on campus
  • Best: Formally challenge funding models/priorities at NIH and elsewhere that sustain peer review flaws.  If funding streams shift behavior change will follow

Many people would say that the best option isn’t for librarians–for policy types and researchers, maybe, but not for us.  We are here to serve, not to challenge.  I disagree, respectfully and yet strongly.  MLA’s Code of Ethics for Health Sciences Librarianship states that one of our roles is to facilitate “informed health care decisions.”  How can we ethically do this if we know that peer review has systemic flaws?


Posted at 07 19 AM | 6 Comments » | print this post

Tags: , , , ,

6 Responses to “Questionable research: Don’t genuflect at Cochrane”

  1. Referencegirl says:

    I completely agree. I cringe every time I hear a colleague (librarians, nurses, and physicians) tout Cochrane Systematic Reviews as being the gold standard and when I have the chance I give the following example…

    Two systematic reviews on the effectiveness of Echinacea in treating the common cold. One is Cochrane and one is New England Journal of Medicine. Cochrane said no, NEJM said yes. The only difference is one article that was not included in one review was in the other. Who is right? One must always read, think, and decide for themselves.

    I also agree regarding Peer Review and have to admit that I do not make limiting searches a standard in teaching or practice. I guess I was out sick the day librarians were told not to challenge. I believe strongly that part of serving is challenging everyone to think think think for themselves.

  2. RachelW says:

    I actually have always kind of cringed at systematic reviews and meta-analyses being at the top of the pyramid, period. I think that’s kind of a wishful thinking status – SRs are only ever going to be as good as the available research, job done finding that research, inclusion criteria, and other methods. Just like any study publication, one needs to look at the methods, see what was in and what was out, and evaluate the quality based on the methods and available evidence – there is no magical SR that is perfect only because it’s an SR, or just because it’s Cochrane. And I say this as someone who contributes to (AHRQ) systematic reviews, and who just came back from a Cochrane/Campbell conference.

    I actually disagree with that whole pyramid, because I can’t see any reason why summaries/guidelines are so high or why Cochrane is necessarily higher than all other SRs. Yes, they pay a lot of attention to their methods. So should everybody, and they should be well-documented enough to allow for evaluation of how applicable the resulting review document is. I’m not willing to say that every review coming out of one group is automatically and perpetually better done than every review coming out of another group, any more than I’d say all Vandy study papers are better than all UW study papers (because that would be ridiculous as a blanket statement).

  3. RachelW says:

    Just to clarify, I think SRs and MAs *can* be really excellent and useful when done well, I just question the wisdom of making blanket assumptions about them. Which you probably said much more clearly in your post than I did in my comment. 🙂

  4. MsPhelps says:

    I completely agree that any SR/MA is only as good as the methodology behind it! I think it’s a growing concern given the increasing popularity of e.g. guidelines – I’ve seen several examples of people/organizations thoroughly underestimating the work that goes into making a good guideline (or SR, or MA…) and I cringe at the thought of that work resulting in an authorative guideline.

    I do command Cochrane for their thoroughness and especially their transparency. That does not make a Cochrane review automatically ‘good’/’the best’, but at least it makes it easier to judge it’s quality.

    Which is what happened last year to the Cochrane review on the effectiveness of neuramidase inhibitors (e.g. Tamiflu) – someone picked up on a serious flaw in the 2006 update re: the inclusion of a review based partly on unpublished trials sponsored and conducted by the pharmaceutical industry (which refused to make the data public). BMJ covered this story extensively (several articles and editorials in BMJ 2009 vol. 339).

    One of the authors of the Cochrane review, Tom Jefferson, spoke at Evidence2010 earlier this month in London (UK) about implications this affair could/should/might have on the way Cochrane reviews are conducted, esp. how to deal with unpublished literature. Still very much a work in progress, but it will be interesting to see what comes of it.

  5. […] fool you into thinking that she is afraid to take on real issues. This month’s post entitled Questionable research: Don’t genuflect at Cochrane is a case in point. Nikki uses her blog as it should be used: for reflection and as a learning […]

  6. Laura says:

    Obviously I’m late to the discussion, but I wanted to mention the potential of selection bias in SR/MAs – the inclusion/exclusion of particular studies can be influenced by the coder’s own biases.

    I agree that the EBP pyramid really oversimplifies things! I’d like to see medical librarians encourage researchers to think beyond MEDLINE (although some might at least search EMBASE and Cochrane, if you’re lucky).

    Drug trials are a real problem, even now that many journals are aware of the bias against negative studies. How can manuscripts be published if they are not submitted (or even written)? Plus, protocols can be changed between registry and publication!

    Speaking of peer review, I’d love to really see how the quality actually varies among different journals. There isn’t a standard – how would it be enforced? Perhaps medical librarians could come up with some productive/actionable suggestions.

Leave a Reply