Published in the May 2016 issue of Today’s Hospitalist
DOCTORS BELIEVE they are as much scientists as healers, and they presume their evidence base is as up-to-date and complete as possible. But according to one of the nation’s leading health care researchers, that assumption itself does not stand the test of science. Instead, the body of available knowledge available to doctors is in no way thorough.
Why? Because much of it isn’t shared. In a study in the Jan. 19, 2016, issue of The British Medical Journal, Yale University cardiologist and outcomes researcher Harlan M. Krumholz, MD, and colleagues looked at all the interventional clinical studies registered on the federal ClinicalTrials.gov Web site. They wanted to see how many studies were completed between October 2007 and September 2010 and affiliated with an academic medical center.
They found that only 36% of those studies were published within two years of completion. And only 13% of reported results—a much less rigorous process than writing up a study for publication—were published on the ClinicalTrials.gov searchable repository.
“Maybe we need to say that if you can’t report your results, you should stop doing research until you do.”
Further, “no academic center published more than 40% of completed clinical trials” in that time period. What’s more, there was a threefold variation in dissemination rates across academic centers, ranging from 16.2% to 55.3%.
Not only is evidence unavailable for medical decisions, Dr. Krumholz says, but the findings raise ethical concerns. All these projects involved human subjects, he notes, who consented to be studied because they expected the results to be made public and used to advance medicine.
“Timely dissemination of the findings of clinical trials is not only essential to support evidence-based decision making by patients and providers, but is required to fulfill the ethical obligation that investigators and sponsors have,” the study’s authors concluded. Dr. Krumholz spoke to Today’s Hospitalist.
Why did you think this might be a problem that warranted study?
In the mid-2000s, I got involved with the Vioxx litigation on behalf of plaintiffs. Patients and families sued a drug company after rofecoxib was pulled from the market because of concerns over increased heart attack and stroke risk.
I realized the lawsuit said a lot about the ways in which science can be manipulated because many studies weren’t shared and research was not reported. I wondered whether this was something localized to industry or whether it was true in academic centers as well.
So we put together a report card on how the nation’s academic centers are doing, which took a lot of manual work done by medical students. We were dismayed by the findings.
After all, these weren’t studies where the wheels came off and the research didn’t finish. These are all studies—experiments on people—that the researchers documented as successfully completed, but at the end of the day, they are not sharing the findings. I am asking how can we push open science, more collaboration, and greater cooperation and transparency.
Why don’t researchers report findings?
When you see such a pervasive problem, you have to attribute it to culture. This tells us that somehow we have developed a culture that says it is discretionary— not mandatory—to report results, and that it is almost entirely up to the investigator, with no consequences. Once they do the study, people may feel they own the data and they can do whatever they want with their results.
Our study also says that our current funding approach is very inefficient because some funded work is never reported or published. I think funders should get a refund if you don’t report or publish.
Is some under-reporting due to how difficult it is to get published?
This isn’t a matter of people not being able to publish. We counted the study as reported if results were uploaded to ClinicalTrials.gov. As for the unpublished work, I believe it is at least in part because of people not pursuing publication. There are enough journals and venues in which to publish that there is virtually nothing that you could write that is unpublishable.
Also, an institutional review board has already evaluated these studies and decided that each one is ethical and has scientific merit. I don’t believe that there is a trial that has been approved by an IRB that is unpublishable in 2016. And if it is, then we have to question the IRB that approved it as meritorious in the first place.
Why select two years as the time frame in which to report results?
We chose two years because that’s an ample amount of time. The numbers were so bad that we thought we would be generous so that no one could say the time frame was too short.
What solution would you like to see?
I think we need to change the culture. We have to set good examples and say reporting is not just filling out a bureaucratic checklist. It is about sharing what you found with people in a searchable way in the public domain.
Maybe we need to say that if you can’t report your results, you should stop doing research until you do. Maybe IRBs should follow up to see if you have reported, because I think it’s a human-subjects violation that you conduct an experiment and don’t tell anybody the result. No IRB that I know of investigates that.
What are the implications of not fixing this?
You could be the most conscientious doctor doing the most exhaustive literature review, but this indicates that you are still missing a lot. It means that the best systematic review, best textbook chapter and the best guidelines are based on only some of the evidence.
In terms of implications, most of the time you don’t know what you don’t know. But we do have some information from litigated cases. In the Vioxx case, we learned that two years before Vioxx came off the market, the placebo randomized trials showed harm. But many of those studies weren’t published or presented, so that could be determined only when the judge released the studies. In the case of antidepressant drugs, we learned in a study published in the Jan. 17, 2008, New England Journal of Medicine that trials that are published tend to favor drugs more than unpublished trials.
There is a pretty easy fix: Institutions have to recognize that it’s their responsibility to ensure that everybody reports their results within a year, I would say. Two years is too long.
It seems silly to say it, but we clinical researchers need to make a pledge to the public that if you are in our studies, we will report our results. We owe it to the people who participate and to society and each other to develop a research enterprise that follows the scientific principles of reproducibility, transparency and disclosure.
Deborah Gesensway is a freelance writer based in Toronto who covers U.S. health care.