In Rush to Publish, Some COVID Studies Not Reliable

By Dennis Thompson
HealthDay Reporter

MONDAY, July 27, 2020 (HealthDay News) — The COVID-19 pandemic has prompted a massive scientific response to the crisis, with more than 1,500 coronavirus studies kicking off between March and mid-May of this year, a new study reports.

Unfortunately, much of this research has sown only confusion, producing precious little scientific evidence of sufficient quality to dramatically improve any understanding of COVID-19, researchers argue.

Only about three in 10 COVID-19 studies have been designed with enough rigor to produce valuable evidence about the coronavirus, said lead researcher Dr. Mintu Turakhia, director of the Center for Digital Health at Stanford University in California.

“There’s been an extraordinary activation of clinical research around COVID, and that’s great,” Turakhia said. “The problem is, the majority of these studies are not likely to yield really strong evidence.”

Only 75 out of 664 clinical trials for COVID-19 — about 11% — have all the hallmarks of a scientific study that could be expected to produce solid results, according to the study published online July 27 in JAMA Internal Medicine.

Those hallmarks include random assignment of patients to either the COVID-19 treatment being tested or a placebo; “blinding” that prevents everyone, including the researchers, from knowing who received which treatment during the trial; and the involvement of multiple hospitals with at least more than 100 patients enrolled in the trial.

“If you take this all in totality, what it really tells you is that most of the evidence that’s going to be generated is not going to be very strong and move the needle in terms of the science for COVID,” Turakhia said.

Clinical trials are the gold standard for research, but observational studies also can provide results that inform knowledge and treatment. Those studies involve researchers tracking health trends among people out in the real world, rather than testing treatments in a lab setting.

Unfortunately, the 640 observational studies that Turakhia’s team found regarding COVID-19 also have had weak designs. Only two in five involved more than one hospital, and just 13.6% could be expected to yield strong scientific evidence, the researchers said.

“If you look at the quality of the research that was published in the early days, and this article does a good job of that, you see that it’s really hard to actually come up with something that is actionable from a lot of these early studies,” said Dr. Tracy Wang, director of health services & outcome research for the Duke Clinical Research Institute in Durham, N.C.

As the COVID-19 crisis grew in the late winter and early spring, scientists across the world sprang into action and started hundreds of studies, Wang and Turakhia said.

But they did so without any real coordination or large-scale organization, and as a result their individual efforts have produced weak evidence that can’t even be pooled to create broader analysis, said Wang, who co-wrote an editorial accompanying Turakhia’s study.

To pool data from different studies, you need results that come from a series of randomized clinical trials, Wang said. That way, any interpretation of the pooled data will be based on information from the strongest sources possible.

“So far, an overwhelming majority — 80% to 90% of our studies so far — are not randomized trials,” Wang said, which renders their data “pretty useless” for any larger-scale analysis.

These poorly designed studies tend to frustrate both average citizens as well as health care professionals, because of the conflicting evidence that continues to surface without any way of determining the correct answer, Turakhia said.

“Sometimes these have the potential to influence public opinion, policy, and it confuses the average person and clinician about what to believe,” Turakhia said. “The more noise you have, the harder it is to really convey institutional or societal trust in the really well-done studies.”

Wang recommends that people become “connoisseurs” of medical research, only taking stock in studies that randomly assign treatment to people and that look at health outcomes that actually matter.

For example, only about one-third of the studies in Turakhia’s paper placed an emphasis on tracking mortality, “an endpoint that is really of most importance to a patient and to their caregiver — did it help my family member live longer?” Wang said.

“Mortality or death rate is something we’ve been prioritizing,” Wang said. “This is one of the key points of this article, is that very, very few of these studies used mortality as an end point.”

Turakhia and his colleagues only looked at studies registered with the federal government through May 19. Wang believes that researchers have since dramatically improved their game.

“I can tell you that we’ve now gotten a lot more organized about this,” Wang said. “A lot of these trials are now being designed together in networks in a very collaborative fashion. We’ve got funding agencies standing behind these trials to say, hey, let’s play together so we can get these trials together faster and more efficiently, and do it in a way where if something doesn’t work, we don’t toss the baby out with the bathwater. We can still keep what’s working and proceed to the next idea.”

The COVID-19 pandemic has taught many valuable lessons about how the scientific response to the next major health crisis should unfold, Turakhia said.

“We really have to be better prepared to have a clinical trial network and infrastructure we can activate to do multicenter trials more rapidly in the U.S.,” Turakhia said. “The scientific response is going to have to be part of our pandemic response if we go through this again,” he added.

“You can’t sacrifice well-designed trials just because you’re in a public health crisis. You have to do studies with rigor. That’s really the take-home point,” Turakhia concluded.