Major Hydroxychloroquine study called into question

Major Hydroxychloroquine study called into question
by dulac89 (2020-05-29 10:58:32)
Edited on 2020-05-29 17:04:02

In an open letter to the study authors and the editor of the Lancet a number of concerns are raised.

1. The numbers in the article don't compare to official numbers
2. The research group is, so far, not sharing any of their data or hospitals. The Lancet is a signatory to an international consortium of research groups and publishers that any and all COVID data that is published has to be open source.
3. Hospitals that had to have been included are saying they were never contacted or gave permission for their information to be shared.

I think it's important to point out that when trying to determine the veracity of scientific data, the underlying motivations of those that are posting/publishing must be considered. The signatories on this letter include researchers who are on record as being highly skeptical of HCQ for COVID. When any person or group doubles down on something "scientific" that a large group of people are questioning, that is a big red flag (in medical research at least) that there are motivations other than the truth. Which is why all these conspiracy theories can be debunked so quickly. Most true scientists welcome others attempting to replicate, criticize, and tear apart their work so that the truth can be had. So far, the study authors' only public responses have been defensive. This, combined with some of the people who signed the letter, make me worry that there is more going on here and, if the group does not completely release all of their data publicly, I wouldn't be surprised to see the Lancet issue a retraction.

Here is the text of the letter

Open letter to MR Mehra, SS Desai, F Ruschitzka, and AN Patel, authors of“Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis”. Lancet. 2020 May 22:S0140-6736(20)31180-6. doi: 10.1016/S0140-6736(20)31180-6. PMID: 32450107and to Richard Horton (editor of The Lancet).

Concerns regarding the statistical analysis and data integrity

The retrospective, observational study of 96,032 hospitalized COVID-19 patients from six continents reported substantially increased mortality (~30% excess deaths) and occurrence of cardiac arrhythmias associated with the use of the 4-aminoquinoline drugs hydroxychloroquine and chloroquine. These results have had a considerable impact on public health practice and research. The WHO has paused recruitment to the hydroxychloroquine arm in their SOLIDARITY trial.The UK regulatory body, MHRA,requested the temporary pausing of recruitment into all hydroxychloroquine trials in the UK (treatment and prevention), and France has changed its national recommendation for the use of hydroxychloroquine in COVID-19 treatment and also halted trials. The subsequent media headlines have caused considerable concern to participants and patients enrolled in randomized controlled trials (RCTs) seeking to characterize the potential benefits and risks of these drugs in the treatment and prevention of COVID-19 infections.

There is uniform agreement that well conducted RCTs are needed to inform policies and practices. This impact has led many researchers around the world to scrutinize in detail the publication in question. This scrutiny has raised both methodological and data integrity concerns. The main concerns are listed as follows:

1.There was inadequate adjustment for known and measured confounders (disease severity, temporal effects, site effects, dose used).

2.The authors have not adhered to standard practices in the machine learning and statistics community. They have not released their code or data. There is no data/code sharing and availability statement in the paper. The Lancet was among the many signatories on the Wellcome statement on data sharing for COVID-19 studies.

3.There was no ethics review.

4.There was no mention of the countries or hospitals that contributed to the data source no acknowledgments to their contributions. A request to the authors for information on the contributing centres was denied.

5.Data from Australia are not compatible with government reports (too many cases for just five hospitals, more in-hospital deaths than had occurred in the entire country during the study period). Surgisphere (the data company) have since stated this was an error of classification one hospital from Asia. This indicates the need for further error checking throughout the database.

6.Data from Africa indicate that nearly 25% of all COVID-19 cases and 40% of all deaths in the continent occurred in Surgisphere-associated hospitals which had sophisticated electronic patient data recording, and patient monitoring able to detect and record “nonsustained [at least 6 secs] or sustained ventricular tachycardia or ventricular fibrillation”. Both the numbers of cases and deaths, and the detailed data collection, seem unlikely.

7.Unusually small reported variances in baseline variables, interventions and outcomes between continents(Table S3).

8.Mean daily doses of hydroxychloroquine that are 100 mg higher than FDA recommendations, whereas 66% of the data are from North American hospitals.9.Implausible ratios of chloroquine to hydroxychloroquine use in some continents.10.The tight 95% confidence intervals reported for the hazard ratios appear inconsistent with the data. For instance,for the Australian Data this would need about double the numbers of recorded deaths were reported in the paper.The patient data were obtained through electronic health records, supply chain databases,and financial records. The data are held by the US company Surgisphere. In response to a request for the data Professor Mehra replied:“Our data sharing agreements with the various governments, countries and hospitals do not allow us to share data unfortunately.” Given the enormous importance and influence of these results, we believe it is imperative that:

1.The company Surgisphere Provides details on data provenance. At the very minimum, this means sharing the aggregated patient data at the hospital level (for all covariates and outcomes)

2.Independent validation of the analysis is performed by a group convened by the World Health Organization,or at least one other independent and respected institution. This would entail additional analyses (e.g. determining if there is a dose-effect)to assess the validity of the conclusions

3.There is open access to all the data sharing agreements cited above to ensure that,in each jurisdiction,any mined data was legally and ethically collected and patient privacy aspects respected

In the interests of transparency, we also ask The Lancet to make openly available the peer review comments that led to this manuscript to be accepted for publication.This open letter is signed by clinicians, medical researchers, statisticians, and ethicists from across the world. The full list of signatories and affiliations can be found below.

Click here for link at https://zenodo.org/record/3865253#.XtEi_jpKjZs

If there is one thing COVID has definitely done

it is significantly degrade the quality of medical publications.

We now joke that CVs should have 2 separate sections, non-COVID and COVID publications. Because the bar to publish a COVID study is considerably lower than the bar to publish a non-COVID study in the same journal.

Some of it is the speed with which these journals are trying to get information out there - the review process is hurried and some corners are cut. I get that. But major journals have published some pretty bad science in the last 3 months.

Damn..even economists have to make their data available *

The world’s most boring data *

The world’s most boring data *

An imperfect competition *

Dismal people cannot be trusted *

The world’s most boring data *

I don't know, corn and rice can have their moments, too. *

Cut that out! *

This is not uncommon.

There is a fair amount of heatlhcare research which is a product of analysis from multi-institutional datasets collected and managed by a third parties. The ability to release that data is governed by contractual arrangements for participating hospitals. For the companies/entities using this as a mechanism to sell healthcare analytics platforms, simply releasing the data would be like asking Coca-Cola to release the formula for Coke.

It's perhaps a separate discussion about the appropriateness of such arrangements. My only point is to say that how things worked with this specific study is not necessarily a deviation from current norms.

I don't disagree, but for a topic this controversial

And when the group doing the study is a data analytics group with proprietary software, and the study is published in one of the premier journals in the world without the ability to cross check the veracity of the data, it is going to create problems and at the very least create the perception of possible conflict of interest.

I'm not saying the scrutiny isn't warranted

given that this is what you might consider a high-leverage situation. It's just that for people who may not be familiar it seems like the logical explanation is that there is a cover up, when the reality is that this is fairly commonplace. It should also be noted that contrary to what the open letter states, Lancet only has data sharing requirements for clinical trials. They encourage data sharing for other types of studies, but do not require it.

I also think there are a variety of ways this could be handled outside of publicly releasing the data. For example, allowing an independent statistician access to the data and code would likely be sufficient.

Thanks *

As I said to Cash below

Given the PHI concerns

1) We will see how they navigate the release of the data
2) I'm still amazed there wasn't an ethics review

I will be dumbfounded if there isn't more to that story. *

Reposting what I said below.

Mandeep trained at my institution and my colleagues publish with him regularly. I know he’s a man of good faith and good will. But this is the messy part of science as you know. It’s done out in the open.

I look forward to the response to these thoughtful concerns.

Cash

PS - I want to stress that there remains no good evidence supporting the use of HCQ+AZ in COVID, and the risk of taking those medicines in combination in unequivocal.

I completely agree regarding HCQ and AZ. But a cornerstone

of my posts, both on this board and on my Facebook blog, has been taking the politics out of various topics, and analyzing them as we would at a journal club for the residents. It is only fair, as I have heavily criticized almost every "pro" HCQ published article and pointed out the near fatal flaws, to do the same when something is published that happens to align with what I strongly believe will eventually be proven.

This is going to turn into a real

snake pit with the Lancet's peer-review process front and center.

Yup especially given past mess ups

Lancet was the journal that published the dubious autism and vaccine study.

For a little more context

"Negative findings from largest study to date prompted the move

The World Health Organization has temporarily halted a large, international study of hydroxychloroquine for the treatment of Covid-19 due to new evidence linking the drug to an increased risk of death in hospitalized patients.

The largest international study published to date examining hydroxychloroquine and chloroquine in patients hospitalized with Covid-19 showed use of both drugs to be associated with decreased in-hospital survival and increased frequency or ventricular arrhythmias.

Trial findings, published recently in The Lancet, also failed to confirm a benefit for hydroxychloroquine or chloroquine, when used alone or with a macrolide, on other in-hospital outcomes."

Also, I eagerly await the day people stop using the words "the largest study to date". I get it, and I know why it's said, but given the times, it's not helpful in these studies. There are other bigger studies going on at the moment, but it adds a sense of "authority" to the findings, and in this case, seems like some Bullshit.

Some people questioning the decision

Click here for link at https://www.physiciansweekly.com/covid-19-who-suspends-hydro

Oof

Those guys need to open up the books.

I know it's too much to ask, but people really should chill the F out.

What FaytlND posted about the WHO not pushing forward with HCQ in Solidarity is the most important thing when it comes to this stuff, and this study was a driving force behind that decision.

But this one needs to be looked into, and the Lancet should have done better. I can't believe that they posted that study without an ethics review. Jaysus. The peer review comments will probably be very interesting. People becoming cheer leaders for drugs during this has been extremely disappointing.

The Lancet will probably issue that retraction, and most people won't notice however, which is also a problem.

I wonder if the WHO is going to start back up again? What a cluster.

EDIT - for those interested, the study in question is linked below

Click here for link at https://www.thelancet.com/journals/lancet/article/PIIS0140-6

It's also important to say that the WHO did

the responsible thing. They paused enrollment in that arm pending review by their own DSMB. That's how it's supposed to work when you're running a trial and new data emerges which calls into question the safety of your intervention.

Agreed

This will be interesting. Didn't mean to insinuate they did the wrong thing.

You did not.

Just putting it out there because some of the coverage/reaction I've seen elsewhere is making it seem like the WHO was potentially in the wrong. And given their rocky past few months it risks getting lumped in with other (valid) criticism.

I think it's worth waiting to hear from Mehra et al.

Also from The Lancet.

I expect one will be forthcoming soon.

Cash

Yep

The thing that jumped out to me was the lack of an ethics review. Given how the data was collected, that's going to get sticky across the board.

I suspect the findings were correct, but given how the data was collected, I'm shocked they didn't go through an ethics review for that. I'm not sure how they are going to unpack that.

If it was de-identified data, it would not necessarily

require a formal IRB review. In general, that kind of study would get an exemption, as opposed to other studies which may get referred for a "full board" review (or an expedited review when it's identified data with associated "minimal risk"). Though it still requires registration of the study with an IRB and there would be documentation of that decision.

It's also possible it just wasn't explicitly documented even though it happened, which is a common occurrence (despite journal requirements which mandate it being included).

If it was truly de-identified data, it doesn't fall under

the purview of the IRB according to federal regulations. IRBs are charged with reviewing human subjects research. The definition in the federal regulations defines a human subject as either somebody you interact with personally (i.e. face to face) or who you get identified data on.

So technically, this doesn't need an IRB review.

What HTown said is also correct - despite that being the federal regulation, most institutions require a submission to the IRB for the IRB to "agree that all data elements are de-identified and therefore the project does not meet the definition of human subjects research."

Usually, that "determination" by the IRB is then listed in the manuscript as the ethics or IRB review.

Yeah

But one of the authors said they weren't able to make the data available because on contractual arrangements. I think the IRB would at least dig into that (and I'm breaking out into hives again typing IRB).

EDIT - anytime we dealt with data, even if it didn't include PHI, the IRB and wanted to look at it and make sure we weren't actually pulling PHI, they never took our word for it. And when we dealt with the third parties, they usually looked into the contracts to determine what was the use case for the data, who owned the data when the study was over, etc. Granted, our IRB process was tough but thorough (said in my Maude Lebowski voice), and maybe that was just us, but rarely did we go through the submission without someone stopping us to ask the data questions. Even my pissant grad school study on Demographic use of Epic Tools caught their eye, even though it wasn't PHI, they couldn't get past the Demographics and EMR data. We had to explain exactly what fields out of Epic were being pulled and how they were being used. We didn't pull ages, we used age bands, etc, but that was not a fun process, and that was for a silly grad school study, not the ones I did for work.