KATELYN JETELINA AND KRISTEN PANTHAGANI, MD, PHD

APR 27, 2023

Last October, the Florida Surgeon General issued new guidance around the COVID-19 vaccines: men aged 18-39 should not receive COVID mRNA vaccines. This policy change was based on an analysis they performed that found “an increased risk of cardiac-related death among men 18-39.”

This finding was alarming and inconsistent with analyses from CDC and independent scientists. So, two outside epidemiologists, Drs. Kat Wallace and Jon Laxton and the Tampa Bay Times requested the unreleased drafts through the Freedom of Information Act. They were curious how Florida came to this conclusion because if true, it was a huge revelation.

Those records tell a dramatic story of how science should not be done.

Drafts
There were at least six versions of this analysis and report. The drafts show very different results and conclusions compared to the final version of the report released last October. Here’s how it changed.

Version 1: Completely opposite results
Version 1 of the report is the most distinct from all the others. Importantly, this first version found no increased risk of death in any age/vaccine group.

The table below summarizes the key results in version 1 compared to the final version using color codes:

Green: analysis found vaccination is associated with reduced risk of death;

Red: vaccination is associated with increased risk of death; and

Gray: there was no association either way.

In version 1, you see ALL GREEN AND GRAY—none of the results, for any of the ages or subgroups analyzed, showed any increased risk of death after vaccination. But in the version that was released in October, many of those results changed to red.

Why is there a lot of red now? The study design changed over time.

In version 1, the risk period was weeks 1-6 after vaccination, and the control period was weeks 7-18 after vaccination.

In subsequent versions, the risk period was weeks 1-4 post-vaccine, and the control period was weeks 5-25 after vaccination.

It’s hard to say which risk period is “right.” Sometimes there are clear reasons why you should pick a certain subgroup or risk period. Other times researchers pick a reasonable cut-off and just go with it.

However, knowing the results from both designs, we do have reason to be concerned. The fact that this slight change in study design yielded opposite results suggests the results aren’t reliable. If you tweak your design parameters slightly, without a clear hypothesis-driven reason to do so, and the whole thing falls apart, that suggests your results may not be real but rather are just statistical noise.

Version 2: What about COVID infection?
One of the major criticisms back in October was the analysis failed to look at the risk of COVID infection and balance that with any risk of vaccination. It turns out, they did run that analysis—they just didn’t include it in their report.

In version 2, they evaluated risk of death after COVID-19 infection. They found the risk of death after infection was higher than risk after vaccination, for all age groups (even young men). The same pattern held true when they looked at only cardiac-related deaths.

In version 2, authors included this in the study’s text. But, in all subsequent versions, this text and analysis was removed.

Versions 3-5: mRNA vaccines have two doses
In versions 3-5, the authors added a sensitivity analysis. Sensitivity analyses are used to determine how reliable results are by switching up certain parameters or assumptions and seeing if the key results hold true.

In this case, they performed a sensitivity analysis that accounts for both doses of the mRNA vaccines.

The study design they chose is meant for single dose vaccines, because the risk and control periods are muddled when people get multiple doses. In the main analysis of the study, they ignored the first mRNA vaccine dose and only analyze results based on the last vaccine dose.

But when the sensitivity analysis accounted for the multi-dose scheduling, the significant results disappear. They found no association between vaccination and cardiac-related death in any of the subgroups analyzed.

Version 5 has the same data as versions 3 and 4, but includes the discussion/conclusion and limitations sections. Here is the last paragraph of the discussion/conclusion section:

When we get to the final, publicly posted, version of the study: The sensitivity analysis was completely removed. In fact, no mention of it was made.

Instead, the final version (version 6) concludes that COVID vaccines are associated with an “increased risk for cardiac-related mortality 28 days following vaccination.”

Evolution of conclusions
Throughout the drafts, there was a clear evolution of conclusions. Starting with no risk of death to young men to significant risk.

At its most basic level, science’s goal is to figure out what’s true about reality. It is common practice for scientific analyses to evolve over time. But when that happens, there must be clear methodological reasons for the changes. In this case, the study design, results, and conclusions were altered with no scientific explanation, no transparency, no hypothesis-driven reason provided, and no opportunity for formal peer-review.

Bottom line
This is not how science or evidence-based policy is done.

Love, YLE and KP

P.S. If the authors of versions 1-5 ever read this post, please know we appreciate how hard you tried to make the truth about this data known.

In case you missed it:

COVID-19 and sudden deaths: Separating fact from fiction

The science and business behind vaccine disinformation

Kristen Panthagani, MD, PhD is an emergency medicine physician at Yale. In her free time, she is the creator of the medical blog You Can Know Things. A previous version of this YLE post was published here. You can subscribe to her newsletter here.

“Your Local Epidemiologist (YLE)” is written by Dr. Katelyn Jetelina, MPH PhD—an epidemiologist, data scientist, wife, and mom of two little girls. During the day she works at a nonpartisan health policy think tank and is a senior scientific consultant to a number of organizations, including the CDC. At night she writes this newsletter. Her main goal is to “translate” the ever-evolving public health science so that people will be well equipped to make evidence-based decisions. This newsletter is free thanks to the generous support of fellow YLE community members.