The evidence for plant-based diets being better is about as good as that for Mr. Jesus being real.

I had bacon for breakfast

Less Meat, More Plants: A Rules of Evidence Controversy (Part 1)
GARY TAUBES
FEB 20

“Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.”

George Box, Science and Statistics, 1976

Before we get to the mice and tigers, let’s acknowledge the undeniable: people who eat plant-based diets tend to be healthier than those who don’t. The flip side is equally irrefutable: people who eat a lot of meat – red and, particularly, processed – tend to be less healthy than those who don’t. The meat-eaters, and particularly the processed meat eaters, are more likely to get heart disease and (apparently) diabetes as well. As the New York Times summed it up in November, citing a newly published analysis, “Plant-based foods are linked to a lower risk of heart disease and diabetes.”

Yes, they are. “The case has never been clearer,” the NYT article declared in its opening sentence: “Eat less bacon and more beans.”

But now we have to take issue with the wording: Yes, the evidence has never been more clear, but the case itself – what that evidence tells us about the relative benefits and harms of bacon and beans, meats and plant-based foods -- is at best still plenty muddled. T that people who eat plant-based diets tend to be healthier than those who don’t is a fact. It does not imply, however, that anyone would be healthier if they chose to eat less bacon and more beans. The former could be true, as we accept it is, and yet doing the latter could cause more harm than good. How would we know? That simple question is at the heart of a half-century of nutrition controversy.

I’m guessing that the New York Times reporters no longer ask that question because the health benefit of plant-based eating has for so long been a foundational assumption of nutrition science. It has evolved to the status of common sense. (In a later Substack, we’ll discuss the history of this belief in more detail).

This is why Michael Pollen, once the most influential journalist on the nutrition beat, included mostly plants as the centerpiece of his wildly-popular, nutritional mantra: eat food, mostly plants, not too much. “In all my interviews with nutrition experts,” Pollan explained in his 2008 best-seller In Defense of Food, “the benefits of a plant-based diet provided the only point of universal consensus. Even nutrition scientists who have been chastened by decades of conflict and confusion about dietary advice would answer my question So what are you still sure of?’ with some variation on the recommendation toeat more plants.’” With the nutritionists so confident of this one proposition, dietitians, physicians and journalists came to faithfully embrace and widely disseminate it as well.

Take, for instance, Adam Cifu, a cardiologist, and John Mandrola, an internist and professor of medicine, who were recently discussing the nature of a healthy diet in their typically very sensible Sensible Medicine Substack and podcast. Cifu and Mandrola approach medical science with an appropriately skeptical mindset, and they clearly understand the thorny issues of nutrition research. But the mostly-plant philosophy stands as an exception to their skepticism. They described the “consumption of plant-based foods” as unambiguously “good.” No caveats necessary. In many ways, their thinking is not all that different from that of my health-conscious mother back in the 1960s, when she insisted that I eat my vegetables. She wouldn’t think of preparing a family dinner without them. It was a given.

If you can’t trust common sense…?

But the devil, as ever, is in the details. What’s considered nutritional common sense, what’s considered undeniable, has changed over the decades since my mother was my source of nutritional wisdom. Sixty years ago, serving green vegetables along with meat, fish or fowl, as my mother would, seemed like sound nutritional thinking. Now, the less meat (and to a lesser extent, fish and fowl), the more vegetables the better. Sixty years ago, to be health-conscious was to eat a diet with plants, specifically fiber-rich, green leafy vegetables. Now, it’s to eat a diet that is mostly or even all plants. Nutritional common sense reflects a moment in time.

And common sense has its limits. That everyone believes something is true does not, of course, necessarily mean that it is. That fact alone is among the very good reasons why science evolved over the past several centuries; to establish what is true, whether common sense or not, to generate an evidence base that makes it exceedingly difficult to believe anything else.

Here's the first context in which Box’s mice and tigers come in. It would be a mistake to assume that the meat-vs.-plants controversy is merely about the nature of a healthy diet. That issue is the mice. It’s important, of course, to our immediate conception of a healthy diet and to staying healthy, but far more important, is how it influences our conception of the evidence necessary in the future to establish reliable knowledge. If our health and our children’s health depend on what we eat and drink, on the drugs and supplements prescribed to treat or prevent disease, what level of evidence should be necessary before our public health and medical authorities buy in, and then disseminate what they believe to be true as advice and guidelines to us?

Borrowing from the sociologist of science Robert Merton, if we want to establish whether what we think we know is really so, we first need to understand the nature of the evidence that should convince us. If we are allowed to decide that the reliability of that evidence depends primarily on whether it agrees with our preconceptions, and how difficult it would be to get stronger evidence, we are no longer engaged in a science-based endeavor but one that is faith-based.

Yet this is the position now embraced by authorities arguing for the health benefits of plant-based diets:. Rather than demand that researchers generate reliable, if not unambiguous evidence for an issue of such importance to human health – or set out to do it themselves -- nutritionists and public health authorities (and the journalists on the beat) are telling us that we should trust the unreliable evidence we have, because, in effect, it’s the best we’ve got.

This is the tiger abroad in the meat-vs.-plants dispute. It is a scientific controversy and by no means a trivial one. It’s not about what to believe (the mice). It’s about why we should believe it, based on what evidence (the tiger). Does that evidence justify belief? And if it doesn’t, can health authorities ever be justified in believing that it does?

Another way to think of this issue is to imagine the meat-vs.-plants controversy as a legal case: Prosecuting attorneys are arguing that red and processed meat are dangerous to our health. Defense attorneys are arguing the opposite. The two legal teams have come to an agreement on the rules of evidence: not just on what evidence is relevant to the case, but what’s admissible in court and can be discussed before the jury. They disagree on what that evidence means and how it can or should be interpreted. The jury’s decision, ideally, should depend only on which interpretation of the evidence is the correct one. That’s the conflict.

Thank you for reading Unsettled Science. This post is public so feel free to share it.

Share

The evidence (unvarnished, unspun)

So what’s the admissible evidence in the meat vs. plants dispute? It falls into two categories.

The great majority comes from epidemiologic surveys. In these studies, large groups of people (cohorts, in the lingo) are surveyed about their diet and lifestyle and then followed for years or decades to see who gets sick, from what, and who doesn’t, and then how those illnesses associate with what these people were eating. These are called prospective studies because they track the cohorts forward through time (as opposed to retrospective, which ask the participants about past behavior), and they are observational because the researchers only observe the lives of the participants and then try to draw conclusions from those observations.

The other evidence is from randomized controlled trials. In these trials, participants are randomized either to a control group or an intervention. Each group is counseled to follow a specific dietary pattern and then followed typically for months to years: as long, usually, as the researchers can afford. The researchers tabulate the morbidity (sickness) and mortality (death) in the different groups, and that’s what they report. These studies are experimental, not observational, because the researchers actively intervene in the lives of the participants in the trials – changing what they eat, or at least counseling them to change what they eat – to see the effect that the intervention has on health.

Comprehensive reviews of all the evidence on red and processed meat were presented in 2019, in a series of four systematic analyses published in Annals of Internal Medicine. The authors (18 to 20 depending on the paper) hailed mostly from McMaster University in Ontario, Canada. Gordon Guyatt, a biostatistician, clinical epidemiologist, and a pioneer of the field of evidence-based medicine, was the most prominent of the group.

An accompanying editorial, co-authored by Tiffany Doherty and Aaron Carroll of the Center for Pediatric and Adolescent Comparative Effectiveness Research at the Indiana University School of Medicine, described the analyses as both “the most comprehensive review to date,” which they were, and “sure to be controversial,” which may have been an understatement.

What followed was the last time this controversy with all its stakes (mice and tigers) was aired both in the media and the medical literature. As such, it allows us to fully understand the nature of the dispute and the positions taken by the competing sides, beginning with the evidence itself.

Here’s a summary of the evidence from the observational studies (from one of the four Annals reviews). This is the evidence supporting the proposition that people who eat plant-based diets tend to be healthier than those who don’t:

Eligible studies that followed patients for 2 to 34 years revealed low- to very-low-certainty evidence that dietary patterns lower in red and processed meat intake result in very small or possibly small decreases in all-cause mortality, cancer mortality and incidence, cardiovascular mortality, nonfatal coronary heart disease, fatal and nonfatal myocardial infarction, and type 2 diabetes. For all-cause, cancer, and cardiovascular mortality and incidence of some types of cancer, the total sample included more than 400 000 patients; for other outcomes, total samples included 4000 to more than 300 000 patients.

And here’s the evidence (from another of the four reviews) from the clinical trials. This is the evidence addressing the proposition of whether eating less bacon and more beans (or counseling people to eat less bacon and more beans) makes people healthier than they otherwise would have been.

Of 12 eligible trials, a single trial enrolling 48 835 women provided the most credible, though still low-certainty, evidence that diets lower in red meat may have little or no effect on all-cause mortality (hazard ratio [HR], 0.99 [95% CI, 0.95 to 1.03]), cardiovascular mortality (HR, 0.98 [CI, 0.91 to 1.06]), and cardiovascular disease (HR, 0.99 [CI, 0.94 to 1.05]). That trial also provided low- to very-low-certainty evidence that diets lower in red meat may have little or no effect on total cancer mortality (HR, 0.95 [CI, 0.89 to 1.01]) and the incidence of cancer, including colorectal cancer (HR, 1.04 [CI, 0.90 to 1.20]) and breast cancer (HR, 0.97 [0.90 to 1.04]).

The conclusions to be drawn about dietary advice seemed clear to the authors. It was strongly implied by their description of the evidence from both the trials and the observational studies as “low-to very-low certainty:” We should feel free to eat as we’ve been eating because the evidence was insufficient, one way or the other, to advise otherwise. Or to be precise, from a fifth paper translating the results into guidelines:

The panel suggests that adults continue current unprocessed red meat consumption (weak recommendation, low-certainty evidence). Similarly, the panel suggests adults continue current processed meat consumption (weak recommendation, low-certainty evidence).

It was this conclusion that was sure to be controversial, or, in non-technical terminology, that induced all hell to break out.

Thank you for reading Unsettled Science. This post is public so feel free to share it.

Share

Pre-emptive retraction?

Even before the publication of the reviews, a dozen influential nutritionists, public health authorities and physicians wrote to Christine Laine, then the Annals editor-in-chief, requesting what they called “pre-emptive retraction” of the papers. They had “grave concerns,” they wrote, “about the potential for damage to public understanding, and public health.”

The letter was the product of an organization called the True Health Initiative, founded by a Yale-associated MD named David Katz and dedicated to the proposition that we already know that a “plant-predominant diet in balanced composition stands to reduce the risk of all chronic diseases.” By implication, any analysis or review of the evidence that concludes otherwise must be in error. Many of the 14 authors of the letter had long been committed to the idea that eating animal products is either ethically (Neal Bernard, for instance, president and founder of the animal-rights group, the Physicians Committee for Responsible Medicine) or environmentally unacceptable and that these factors had to be taken into account in such reviews.[1]

A half dozen signatories to the letter hailed from the Harvard School of Public Health. They, too, may have had ethical and environmental issues about meat consumption. They had certainly championed a plant-based diet for decades (as Nina discussed here). But they may have been equally concerned about what the analysis implied about the fruits of their research: that the observational/epidemiological studies on which they had built their careers could offer up, at best, only low quality of evidence. Indeed, Doherty and Carroll[2] had gone so far as to suggest in their accompanying editorial that “it may be time to stop producing observational research in this area,” which would put these Harvard nutritional epidemiologists out of business.

Either way, Katz and his co-authors failed to convince the Annals editors of their argument. Their primary complaint was that the reviews had “omitted prominent trials” that appeared to satisfy the requirements of the review. But they cited only two trials, “PREDIMED” and the Diabetes Prevention Trial, both of which seemed irrelevant to the controversy: neither included trading off red and/or processed meat with plant-based foods in the interventions they had tested. (I can imagine the Annals editors thinking to themselves, as I found myself doing, "If this is all they’ve got…&rdquo

Publication of the reviews “stunned scientists and public health officials,” The New York Times reported, “because [they] contradicted longstanding nutrition guidelines.” Pollan might have described them as contradicting “the only point of universal consensus” among nutritionists.

The resulting controversy then played out in articles, comments and letters to editors, not just in the Annals of Internal Medicine (twice), but also in the American Journal of Clinical Nutrition, JAMA and even Diabetes Care. [3]

Physicians and various academic authorities wrote into Annals, offering explanations for why such seemingly respected academic researchers might have published what amounted to nutritional heresy. One letter in Annals accused these researchers of having “abdicated the moral responsibility that physicians have to our society” by ignoring the contribution of red meat consumption to climate change. Another suggested that the interpretation of the evidence was biased by the researchers eating habits: “none of the [co-authors] is vegetarian,” the author pointed out, and “the panel members may have interpreted the evidence differently if the eating patterns of participants had better reflected the diversity of alternatives to an omnivorous diet.”[4]

Guyatt and his co-authors responded by arguing (as we believe here at Unsettled Science) that data analysis on scientific issues should be independent of ethical or moral beliefs. “We believe the issue at hand,” they suggested, “should be argued not on the basis of who was on the panel, but whether the panel judgments were appropriate.” (Or to borrow again from Robert Merton, “Science must not suffer itself to become the handmaiden of theology or economy or state.&rdquo

This left the rules of evidence used in the analysis as the great point of contention. On what basis could Guyatt et al write of the observational (or epidemiological) evidence as “low to very-low quality?” If they were right to do so, then the mostly-plants position should never have risen to the level of nutritional consensus to begin with, and the nutritional epidemiologists themselves had a lot of explaining to do.

These rules have always been a critical issue in medical decision-making, and even more conspicuously so since the beginning of the evidence-based medicine (EBM) movement in the mid-1980s. In 1992, Guyatt chaired a 31-member Evidence-Based Medicine Working Group, which had asserted the obvious: medical decision-making depended on “the understanding of certain rules of evidence;” which are those “necessary to correctly interpret literature on causation, prognosis, diagnostic tests, and treatment strategy.”

Upgrade to paid

The GRADE problem

In 2001, Guyatt and his (mostly) McMaster University colleagues began developing an “evidence to decision” program called GRADE (The Grading of Recommendations Assessment, Development and Evaluation) that would incorporate the rules of evidence into its methodology and become widely used and embraced internationally as a tool for making decisions about diagnosis, treatment, and prevention of diseases. It was GRADE that Guyatt and colleagues used in their 2019 reviews to assess the dangers of meat consumption and that had found the evidence to be so uncompelling.

The use of GRADE for such a task became the primary point of contention in the post-publication firestorm. Critics argued that the decision-making methodology was inappropriate, and Guyatt et al had to respond by reminding them just how conventional and widespread its use had become: “The GRADE approach has been adopted by over 110 organizations worldwide, including the World Health Organization, the Cochrane Collaboration, the American Thoracic Society, the American College of Physicians, and the Centers for Disease Control.”

But if employing the same-decision making criteria for nutritional advice that is used and embraced by public health organizations and medical organizations worldwide fails to confirm the mostly-plant universal-point-of-consensus, the obvious question is why?

The answer: GRADE treats the observational (epidemiological) studies that constitute essentially all of the evidence for the plant-based diet to be “low-certainty evidence,” at best, which means in an ideal world and with an ideal study. GRADE then downgrades this level of certainty further, depending on the details of the study, the fact that this is never an ideal world, and the extent to which the study suffers from the various biases and confounding factors that make all scientific investigations so very challenging, but may be particularly pernicious in nutritional epidemiology.

While journalists and even medical and public health authorities will often refer to randomized controlled trials as the “gold standard” of evidence, GRADE simply considers those trials the only source of high-quality evidence—i.e., the only source of evidence that can be trusted to tell us something on which we can base our convictions. This level of certainty, too, can be downgraded based on the details of the trial, but it starts off high. Not so the evidence from observational/epidemiological studies.[5]

Here’s Guyatt and his GRADE collaborators on the precise meaning of these different levels of quality of evidence, from a 2008 BMJ article introducing GRADE widely to the medical community:

High quality— Further research is very unlikely to change our confidence in the estimate of effect

Moderate quality— Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate

Low quality— Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate

Very low quality— Any estimate of effect is very uncertain

These definitions raise another obvious question, again the one that sits at the heart of the meat-vs.-plants controversy: what good is an “estimate of effect” if you can have little confidence that the effect itself is real, let alone how large or small it might be? In this case, if the observational epidemiology – i.e., what nutritional epidemiologists do for a living -- can provide at best low-quality evidence, and that evidence can only become less certain, lower quality, depending on the details of the study, what good, then, is the nutritional epidemiology?

In the best case scenario – ideal world, ideal study -- further research is “very likely to have an important impact” on the estimate of the effect. In the worst case (what we might call reality), these epidemiological studies provide no meaningful information at all: “Any estimate of effect is very uncertain.” Even an effect that seems positive (a good thing) can just as likely be negative (a bad thing) and vice versa. There’s no way to tell.

A consensus in medicine and public health, not so nutrition

In 2008, when Guyatt and his colleagues published their BMJ article on GRADE, the subtitle described it as “an emerging consensus on rating quality of evidence and strength of recommendations.” But this consensus was emerging only in medicine and public health. Those were the organizations that embraced the use of GRADE and accepted its dismissive assessment of observational epidemiology.

The nutritionists and, specifically, the nutritional epidemiologists, never signed on. They may not have been paying attention to the implications. (Although they certainly knew that the absence of randomized controlled trials supporting their beliefs was a problem, as discussed here and here). When Guyatt and his colleagues used GRADE to assess the putative dangers of red and processed meat,, their analyses forced the nutrition community to either deal with the implications or justify otherwise.

Not only had Guyatt et al concluded that the universal consensus on diet had no basis in evidence, implying that the nutritionists were laboring under serious delusions, but they also made clear that observational studies could not provide sufficiently reliable evidence (high enough quality) to tell us what to eat at all. If the consensus in medicine and public health is that evidence from observational studies is unreliable, why is this evidence considered of value in nutrition (let alone, as Nina has pointed out, the near-entire for the Dietary Guidelines for Americans)?

This is why the scientific controversy over the Annals papers reverberated through the literature for more than a year afterward. There were tigers abroad, the mice were the least of the problems.

The controversy faded, of course, as controversies do. Guyatt and his colleagues moved on to other issues. The nutritionists and epidemiologists (and defenders of the health benefits of plant-based diets) who took umbrage with the Annals papers, also went back to business as usual. They had defended their beliefs, and nothing more needed to be said. The belief that a mostly plant diet is best continues to be a point of almost universal consensus, and the media, with its day-to-day approach to the news, follows along reporting on the results of each observational study as it arises, as though they provide meaningful evidence. Inconvenient truths, in science as in life, can have short half-lives.

This discussion of science will be continued in Part 2. I’ll address the simplest way to think about the low-to-very-low-quality-evidence issue. Yes, we’re talking here about associations, which do not imply causality, but I’d like to explain why these associations in nutrition are particularly unreliable. I’ll also have a response to the many nutritional epidemiologists who say, in effect, “Trust us, we’re the experts.”

Unsettled Science is a reader-supported publication. To receive new posts and support our work, consider becoming a free or paid subscriber.

Upgrade to paid

[1] As Katz has explained, “The plight of our beleaguered planetary health is arguably the single great medical imperative of our time, for failure to address this one may soon make all others moot. There are no healthy people on an uninhabitable planet. No public health crisis ever encountered even hints at the menace of our extinction- but the collapse of planetary health and the loss of biodiversity do exactly that. Consequently, there can be no “best” diet anymore that isn’t as good for planet as it is for people. We can hope to be healthy, vital people on a healthy, vital planet- or we will have no hope of being healthy, vital people at all.”

[2] Carroll is now the chief health officer of Indiana University.

[3] It was also discussed in the New York Times by Aaron Carroll in his Upshot column, in which he covered medical and nutrition issues from an evidence-based perspective.

[4] Another explanation was offered by The NY Times, which implied that the authors’ nutritional heresy was the product of undue industry influence: Bradley Johnston – a primary author, with Guyatt, of the reviews -- had received funding four years earlier from the non-profit, industry-funded International Life Sciences Institute (ILSI). That money had gone to an analysis of the evidence on sugar consumption, a paper that had been published three years earlier and so, within the time window of concern that journals specify for conflict of interest declarations. “I have no relationship with [ILSI] whatsoever,” Johnston told the paper. But he once did. JAMA later reported that the academics who signed the letter demanding pre-emptive retraction – Katz foremost among them – also had copious industry relationships, including with ILSI.

[5] “In the GRADE approach,” explained Guyatt and his colleagues in a 2011 paper, “both randomized trials (which start as high quality evidence) and observational studies (which start as low quality evidence) can be rated down if relevant evidence comes from studies that suffer from a high risk of bias.”