You can enjoy this ‘chicken’ with a clear conscience. It contains no animal products, and producing it takes up half as much land. | Image: Florian Kalotay

Coffee is carcinogenic, so several studies used to claim. They were wrong – as we know today. Whoever eats egg yolks will have high cholesterol, they said. Also wrong, as we again know today. The nutritional sciences are known for producing studies whose results are later proven to be a mistake. But why is it so difficult to get unambiguous answers from nutrition studies?

Sabine Rohrmann is a nutritionist at the University of Zurich. When this question is put to her, she replies with another question: “What did you eat yesterday?”. We might well remember whether we had a hamburger, a vegan vegetable risotto or a salad. But what was actually in the risotto? Or in the salad dressing?

Forgetting to count the cake

What seemed a simple question soon proves to be more complicated. It also demonstrates one of the biggest problems for nutrition studies. The data they gather are uncertain because they’re based on information from the study participants themselves. This means that their data suffer from the so-called self-reporting bias – distortions that arise because test subjects provide false or incomplete information. This might be because they have a bad conscience about something, or simply because they have forgotten certain things. Who would willingly admit that they didn’t eat one piece of cake, but actually wolfed down two? Who can remember the little biscuit that we ate with our coffee in a restaurant, let alone the exact time we did so?

“We know about this problem, of course, and we try to account for it with our survey methods”, says Rohrmann. For example, researchers will ask separately about specific foodstuffs, such as meat or sweet things, and they show a selection of portion sizes in order to help their test subjects to remember. Nevertheless, they still get information that can be drastically incorrect. For example, the big ‘National Health and Nutrition Examination Study’ in the USA found that the calorie intake people stated was unrealistically low. If they took in as few calories as they claimed, none of them would have been able to survive.

It’s not just our eating behaviour that is complex. So too is the food we eat. Cheese from Switzerland has different ingredients from French cheese, and the same applies to convenience products from different manufacturers. “We know that we can’t determine our data down to the gram”, says Rohrmann. But they nevertheless suffice for comparisons between different eating habits to find out the effect that different diets have on us.

They also need to include the other factors that have an influence on our health, some of which have a considerably bigger impact on us. Such as whether we smoke or not, drink a lot of alcohol and get enough exercise. But our level of education and other sociocultural factors are also important, and these can hardly be measured. In practice, researchers can never correct their data so as to cater for absolutely all disruptive factors. But it is these factors – together with distortions when data-gathering – that cause uncertainties. They become a kind of background noise in the data, in which certain effects can fail to be heard. On the other hand, it can also happen that chance correlations emerge out of this data noise that cannot be confirmed by subsequent studies.

“Essentially, we should not overrate the results of individual observational studies”, says Murielle Bochud, who is a nutritionist and epidemiologist at the University of Lausanne, and a member of the Federal Commission for Nutrition that advises the Federal Council. “They are always just one piece of a very complex jigsaw”.

Abandoning vegetables for decades?

More precise results can be provided by so-called intervention studies. Just like studies in the medical world, these investigate the effect of individual measures using test subjects and a control group. These studies allow researchers to find out, for example, both whether there is a connection between our salt intake and high blood pressure, and whether or not the former actually determines the latter.

However, it is difficult to carry out such studies cleanly. If we want to find out whether illnesses are triggered by a specific group of foodstuffs – such as red meat or vegetables – then nutritionists have to carry out decades-long studies with thousands of test subjects who have been randomly selected, and they also have to have a representative control group. This is neither practicable nor ethically justifiable. “Intervention studies generally can’t be carried out for very long, because the test subjects suffer from dwindling motivation after a while”, says Bochud. What’s more, the researchers have only limited means of verifying just how disciplined their test subjects have been in following the instructions given them at the start of the study. As a result, intervention studies are riddled with uncertainties.

Tortured data

Besides these practical difficulties, researchers also have an incentive to squeeze as much as possible out of their data. “It’s not for nothing that the statistics community has the saying: “Torture the data until they confess”, says David Fäh, a nutritionist at the Bern University of Applied Sciences. This is because a statistically significant connection will get your study published in a more prestigious journal. And this in turn determines your scientific success. Careers depend on it. “We scientists should try and emancipate ourselves from this pressure to publish”, says Fäh, although he admits that this is easier for him to say as a professor than it would be for young researchers who still have to get established.

Fäh says he has himself become ever more cautious over the course of his career. As an example, he mentions a study that he carried out several years ago. It was about whether high cholesterol is also linked to an increased risk of cardiovascular diseases here in Switzerland. Fäh was able to prove a connection, but only when cholesterol was very high, and only when he subjected his statistical evaluation to specific limitations. Instead, Fäh took a broader view and as a result got a less spectacular, but more realistic result, according to which the connection is neither clear nor generally applicable.

There are times when it’s easier to use tricks to reach a statistically significant, seemingly meaningful result. “One way of preventing this would simply be to insist on greater transparency”, says Fäh. “It is important for the authors of studies to stipulate how they deal with the boundary between statistically significant and insignificant”.

One prominent, negative example was PREDIMED, a large-scale investigation into the impact of a Mediterranean diet, back in 2013. This study found clear advantages to a diet that included a lot of olive oil, legumes and fish, and was initially taken up enthusiastically. But when its authors finally gave in to pressure from the research community and several years later revealed details of their methods, several shortcomings became evident, and the study’s results proved to be largely illusory.

This was no isolated case. The American epidemiologist John Iaonnidis is a prominent critic of nutrition studies, and he proved several years ago that the findings of many studies melt away into nothing upon closer inspection. He picked 50 random ingredients from cookbooks, and found that 40 of them had already been investigated for their carcinogenic properties in more than 260 individual studies. More than 70 percent of those studies had found that these ingredients had an impact on the risk of cancer – though some had contradictory findings. Beef, bread and tomatoes had variously been found to cause both an increase in cancer, and also a decrease in it. In a second step, Ioannidis looked at just the meta-analyses for these 40 ingredients – overview studies that drew on the results of all individual studies that had previously been carried out, regardless of their quality. Less than 30 percent of these meta-analyses found that these ingredients had an influence on the cancer rate – and they generally found that influence to be less than before.

Increasingly individual

Today, many nutrition researchers think that it is excessive to make any individual foodstuff responsible for the state of our health or for any specific disease – whether it be meat, nuts or butter. “The impact exerted by individual foodstuffs is so small that it is simply inconsequential when compared with other factors”, says Hannelore Daniel, a nutritional physiologist and professor emeritus at the Technical University in Munich, Germany.

These other factors also include the fact that different people react differently to different foodstuffs. Daniel has proven that such individual differences even arise in a simple glucose tolerance test. A homogenous group of test subjects drank a glass of water with glucose in it. Over the ensuing hours, the blood sugar level of a third of them rose as it says it should in all the textbooks. But in another third of them, it only rose half as much. In the final third, it remained unchanged. “Why this is the case is something we still don’t understand”, says Daniel.

Such individual differences are being researched intensively today. Personalised nutrition is still in its infancy. But Daniel thinks it likely that individual nutritional recommendations will prove beneficial in future. She could imagine an app being developed that helps you to make shopping decisions in the supermarket. Such recommendations ought not to depend solely on health criteria, but also take ecological considerations into account. Daniel finds this equally important.