Weak associations that we agree with are still weak associations

(Richard Morris) #1

Originally published at: http://blog.2keto.com/weak/

For decades we’ve been plagued with dietary regulations based upon studies that found weak associations generated from food frequency questionnaires. Here’s how it works: some researchers want to look into whether something people eat affects their health, so they come up with a survey, send it to say 100 people, and ask questions like Q25: Over…

New Study in NY Times
#2

BRAVO, @richard! I’ll be reading this several times. Very well done.

Two observations:

  1. Although the PURE study is of limited value, I did chuckle when I thought of the fact that sometimes people lie or otherwise skew their food diary responses toward what they think the researchers might want to hear. (Which, of course, is a near-fatal flaw for questionnaires.) But here, if that phenomenon was true, and bacon still won out? Wow…

  2. Although not appropriate for the topic of your post, you touched upon another major distinction people need to understand. The difference between absolute and relative risk. Something that the press has no idea about, and completely messes up when reporting on things.

(Crippie) #3

Amazing Richard! I am still catching up on all of your podcasts and just got to the Conventional Wisdom Show this morning where you and Carl spoke on this exact topic, and the problem with doctors and dietitians referencing epidemiological studies and not controlled randomized studies which are much more definitive. It really got me thinking. You specifically mentioned how someone responding to this survey might not even be 100% truthful, they may sway their answers to what they believe are the healthy answers. I know I am guilty, taking a survey on myself I tend to rate myself higher to look better. So these individuals in the epidemiological study may be doing the same thing(it really is human nature to want to make yourself look as good as possible) They get the survey and due to the conventional wisdom thing “bacon is bad, fruit is good” and they answer with a slight sway to make themselves more in line with that mentality, which can really disrupt any valid results a study like that could provide.

(Richard Morris) #4

If you get a effect over 200% then you can still do something with this kind of study.

But anything short of that is just so much trying to find Animal shapes in clouds. It’s whatever you want it to be.

(Doug) #5

So true, Brian. One group has a very low absolute risk, let’s say 0.0015% chance of dying from cause X. Another group has 0.00263% chance. OHMIGOD, the second group has a 75% higher chance of dying, there!!

(Carpe salata!) #6

My psych course made a big deal about surveys and the expected errors. The biggie is called ‘demand characteristic’ and it’s as you said, the survey-taker unconsciously trying to look good.

#7

Very interesting/helpful, @Peter. Now I can put a title with the concept. Good stuff!

(Ben) #8

I’d like to submit this as possibly the best example of a weak association being used for a silly headline. Particularly appropriate as it involves a relative risk of just 21%, and because it’s a data dredge of that source of much of this nonsense, the Harvard Nurses Health Study.

(Jim Russell) #9

In my opinion, observational studies are not science. Or at least not the science that they think they are studying. They are more along the lines of psychology and sociology than biology or nutrition.

What observational studies are is a starting point for the real science. Eating X or doing Y is associated with outcome Z. The first thing to ask is “is this plausible?” i.e. does it make sense from an evolutionary biological viewpoint that the behavior could cause the effect? For example, being a basketball player is positively associated with height. So does playing basketball cause people to be tall? If you can come up with a biological reason for that then the real science would be to take a group of people, assign them to basketball playing and non-basketball playing groups and see if one group gets taller than the other. I’m going to guess that the answer will be no, playing basketball does not make you taller and that the association is backwards. Taller people tend to play more basketball. But i don’t know which is true merely from the observation that basketball players tend to be tall. Or if there is any causal relationship at all.

Great summary @richard. Confirmation bias is a problem for everyone.