Can you spot the problem with these headlines? (Level 1) - Jeff Leek & Lucy McGowan
"New drug may cure cancer."
"Aspirin may reduce risk of
heart attacks."
"Eating breakfast can help
you lose weight."
Health headlines like these
flood the news,
often contradicting each other.
So how can you figure out what’s a genuine health concern or a truly promising remedy, and what’s less conclusive? In medicine, there’s often a disconnect between news headlines and the scientific research they cover. That’s because a headline is designed to catch attention— it’s most effective when it makes a big claim. By contrast, many scientific studies produce meaningful results when they focus on a narrow, specific question. The best way to bridge this gap is to look at the original research behind a headline. We’ve come up with a simplified research scenario for each of these three headlines to test your skills. Keep watching for the explanation of the first study; then pause at the headline to figure out the flaw. Assume all the information you need to spot the flaw is included. Let’s start with this hypothetical scenario: a study using mice to test a new cancer drug. The study includes two groups of mice, one treated with the drug, the other with a placebo. At the end of the trial, the mice that receive the drug are cured, while those that received the placebo are not. Can you spot the problem with this headline: "Study shows new drug could cure cancer" Since the subjects of the study were mice, we can’t draw conclusions about human disease based on this research. In real life, early research on new drugs and therapies is not conducted on humans. If the early results are promising, clinical trials follow to determine if they hold up in humans. Now that you’ve warmed up, let’s try a trickier example: a study about the impact of aspirin on heart attack risk. The study randomly divides a pool of men into two groups. The members of one group take aspirin daily, while the others take a daily placebo. By the end of the trial, the control group suffered significantly more heart attacks than the group that took aspirin. Based on this situation, what’s wrong with the headline: "Aspirin may reduce risk of heart attacks" In this case, the study shows evidence that aspirin reduces heart attacks in men, because all the participants were men. But the conclusion “aspirin reduces risk of heart attacks” is too broad; we can’t assume that results found in men would also apply to women. Studies often limit participants based on geographic location, age, gender, or many other factors. Before these findings can be generalized, similar studies need to be run on other groups. If a headline makes a general claim, it should draw its evidence from a diverse body of research, not one study. Can you take your skills from the first two questions to the next level? Try this example about the impact of eating breakfast on weight loss. Researchers recruit a group of people who had always skipped breakfast and ask them to start eating breakfast everyday. The participants include men and women of a range of ages and backgrounds. Over a year-long period, participants lose an average of five pounds. So what’s wrong with the headline: "Eating breakfast can help you lose weight" The people in the study started eating breakfast and lost weight— but we don’t know that they lost weight because they started eating breakfast; perhaps having their weight tracked inspired them to change their eating habits in other ways. To rule out the possibility that some other factor caused weight loss, we would need to compare these participants to a group who didn’t eat breakfast before the study and continued to skip it during the study. A headline certainly shouldn’t claim the results of this research are generally applicable. And if the study itself made such a claim without a comparison group, then you should question its credibility. Now that you’ve battle-tested your skills on these hypothetical studies and headlines, you can test them on real-world news. Even when full papers aren’t available without a fee, you can often find summaries of experimental design and results in freely available abstracts, or even within the text of a news article. Individual studies have results that don’t necessarily correspond to a grabby headline. Big conclusions for human health issues require lots of evidence accumulated over time. But in the meantime, we can keep on top of the science, by reading past the headlines.
So how can you figure out what’s a genuine health concern or a truly promising remedy, and what’s less conclusive? In medicine, there’s often a disconnect between news headlines and the scientific research they cover. That’s because a headline is designed to catch attention— it’s most effective when it makes a big claim. By contrast, many scientific studies produce meaningful results when they focus on a narrow, specific question. The best way to bridge this gap is to look at the original research behind a headline. We’ve come up with a simplified research scenario for each of these three headlines to test your skills. Keep watching for the explanation of the first study; then pause at the headline to figure out the flaw. Assume all the information you need to spot the flaw is included. Let’s start with this hypothetical scenario: a study using mice to test a new cancer drug. The study includes two groups of mice, one treated with the drug, the other with a placebo. At the end of the trial, the mice that receive the drug are cured, while those that received the placebo are not. Can you spot the problem with this headline: "Study shows new drug could cure cancer" Since the subjects of the study were mice, we can’t draw conclusions about human disease based on this research. In real life, early research on new drugs and therapies is not conducted on humans. If the early results are promising, clinical trials follow to determine if they hold up in humans. Now that you’ve warmed up, let’s try a trickier example: a study about the impact of aspirin on heart attack risk. The study randomly divides a pool of men into two groups. The members of one group take aspirin daily, while the others take a daily placebo. By the end of the trial, the control group suffered significantly more heart attacks than the group that took aspirin. Based on this situation, what’s wrong with the headline: "Aspirin may reduce risk of heart attacks" In this case, the study shows evidence that aspirin reduces heart attacks in men, because all the participants were men. But the conclusion “aspirin reduces risk of heart attacks” is too broad; we can’t assume that results found in men would also apply to women. Studies often limit participants based on geographic location, age, gender, or many other factors. Before these findings can be generalized, similar studies need to be run on other groups. If a headline makes a general claim, it should draw its evidence from a diverse body of research, not one study. Can you take your skills from the first two questions to the next level? Try this example about the impact of eating breakfast on weight loss. Researchers recruit a group of people who had always skipped breakfast and ask them to start eating breakfast everyday. The participants include men and women of a range of ages and backgrounds. Over a year-long period, participants lose an average of five pounds. So what’s wrong with the headline: "Eating breakfast can help you lose weight" The people in the study started eating breakfast and lost weight— but we don’t know that they lost weight because they started eating breakfast; perhaps having their weight tracked inspired them to change their eating habits in other ways. To rule out the possibility that some other factor caused weight loss, we would need to compare these participants to a group who didn’t eat breakfast before the study and continued to skip it during the study. A headline certainly shouldn’t claim the results of this research are generally applicable. And if the study itself made such a claim without a comparison group, then you should question its credibility. Now that you’ve battle-tested your skills on these hypothetical studies and headlines, you can test them on real-world news. Even when full papers aren’t available without a fee, you can often find summaries of experimental design and results in freely available abstracts, or even within the text of a news article. Individual studies have results that don’t necessarily correspond to a grabby headline. Big conclusions for human health issues require lots of evidence accumulated over time. But in the meantime, we can keep on top of the science, by reading past the headlines.
