| A Brief Guide to Statistical Manipulation
ALLSCI MONTHLY SCIENCE MAGAZINE
URL: http://www.allsci.com/stats.html
by Sam Sachdev May, 2004
Try and think of a statistic you’ve recently seen, heard, or read. Maybe
it was from a commercial. Three out of four dentists recommend Dentine.
Or, maybe it was from the news. The President’s approval rating has
fallen five percent. Or, maybe it was from a health article from a
newspaper. Frequent use of antibiotics greatly increase your chance of
cancer. The power of statistics, perhaps, is because of their quick and
efficient means to present a conclusion persuasively. Their effectiveness,
however, can be used to mislead, intentionally or otherwise. They can be
used, for instance, to distort facts or only present a limited amount of
data that helps to support a conclusion. Or, it could be unintentional
manipulation. If you however have a basic understanding of how they’re
misused or how they’re calculated, you can alert yourself to the most
common and basic statistical manipulation.
The first, and probably most important, example of how statistics are
used improperly is the difference between what, usually, the media
considers worthy and what statisticians, psychologists and others who
professionally use statistics do. The President’s approval rating, for
instance, is a widely reported statistic. This statistic, however, is only
likely to be reported if it suddenly rises or falls from its expected
pattern. Tom Smith, a statistician at the University of Chicago, explains
why this is particularly unreliable. “The media considers [a statistic]
newsworthy because it’s different from what the most recent figures
were. But numbers that are most unusual are likely to be the most error
prone. It’s a systematic problem. The media are attracted to results
that are the least reliable,” said Smith.
A poll, study, or any other statistic, then, is only considered
reliable when its results have been replicated. There are many possible
explanations. A poll, for instance, could be influenced by a national
holiday, which could cause the President’s approval to suddenly, and
without apparent reason, rise. Or, there was a subtle bias in the
questions. And, one of the most important reasons is chance. Regardless of
how well designed a poll or study is designed, it’s possible that chance
could be the cause of the results. The causation of smoking to cancer, for
instance, was only confirmed after many studies repeated the results. If
you hear or read about a poll, study, or any other statistic whose results
haven’t been repeated, it’s unlikely that researchers are going to
take the results seriously.
Another common mistake, when the media reports statistics, is confusing
a correlation for causation. Cooper Holmes, in “The Honest Truth About
Lying with Statistics”, presents a common sense example to help explain
the difference. There is, in the Spring and Summer, a high correlation
between newly planted trees and the rate of the growth of the grass around
them. The trees, of course, aren’t causing the grass to grow at a fast
rate. The cause, obviously, is water and sunshine, not the trees. Or,
consider this starker example. A friend of this journalist found that
there was a high correlation between the number of car accidents in
Florida and the amount of rain in Japan. There, however, isn’t causation
between the two.
A correlation, then, is only the statistical relationship between two
events. It doesn’t mean that one event is causing the other. Joel Best,
a sociology professor at the University of Delaware and author of
“Damned Lies and Statistics”, points out that confusing correlation
and causation is often reported whenever the weekly medical journals come
out. As an hypothetical example, he notes an example between eating
Brussel Sprouts and reducing your chance of cancer. “[A] news story is
always written as the Brussel Sprouts are going to help you reduce the
chance of cancer,” said Best. In order for this to be causation, that
the Brussel Sprouts actually helps to prevent cancer, it has to be explain
why this is so. So, the next time you hear or read about research that
states that there’s a correlation between two events, remember that this
doesn’t mean that there’s a cause and effect relationship.
Statistical significance, the next example, is also a persuasive
problem when the media presents statistics. Statistical significance,
however, probably because it’s difficult to explain to those who
aren’t familiar with statistics, is almost entirely absent from the
presentation of statistics. In the evening news, for instance, you hear
that President Bush’s approval rating has fallen five percentage points.
The presenter of the information, almost certainly, isn’t likely to tell
you that the results aren’t statistically significant. That is, it
wasn’t explained that the five percent difference is probably only
because of chance. Or, President Bush’s drop in the polls is probably
meaningless.
Statistical significance means that there’s a relatively small chance
that the results are because of chance only. Or, to state it more clearly,
it means that if, say, the poll were repeated again there’s a large
probability that the results could be repeated. Tom Smith, referring to a
presidential approval rating poll, points out that without this
information, the poll may be meaningless. “The presenter of the poll
didn’t present that [it was] marginally...significant. The language
should’ve been more circumspect...[This poll] shows a gain of five
percentage points but it’s not statistically significance.” Or, to
state it another way, imagine if the anchor of your nightly news told you
that President Bush’s approval rating dropped, but it’s mostly likely
only because of chance only. The poll, therefore, should be ignored.
Unfortunately, because there isn’t any mathematical calculation to
determine what’s statistically significant and what’s not, it’s
definition is hard to quantify. [John's
Note] Nonetheless, it is associated with correlations. If
there’s a strong correlation, there’s likely to be statistical
significance. This, however, doesn’t assure that it is. There still has
to be the likelihood that the results weren’t because of chance, a cause
and effect relationship was proven, or the results were repeated. What’s
important to keep in mind is that chance is an important concern when
evaluating results. The next time you hear a statistic, say, the
President’s approval rating, and the presenter doesn’t tell you if
it’s statistically significant, keep in mind that the results of the
poll could only be because of chance only.
The next problem in polls, agreeing upon definitions, is also a common
problem outside of statistics, in marriage, politics, work places, or
probably anything that requires communication. Cooper Holmes points out
that when one place where he’s noticed this problem is in advertisements
for drug/alcohol treatment programs. Most treatment programs, Holmes
notes, advertize their success rates, such as 90% are drug-free after
completing the program. The definition of “drug-free” could be simply
completing the program or not using drugs or alcohol for one month. Most
likely, the definition is one that favors the treatment program. It’s
not one, however, that asks the question if the time spent sober is long
enough to warrant a program that’s effective.
In statistical studies, definitions can lead to misunderstanding about
the reported results. In a survey that reported on violence in schools,
Tom Smith points out, there was a large difference between how most
normally understood how the term “violence” is used and how the survey
defined it. “[The survey] said that a majority of students had
experienced violence. This was the number of students who had said yes one
of a number of events. Among them were things like bumping into the
hallway, shoving them or verbal abuse,” said Smith. Most, Smith notes,
wouldn’t consider these events as violence. Rather, they would think
that violence is, say, when a student is punched, kicked, or otherwise
attacked. The result of the study was, without understanding how
“violence” was defined, likely to give the impression that violence in
schools was a serious and largely unreported problem. So, the next time
you hear that poverty has gone up, or unemployment down, you should
understand that your definition might be quite different from how it’s
actually defined.
In this last example, researchers and the media selectively choose a
conclusion from the data. This, infamously, is described by an example in
“The Honest Truth About Lying with Statistics”. A psychologist wanted
to see how accurately patients who are admitted into a mental hospital are
evaluated. He had students, who were not mentally ill, act as if they
were, enough so, anyway, to get admitted. Once admitted, however, they
were told to behave as they normally do. That is, behavior that would be
considered “normal”. The results were shocking and are described in
the famous 1973 study “On Being Sane in Insane Places”. Many of the
students couldn’t convince psychologists that they were “normal”.
The psychologists, in this example, selectively chose from data.
Although, according to Cooper Holmes, the failure to correctly diagnosis
the students who were acting as patients is somewhat controversial,
nonetheless the practice of selectively choosing from data is so common,
he only trusts conclusions when he himself can examine the data.
In a more common example, Tom Smith points out, a poll, twenty years
ago, commissioned by an anti-gun control lobbying group. The group wanted
to present evidence that the public wasn’t in favor of gun control
legislation. “There was maybe twenty questions on gun control. They
basically found one question that had gun control marginals and reported
that,” said Smith. Unless the data is examined, there probably isn’t
any way to determine if what the media is presenting you is what the data
says it is. Nonetheless, the wariness of this seemingly common deception
should make one aware of how a conclusion can be true and not represent
the data.
The next time, then, you see an ad, or hear about an approval rating,
or any statistic in the media, you should understand that there’s
probably much more about the statistic than either the statistic can tell
or the presenter of the statistic wants, or knows, to reveal.
|
