Lies, D*mned Lies, and Statistics: How to Understand Research

“There are three kinds of lies: lies, d*mned lies, and statistics.”  — Mark Twain

It can certainly seem that way, can’t it?  Most people are not given the opportunity to take a rigorous research methods class over the course of their educational careers, and even our high schools rarely teach students the skills they need to be informed consumers of the research and statistics that surround us every day.  When confronted with the sea of information on hearing loss, communication modes, and amplification options, what’s a parent of a child with hearing loss to do?

I believe that it is vital for all adults to have at least a small working knowledge of how to interpret research and statistics.  If not, you are going to be taken for a ride!  Here are some key points to keep in mind:

• Consider the legitimacy of the source.  Did this study come from a peer-reviewed journal?  Peer-reviewed journals require an impartial editorial board to review articles before publication.  The review board is composed of professionals who are highly qualified in the field, and their job is to inspect articles for any hint of bias or faulty research methods.  An article published on a mainstream news source (say CNN or MSNBC) does not undergo the same testing.  Often, mainstream news outlets will report on the latest scientific studies from peer-reviewed journals, but they usually provide just a summary that rarely gets all the facts straight.

• Consider possible bias.  I wouldn’t go to a university that promotes ASL for research on cochlear implants, just as I wouldn’t ask a Pepsi manufacturer to give me an honest review of Coca-Cola.  I wouldn’t believe a study sponsored by CI Brand X that (surprise, surprise) shows that their CIs clearly beat out those from Brands Y and Z.

• Consider the sample size.  Sample size means the number of participants in the study.  If a study of one class of ten children found that the children who wore blue scored better on speech and language measures than children who wore other colors, is that really a significant finding?  In general, the more participants in the research, the stronger the findings will be, and the more generalizable they will be to the population at large.

• Consider the age of the study.  A study on cochlear implants from the 1990s was a study done with devices that are dinosaurs compared to today’s technology.  Likewise, a study done on listening and spoken language outcomes for children in the days before high-powered digital hearing aids and cochlear implants is like comparing apples to oranges when we think about today’s listening, talking deaf kids.  Look for current research using the latest technology.

• Consider the subjects.  Who participated in this study?  What were the characteristics of the people who participated?  A study that measures the spoken language outcomes of children implanted at four years old or older in manual or total communication classrooms and concludes that CIs are not effective based on those children’s outcomes does not necessarily hold true for a child implanted at 12 months whose family has chosen an auditory verbal program.  If you’re looking for information to help you understand your child’s case, look for studies that profile children of similar age and implantation/intervention history.

• Consider the measures used.  How are the researchers measuring what they say they’re measuring?  Do the tests they used actually relate the skills being investigated?  For example, if the researchers want to measure children’s conversational competence but only give a single-word vocabulary test, does this really tell us so much about how children will fare in real life?

• Proof.  Studies do not conclusively “prove” anything.  They either support or do not support the hypothesis.  People who claim that one study they’ve found in support of their viewpoint “proves” it is right do not understand research methods.

And now for my two favorite topics in the world of research, two things I wish I could help EVERYONE understand:

Correlation is NOT causation.  Correlation means that two things happen together.  When A goes up, B goes up OR when A goes down, B goes down OR A and B go up and down together.  This does not, however, mean that A causes B.  For example, there was a HUGE outcry among parents of children with cochlear implants when this article was released in the mainstream media, “Some Kids with Cochlear Implants Face Cognitive Risks.”  Parents across the board were freaking out… all due to a serious misunderstanding of correlation vs. causation.  The research described in the article found a correlation between children with cochlear implants and increased prevalence of difficulties with executive function skills.  When A goes up (number of children with cochlear implants), B goes up (number of children in that group who also struggle with executive function).  That’s a correlation.  That does NOT, however, mean that the cochlear implants CAUSED those problems with executive function skills.  People who were upset by the article incorrectly attributed the causation of these learning deficits to the cochlear implants, and were outraged that the article would imply that CIs “gave their children mental problems.”

Anecdotes are NOT evidence.  Another comment I often see when research articles are posted is, “Well, this article says XYZ but my child doesn’t fit that pattern so I KNOW it must be wrong,” or “ABC happened to my child so I KNOW it is true.”  Those are anecdotes.  They’re lovely stories, and I believe that they happened for your child,in your family.  But in terms of their weight in the world of scientific research… they’re basically worthless.  Remember that if a study says, hypothetically, that children who were enrolled in AV programs at age five were more likely to get a perfect score on their SATs in high school than children who use other communication modes, and your child who signs got a perfect score, this does not invalidate the study.  You are sharing your one experience with your one child.  Just because it happened for you doesn’t outweigh a peer-reviewed, scientific study with multiple participants where the results were found to be statistically significant (that is, they occur at a rate too high to just be attributed to chance).

In addition, just because the results of a study do not fit with your personal experience does not mean they are incorrect or not true.  Let’s say a study showed that 90% of cochlear implant users enjoy music, and you’re a CI user who can’t stand it.  It doesn’t mean the study was wrong.  Maybe it just means you fall in that 10% group.  An area where I often see this is in CI simulations.  People become outraged when this simulation doesn’t match what they hear, or what their child reports hearing.  “I don’t hear or speak like this, so it is garbage!  Don’t share this with parents, it’s not true!”  Actually, it is.  These simulations are just that — simulations.  They take the things we know about acoustics and hearing science and use them to produce a simulation that, when compared to normal hearing, is of similar quality (in terms of pitches, channels, timing, etc.) to that which a CI would provide.  A person listening to this simulation of a CI, through a CI is not going to have the same experience as a person listening to it through an unimpaired system.  That CI users enjoy hearing and can speak just like a person without hearing loss does not mean this simulation is wrong.  Instead, it is a testament to the power of our brains to adapt.  How incredible!  I also think it’s a harmful attitude to say that parents should be shielded from any research or information that is anything less than 100% positive about hearing loss.  Parents are smart, they deserve full access to information, and, ultimately, I believe that cochlear implants and auditory verbal therapy are great, are evidence-based, and are more than strong enough to stand up to the scrutiny.

There’s a jungle of information out there for parents to navigate.  They want to do the research and find out what’s best for their children, and rightly so!  I’ve never been one to believe that parents “can’t handle” the research or should only be fed information in tiny, summarized chunks by “the professionals.”  If you’re interested and you want more information, go out there and find it!  Just make sure you’re not looking at it backward when you do.