Do research studies annoy anyone else as much as they annoy me?

I hardly ever mention research studies here on CoachLevi.com, and it’s highly unlikely I would ever alter my training or nutrition based on a single new study, no matter how promising it sounds.

Don’t get me wrong – I think it’s a great concept. Research studies play a vital role in the scientific community, and some of them are excellent. But, just like politicians, very few of them truly deliver on their promises. (And the way the mainstream media presents the findings is abysmal.)

In today’s article, I’m going to explain all the reasons why I don’t like research studies!

 

There is always a conflicting study.

We’ll start with the most obvious problem that everyone should be aware of already – there’s always a conflicting study! One day studies show something is bad for you, but the next study shows that same something is good for you!

Coffee must be the best example of this. Researchers are flip-flopping back and forth constantly, and the news outlets making it into big news stories each and every time. It looks something like this:

Monday: New research shows coffee is good for you!
Thursday: Nope, this new study says coffee is bad for you again! Too bad. You’ll have to wait another week until coffee is proven to be good for you!

Sodium is another good example. If you read through this analysis of some salt studies on SweatScience.com (one of my favorite science blogs), you’ll find that people are evaluating a bunch of different conflicting studies and citing whatever ones they happen to agree with.

Person A discredits Person B by referencing Study C, etc. It’s an endless cycle of useless bickering.

 

Studies are designed to produce the desired results.

The reason for so many conflicting studies, and the main reason I see these studies as useless, is that studies are typically designed to produce a desired result. Most studies, at least ones we see relating to endurance sports, are funded by a big corporation who wants the research team to prove that their product is superior to others.

Even outside of commercial products, studies are still designed as to ensure a particular outcome. This next example was either designed by someone with a bias towards aerobic exercise or someone with no knowledge of proper resistance training. (Thanks to Conditioning Research, another science blog I enjoy, for pointing this out.)

The study compared aerobic exercise to resistance training to see which was better at burning belly fat. They took overweight, sedentary individuals and split them into two groups; one group ran 12 miles per week while the other performed a resistance training routine involving eight different machines.

Just from that, I already know the outcome. Think about it. Running 12 miles per week is a serious endeavor for any sedentary person – for someone both overweight and sedentary, it’s a grueling training plan! On the other hand, a gym workout on machines that stabilize both your body and the weight you’re lifting sounds like a walk in the park.

If they wanted to be fair, the resistance training group would be doing squats and deadlifts (full-body, demanding exercises), or the aerobic group would be jogging 3 miles per week (not running 12!)

To make matters worse, bring in prescription drug companies…

The best article I’ve ever read regarding problems with medical research is this one in The Atlantic. It’s a long two pages, but it’s awesome. Check out the teaser and see if it doesn’t draw you in:

Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice? Dr. John Ioannidis has spent his career challenging his peers by exposing their bad science.

 

Nothing provides 100% results.

Here’s the funny part the news media doesn’t mention – the part about how successful the study actually was. Even when a study “proves” something, rarely if ever will it have had a 100% effectiveness rate in the group of subjects.

That means, even if something works for many people, it doesn’t mean it will work for you. You could very well be in the minority.

For example, if 68% of subjects in Group X (who ate PowderyBars before their workout) felt better than the subjects in Group Y (who ate Clifford Bars,) the headlines will read “PowderyBars Outperform Clifford Bars In Recent Study.”

They won’t mention the 32% of subjects who performed terribly after eating a PowderyBar. How do you know, then, that the PowderyBar is the choice for you?

You have to do an n=1 study! That’s why I test stuff myself, and although I present my results here, I still suggest you do your own n=1 studies to see what works best for you.

 

Test subjects are rarely experienced athletes.

Since you’re reading about research studies on a cycling training and racing site, I’ll assume you’re an experienced athlete. But even if not, you can categorize yourself into a certain group.

Well, rarely are the groups studied actually representative of racing cyclists like us!

Now, the point here isn’t specifically related to racing cyclists. What I want you to do is look at the subjects in any study you’re interested in and see if you can identify with them. If you’re a racing cyclist, be wary of studies testing overweight, sedentary individuals. If you’re a bulky powerlifter, be wary of studies conducted on skinny endurance athletes.

 

Too few test subjects.

If you do get lucky and find a study where the subjects are fit endurance athletes such as yourself, see how many people total are participating. The greater the fitness level, the fewer available subjects there are. (This is probably why many studies in the USA use overweight, sedentary individuals!)

For example, I was reading a study about how the nitrates in vegetables can increase your endurance by making your mitochondria more efficient. Cool. The test subjects were athletes. Even cooler. You know how many people they tested? 11.

That’s more than you’ll have in your n=1 study, yes, but it’s still not statistically significant.

 

The testing procedures are unrealistic.

Most studies are set up in a way that doesn’t make much sense to athletes. For example, workouts used in the lab don’t necessarily simulate actual workouts or race conditions.

Some studies reviewed by Joe Friel on his blog demonstrate a common theme amongst research studies – the fact that the subjects exercise till exhaustion, then quit. If that’s representative of your typical training plan, you better read up on better training methods!

I have an example for this one and it’s another one from SweatScience.

They had cyclists for this one (yay!), but the testing was almost too extreme. It was like a stage race in one day!

Basically, the study compared two nutrition plans: one backed by science chosen by the researchers, and one the athlete’s picked themselves. The cyclists actually picked out a good diet, but apparently they didn’t realize they needed to alter their plans since they were not just doing a 40k time trial, they were doing a 40k time trial the same day after cycling 2.5 hours earlier!

The only real takeaway was that cyclists can pick a good nutrition plan for a typical race, but can’t pick out a perfect plan for a day in the lab.

 

The results are way too vague.

Though the news media loves to generalize results and put their own spin on things, the actual results are usually too vague to apply to your own training. That, or they purposefully leave out the important details!

So we head back to SweatScience again, for research about ice baths and fatigue.

Some of these studies show that ice baths provide no recovery help. But many endurance athletes swear by post-workout ice baths. What’s the deal?

They left out the details! It’s all about the type of muscle soreness you’re inducing. Researchers doing a meta-analysis of 14 different studies found that ice baths work best only in certain circumstances. Ice baths will show recovery from high-intensity exercise like running and cycling interval workouts, but not show much of anything when you’re talking “eccentric muscle contractions” (such as lowering a dumbbell over and over) used in the lab.

Here’s another problem I saw recently when Joe Friel reviewed carb/protein drink studies.

A lot of those studies are flawed. For example, the carbohydrate drinks will be one type of sugar, while the carb/protein drinks will offer multiple sugars (which can offer better absorption). Also, sometimes the carb/protein drinks also offer more calories.

Hello!! Different calorie amounts is a very, very significant difference! You might as well compare the difference between fueling with 8oz Gatorade (50 calories of sugar) or a PowerBar (240 calories, carbs, fat, and protein). (Which one lets you ride longer? Hmm…)

The study that really stuck out showed a 35% increase in recovery by using carbs + protein. But, the group taking in carbs + protein was also taking in 40% more calories! If they had simply split subjects into three groups (the third group would have been all carbs, but increased calories to match the carbs + protein group), results would have been much more worthwhile.

And here’s a good one for you to read that really illustrates just how many details get thrown out the window:

That’s written by Denise Minger, who also ripped apart The China Study. She’s pretty good at this!

 

They attempt to apply one study’s results to everyone.

Last but not least, people take the finding from one study and blindly apply it to a completely different group of people or scenario.

A recent study where this could happen involved testing a typical warm-up vs a short warm-up and found that the much shorter warm up resulted in a higher peak power output. So the conclusion was, shorter warm ups are better for cyclists.

This means people will tell you to warm up less (no matter what workout you have planned). But hold on a minute.

Look at the details of this study. It compared a 15-minute light warmup including one sprint to a 50-minute warmup with multiple sprints, up to 95% of Max HR (designed in consultation with elite track cyclists and coaches). What was the power test? A 30-second sprint, basically.

So, this study will apply to track sprinters and certainly gives them something to think about. But, it has virtually no meaning when it comes to warming up for a road or mountain bike race!!

 

To recap, here are the reasons I don’t put much faith in research studies:

  • There is always a conflicting study.
  • Studies are designed to produce the desired results.
  • Nothing provides 100% results.
  • Test subjects are rarely experienced athletes.
  • Too few test subjects.
  • The testing procedures are unrealistic.
  • The results are way too vague (and important details are left out.)
  • They attempt to apply one study’s results to everyone.

There are good research studies out there, but they’re hard to find.

So in the end, what’s worth your time – spending all day evaluating research studies, or getting a workout in?! I say, focus on your workouts and don’t get caught up reading every new study that comes out!

6 Comments
  1. Here is one bias that you missed — White Hat Bias — which especially affects “researchers” in the public health field (and they aren’t usually corporate-funded).

    White Hat Bias: http://en.wikipedia.org/wiki/White_hat_bias

  2. @Jon

    Good addition – I like their example with data to prove it.

  3. I think this post confuses research studies with media coverage of research studies.

  4. @JRR

    Thanks for stopping in to comment. I certainly included both research studies and the resulting media coverage here, but that’s not because I’m confusing the two, it’s because they’re both problematic.

    If the media influences the public perception of a study, that still falls under my “problem with research studies” umbrella.

    Anyway, out of the 8 problems I listed, only 2 of them are related to media coverage. The media has nothing to do with the poor study design or the researchers’ motives/bias when conducting the study and their conclusions, which is the root of the problem.

  5. @Levi

    You are correct. That is what I get for skimming the second half of the post. Props for trying to inform people on how to critically read and interpret research studies.

  6. @JRR

    Well, it was a long post! 🙂

Leave a Reply