How informative are cognitive tests (Part 1 of 3)? Why you shouldn’t trust your results

You probably think I just went a little nuts because making web-based cognitive tests that get at brain functioning is What I Do.  It’s what I am building my career on, and if I’m saying the results aren’t trustworthy, I should throw in the towel and start selling UFO detectors.  I’m looking at you, Curtis Mead.

So I guess I should qualify the title of this blog post.  I think taking tests like the ones offered on TestMyBrain.org is a worthwhile activity and can teach you a lot about yourself… but your results should always be taken with a grain of salt.

Here’s an example.  For the last few years, we have had one brain test on TestMyBrain.org called “How good is your number gut?

gut number sense

Image borrowed from Panamath.org

This is a test developed in collaboration with Justin Halberda, a professor at Johns Hopkins University.  You can learn more about your gut number sense and take the test at this website.

We found that a person’s “number sense” predicts how good they are at math.  This test has gotten some publicity, with lots of people coming to TestMyBrain to try and find out whether they are a math genius.  I typically get two types of emails from people who do these tests:

(1) I am good at math, but I didn’t do well on the gut number sense tests.  Does this mean I should quit my accounting job?

(2) I am not good at math, but I did really well on the gut number sense test.  Does this mean I should become an accountant?

HOLD UP PEOPLE.  Scientists often say things like, “Based on our research, A predicts B.”  For example, a recent longitudinal study in children found that drinking milk predicts weight gain.  However:

(1) If you recently started drinking more milk and haven’t gained weight, it’s (probably) NOT because your local supermarket has switched out “milk” for “milk-like substance” and hasn’t bothered changing the labels.

(2) If you recently reduced your milk consumption and gained some weight, it’s (probably) NOT because your spouse/friend/cat has been slipping a bit of milk into your evening glass of wine without your noticing.

One of the problems is that increased milk consumption is only one of many, many factors impacting weight gain.   Similarly, one’s gut number sense is only one of many (many) factors that might impact variations in math ability.

math ability and gut number sense

How much differences in math ability are predicted by gut number sense

For a test that doesn’t actually make you use numbers, explaining this much variation in math ability is actually pretty good!  Still, it’s quite clear that gut number sense doesn’t tell the whole story and it’s conceivable that you could be very good at math, with a relatively poor gut number sense.

When we zoom out and look at YOU as a whole, the amount of YOU-ness that is predicted by even a dozen of these cognitive tests is vanishingly small:

you and cognitive tests

How much cognitive tests tell you about yourself

If you are scientifically-minded, you are probably shaking your head vigorously and thinking “Seriously?  What the @#$! is ‘you-ness’ and how can you make any claims about how to measure anyone’s ‘essential you-ness’?”  Bear with me, it’s meant to be an ILLUSTRATION.

In parts two and three of this post, I will tell you all how taking tests to assess your brain and cognitive uniqueness can be useful despite these limitation.  I will start in part 2 of 3 by telling you how taking cognitive tests saved my marriage.

Sort of.

This entry was posted in Background/Introduction, Why take cognitive tests and tagged , , , , . Bookmark the permalink.

One Response to How informative are cognitive tests (Part 1 of 3)? Why you shouldn’t trust your results

  1. Margaret says:

    love

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>