Science That Will Keep You Up Nights


The news on research quality could be better. According to The Natural Selection of Bad Science, a new paper by Paul E Smaldino and Richard McElreath, Department of Cognitive Science , University of California/Max Planck Institute:


Poor use of methods in scientific research persist, in part, because of incentives that favor it. The most prolific publishers are the most successful, but the quality of work suffers. The professors note that replication helps, but state that “change must happen at the institutional level if there is to be any improvement.” 


Another Possible Factor — Asleep at the Wheel? The Insomnia Study


Over-catergorization — of the sort that amounts to pre-judgement — is a frequent find if you’re looking for mind-sets that undermine the purpose of science.

But here’s a recent example of miss-categorizing altogether.

Thanks to the contribution of 23andMe customers, who’ve allowed the results of their personal genetic tests to be used for genetic studies, researchers had access to data from 1,331,010 people, to do what they are calling a genome — wide study of insomnia.


Quantity vs Quality


It most certainly is a huge study per the field of genetics — the biggest one to date, in fact. But despite the unprecedented amount of data — “genome-wide” is a bit of a stretch. And, although such testing has great potential, at this point only a tiny percentage of a person’s genetic information is revealed.

Genetic science has only just begun to examine each little bit of DNA, called SNP’s (single nucleotide polymorphisms). There are vast numbers of SNP’s to be fully evaluated. Much of the research is at a very tentative stage. There are decades worth of work to test and replicate it all yet to be done.

So, while I was initially pretty excited by the announcement that a study of this size has finally become reality, I was completely let down by the researchers’ opening statement:


“Insomnia is the second-most prevalent mental disorder … “ Say, what?


Mental disorder? How on earth has sleeplessness come to be categorized as a mental disorder? Where is the evidence to back this claim? With so many factors to consider, and with the research to date so limited, there’s no reasonable basis for classifying insomnia as anything but a symptom. The researchers’ conclusion is entirely inappropriate.

The folks consenting to the use of their genomic data in this, and any other, study might want to sit up and take a look-see, here. I’ll definitely be paying much closer attention, not just to what the data is being used for, but also to the interpretations.

Critical analysis counts, and it’s obvious something has run amok. For science to be worth doing, it absolutely must be done properly. The assumption that insomnia is a major cause of mental disorder is just plain over the top.


Responsibility — It’s Time to Take it Seriously


If researchers want access to personal data, they need to be damned sure they’re using it strictly to rule. Jumping to conclusions rates as the baddest of bad science. Big numbers don’t count for much if the data isn’t evaluated accurately.

Genetic testing companies also have a deep responsibility — to protect their clients’ information. How their data is used in this investigation gets a big thumbs down!

And, while it’s heartening to see that reasons behind bad science are being looked at, more effort is needed to over-see and correct errors right now, no matter what the excuses for it turn out to be.


Contributors to this study are listed as: VU University, Karolinska Institutet, Utrecht University, Erasmus University Medical Center, 23andMe, Inc.,University of North Carolina; Pre-print of abstract link here.



Leave a Reply