PDA

View Full Version : Why Reporting On Scientific Research May Warp Findings



Extravoice
2014-May-21, 02:52 PM
I'm not quite sure this is the right forum for this; the mods are encouraged to move it if they feel fit to do so.

This story on NPR's Morning Edition
http://www.npr.org/2014/05/19/313844261/why-reporting-on-scientific-research-may-warp-findings

Brings up an unsettling issue with the way science is conducted today. The gist is that there is an enormous amount of pressure for researchers to break new ground, and very little incentive for them to do something that is fundamental to the scientific method...repeat previous experiments for verification.


...this model creates perverse incentives, because if a study confirms an older result, the journals tend to say, well, we knew that already, it's not a novel finding and they're less inclined to publish it. Now, if the replication contradicts an earlier finding, where the journal sends out the study to the peer reviewers, some of the peer reviewers might have been the researchers who conducted the original study and they can now find ways to shoot down the study and reject the study and say we shouldn't be publishing it anyway.

And so what this does is it creates a disincentive for researchers to conduct replications at all.

A question to those members CQ who are in the "science business"; is this as big a problem as the article claims?
Could we be building a scientific foundation that is weaker than it should be?

John Mendenhall
2014-May-21, 03:24 PM
Surely you're jesting. Just think about the squabbles that go on here. Skepticiism and science are opposite sides of the same coin. Consider this post, for example. That idea above appears to me to be speculation without research.

Extravoice
2014-May-21, 08:25 PM
Vulcans never jest...or is that bluff?

At any rate, they do cite one example where an experiment from 1999 was repeated and the results were different.
I'll concede that it was a social psychology experiment, which may be tricky to replicate.

John Mendenhall
2014-May-21, 10:38 PM
Off topic. That would make a rollicking good Star Trek episode if Spock told a whopper of a lie.

Jens
2014-May-22, 03:22 AM
A question to those members CQ who are in the "science business"; is this as big a problem as the article claims?


I don't know if it is as big, but I would agree that it is a big problem. Scientists tend to work under outcome-based funding projects and have to show original research. There is a lot of pressure to make new discoveries, and not so much to take time to replicate other people's findings.

Swift
2014-May-22, 01:13 PM
I don't know that this is anything new. I recall some work I did almost 20 years ago, trying to reproduce something that had been published many years before that (not for the sake of just reproducing it, I needed to do the same synthesis) and had difficulty repeating it. I eventually worked it out and attributed it to differences in how I was measuring temperatures in the pressure vessel, versus how they did.

I think this is probably the most common method of reproducing earlier work - not trying to reproduce for the sake of reproducing it, but doing it because you need the earlier work to build on something you are doing. At that time you find out it is exactly correct... or it isn't, and you publish the corrections, either minor or major, when you publish your work.

I haven't made a study of this, I feel it is not a huge problem, but it is a concern, it always has been a concern, and I'm not convinced anything has really changed on this.

Cougar
2014-May-22, 04:37 PM
VEDANTAM: Well, the problem is that the incentives in science today are all about breaking new ground, and there's much less incentive to making sure that the ground you're standing on is actually solid. Most people think about science as being about breakthroughs, but science really is about the gradual accumulation of knowledge and retesting what we think we know.

I imagine one is apt to get quite a variety of responses when asking "What is science really about?"


"More often than not, the way science goes from point A to point B is by a random lurch through points X, Y, and Z. Even when great leaps of progress do occur, they only rarely come "out of the blue." Advances are nearly always preceded by years, decades, or even centuries of patient accumulation of facts and data and ideas." [Rocky Kolb, Blind Watchers of the Sky]

"Scientific problems have a natural life cycle. They are born with an unexpected discovery, such as the observation of gamma-ray bursts by the Vela satellites. This immediately calls for explanation. The information available is preliminary and incomplete, so new measurements must be made. Theoretical models are floated and tested against our understanding of the laws of nature and against the data. Sometimes a satisfactory explanation is quickly developed. Sometimes closer examination shows that there is really nothing to explain, for the original data were spurious or misinterpreted. It is more interesting if the discovery is genuine, and remains enigmatic, as was the case for gamma-ray bursts. Then the next stage consists of the development of new instruments to answer the experimental questions, and of new theoretical ideas to explain their results... For gamma-ray bursts, the second stage lasted 25 years from their discovery. -- Katz

The excitement of discovery is not replaced by routine science, as some philosophers and historians of science would have us believe [Thomas Kuhn, take note], but by the silence of the grave, as scientists move on to something else, new and more interesting." - Katz

"...science is not merely a collection of facts, but is instead an ongoing detective story, in which scientists passionately search for clues in the hope of unraveling the mysteries of the universe." [Alan Guth, The Inflationary Universe]

"What, then, is the meaning of this work of science? Briefly put, it consists in the task of introducing order and regularity into the wealth of heterogeneous experiences conveyed by the various fields of the sense world... Scientific reasoning does not differ from ordinary everyday thinking in kind, but merely in degree of refinement and accuracy... [E]ven scientific logic cannot deduce anything else from given presuppositions than can the ordinary logic of untrained common sense." - Planck

The essential point in science is not a complicated mathematical formalism or a ritualized experimentation. Rather the heart of science is a kind of shrewd honesty that springs from really wanting to know what the hell is going on!" - Herbert

Grey
2014-May-22, 05:15 PM
I heard the same story. One thing I noted was that all the examples they provided were from social sciences, like psychology, rather than "hard" sciences like physics and chemistry. When reading Feynman's memoirs, I recall that was a specific complaint he raised about experiments in psychology and sociology, that nobody felt a need to try to replicate those experiments. So I wonder whether this is more of an issue in some fields than in others.

I know that in physics, at least, there's usually a great deal of replication, particularly when there's a surprising result. For example, when the OPERA experiment (http://en.wikipedia.org/wiki/OPERA_experiment) showed a result that seemed to indicate that neutrinos were travelling faster than light, both the original researchers and others immediately set about seeing if they could confirm those results using different equipment. Or there's the Eöt-Wash Group (http://www.npl.washington.edu/eotwash/) that spends its time making more and more careful measurements of gravity, using variations on this device (http://en.wikipedia.org/wiki/Torsion_spring#Torsion_balance) but with more sensitive equipment. They know that they probably won't find any deviations from expectations, but they're still looking. It's true that there's some pressure to find publishable results, but I think that in physics at least, just measuring something to one more decimal point is considered an accomplishment well worthy of publication.

LaurieAG
2014-May-27, 10:44 AM
Here's a bit of "hard" chemistry from 2012.

http://www.washingtonpost.com/business/economy/as-drug-industrys-influence-over-research-grows-so-does-the-potential-for-bias/2012/11/24/bb64d596-1264-11e2-be82-c3411b7680a9_story.html?wpisrc=al_excl

Grey
2014-May-27, 07:53 PM
Here's a bit of "hard" chemistry from 2012.Actually, I'd disagree. That's medicine, not just chemistry. I'd think that the effects of drugs on humans, where there can be so many things that are hard to control, is a field where you'd especially want there to be numerous repeated tests and lots of cross-checking. As the article you post notes, the fact that there can be some pretty strong financial incentives to gloss over some of that testing is a serious concern.

publiusr
2014-May-31, 06:20 PM
NPR did some other related stories of merit
http://www.npr.org/blogs/health/2014/04/08/298335701/how-mouse-studies-lead-medical-research-down-dead-ends

Especially these two
http://www.onthemedia.org/story/worried-well-whipped-frenzy/
http://www.onthemedia.org/story/whats-health-journalist-do/?utm_source=/story/worried-well-whipped-frenzy/&utm_medium=treatment&utm_campaign=morelikethis

I was surprised by this quote from Virginia Hughes:

“[Insert] debate and nuance and doubt. I hate stories that treat science as some holy seer that has the answers to everything, because it’s absolutely not, and being wrong is sort of built into the process of science.”

I often wonder if resistance to evolution AGW and the like comes not from folks like Inhofe coming from flat land, as Bill Nye thought, but all these blasted medical stories that seem different every week. The quote above might not help matters much either, though...

ShinAce
2014-Jun-02, 02:34 AM
I don't know that this is anything new. I recall some work I did almost 20 years ago, trying to reproduce something that had been published many years before that (not for the sake of just reproducing it, I needed to do the same synthesis) and had difficulty repeating it. I eventually worked it out and attributed it to differences in how I was measuring temperatures in the pressure vessel, versus how they did.

I think this is probably the most common method of reproducing earlier work - not trying to reproduce for the sake of reproducing it, but doing it because you need the earlier work to build on something you are doing. At that time you find out it is exactly correct... or it isn't, and you publish the corrections, either minor or major, when you publish your work.

I haven't made a study of this, I feel it is not a huge problem, but it is a concern, it always has been a concern, and I'm not convinced anything has really changed on this.

I completely agree. The reason I agree is because of the research project I'm working on. I can't say it's hardcore, because I'm still an undergrad.

I need to rebuild a confocal microscope that's been cannibalized before moving on to the features we want. In the process, I am forced to re-invent the confocal microscope. Sure, it's almost 50 years old and in common use today, but that's the way science works. Someone does it first and then others replicate it. They don't replicate it in the name of 'repeatability'. They replicate it because there's something else they want to do with it.