PDA

View Full Version : Science is in a reproducibility crisis



A.DIM
2013-Sep-30, 01:57 PM
Science is in a reproducibility crisis: How do we resolve it? (http://phys.org/news/2013-09-science-crisis.html)


Over the past few years, there has been a growing awareness that many experimentally established "facts" don't seem to hold up to repeated investigation.
...
Articles in recent days have discussed how the majority of scientists might be more interested in funding and fame than "truth" and are becoming increasingly reluctant to share unpublished details of their work.
...
There can be little doubt that the "publish or perish" research environment fuels this fire. Funding bodies and academic journals that value "novelty" over replication deserve blame too.


So how do we fix Science?

R.A.F.
2013-Sep-30, 02:32 PM
So how do we fix Science?

Since you started this thread, why don't you tell us how you would "fix" science.

neilzero
2013-Sep-30, 04:46 PM
This may not be what A.Dim had in mind, but science has a rather knee jerk reaction of debunking or attacking a long list if topics, such as ET = extra terrestrial sightings, ghosts, and other paranormal. We should say "probably not", instead of "no way" and explain why it is improbable.
How about a journal with the title "Reasonable Doubt"

NEOWatcher
2013-Sep-30, 04:53 PM
I don't see anything in that article that says it's broken.
That's the way it works. Publish a piece and hope someone can confirm it.

The fact that an increasing number of works are being withdrawn could just be a side-effect of the increasingly expensive and technical challenges that are being used.

I'd like to see a graph that shows how this increase compares to the rate in decades past.

R.A.F.
2013-Sep-30, 06:32 PM
I'd like to see a graph that shows how this increase compares to the rate in decades past.

Well, that would be good for a start.

Hlafordlaes
2013-Sep-30, 06:44 PM
I'd bet top dollar that if a recent trend did exist and could be charted, it would correlate with economic pressure on universities, leading to more threats of publish, and publish famously to bring in the donations, or perish.

Coincides with a lot of sloppy work in publishing across the board, imo. Whole shebang is in crisis.

R.A.F.
2013-Sep-30, 08:50 PM
....science has a rather knee jerk reaction of debunking or attacking a long list if topics, such as ET = extra terrestrial sightings, ghosts, and other paranormal.

Perhaps that "knee jerk" reaction is related to/because of all the anti-science that has come from the promoters of these subjects.

Solfe
2013-Sep-30, 08:57 PM
I hate to say it, but it seems to me that the article(s) are point to issues and problems within the soft sciences or money sciences (medicine) and trying to extend that to other all other sciences. When there is a cash or fame on the line, people do really crummy things, but when neutrinos go to fast or universal expansion is accelerating, we don't see these sorts of goof ball attempt to fleece people.

Did anyone else get that impression?

(I am totally ignoring woo on the web, it canít even be phrased in terms of science in most cases.)

Squink
2013-Sep-30, 11:37 PM
Looks good on paper (http://www.economist.com/news/china/21586845-flawed-system-judging-research-leading-academic-fraud-looks-good-paper)


On retractions due to fraud, China ranked fourth, behind America, Germany and Japan.

R.A.F.
2013-Oct-01, 12:40 AM
Looks good on paper (http://www.economist.com/news/china/21586845-flawed-system-judging-research-leading-academic-fraud-looks-good-paper)

For what purpose?...it's not like a really bad idea won't be found out...

Squink
2013-Oct-01, 09:42 PM
...it's not like a really bad idea won't be found out...How often, and at what cost? What about just kinda bad ideas? A decreased S/N ratio (http://en.wikipedia.org/wiki/Signal-to-noise_ratio) makes it harder for honest researchers to do useful work.

JustAFriend
2013-Oct-02, 03:05 AM
it's not like a really bad idea won't be found out...

It can sometimes take a long time and at a terrible cost.

It took nearly 50years to stop the ideas of Trofim Lysenko (http://en.wikipedia.org/wiki/Trofim_Lysenko) from being applied to genetics and agriculture in many Marxist countries; millions may have starved as a result.

Don Blazys
2013-Oct-03, 11:01 AM
Over the past few years, there has been a growing awareness that many experimentally established "facts" don't seem to hold up to repeated investigation.
...
Articles in recent days have discussed how the majority of scientists might be more interested in funding and fame than "truth" and are becoming increasingly reluctant to share unpublished details of their work.
...
There can be little doubt that the "publish or perish" research environment fuels this fire. Funding bodies and academic journals that value "novelty" over replication deserve blame too.


I tend to agree with the above. Today's scientific protocols do seem a bit shoddy. Take for instance this Wikipedia article on what is perhaps the most important number in all of science... the Fine-structure constant:

http://en.wikipedia.org/wiki/Fine-structure_constant

Has anybody even noticed that the CODATA value: \alpha^{-1}=137.035999074(44) and the "most precise" experimental value: \alpha^{-1}=137.035999173(35) don't even come close to being within each others margin of error? How embarrassing is that!?

Here is the paper cited by Wikipedia:

http://arxiv.org/pdf/1205.5368v2.pdf

Note that this more recent paper:

http://arxiv.org/pdf/1309.3393v1.pdf

gives the value: \alpha^{-1}=137.035999044(90) which is also well outside that margin of error cited by Wikipedia.

However, it is in good agreement (and shares an upper bound) with this 2008 paper:

http://gabrielse.physics.harvard.edu/gabrielse/papers/2008/HarvardElectronMagneticMoment2008.pdf

which gives the value: \alpha^{-1}=137.035999084(51) .

What a mess! What a waste of time, energy and money! Is it any wonder why so many "numerologists" are also putting in their "two cents worth"?

Maybe someday, some transcendent genius will demonstrate that the Fine-structure constant has a definite mathematical purpose. That would take it out of the realm of numerology and since mathematical investigations are typically a lot cheaper to conduct than are laboratory experiments, it would probably save a lot of money too.

Shaula
2013-Oct-03, 03:58 PM
1) Those are probably 1 sigma values, which makes your point invalid
2) It is not surprising that different ways to measure something come up with slightly different results if they are using slightly different assumptions
3) Nowhere in the paper cited by Wikipedia does that exact value appear anyway

It is not a mess, it is simply science. Headline numbers have to be taken in context, ignoring that context leads to this kind of 'proof' of how 'broken' everything is.

Ken G
2013-Oct-03, 07:40 PM
What a mess! What a waste of time, energy and money! Is it any wonder why so many "numerologists" are also putting in their "two cents worth"?
Here are the numbers you quote:
137.035999044
137.035999073
137.035999084
137.035999173
The mean of those is about
137.035999094
and the high and low are within 80 parts in 137000000000. Half that would probably be a statistically reasonable estimate of the true error in the value, so call it
137.035999094(40)
as a good representation of the value and its uncertainty overall, assuming the other errors are random. The error is about 1 part in 3 billion. You might want to call that "numerology", but I'm not aware of any numerology results that have an uncertainty in the part-per-billion range.

starcanuck64
2013-Oct-07, 06:37 PM
How often, and at what cost? What about just kinda bad ideas? A decreased S/N ratio (http://en.wikipedia.org/wiki/Signal-to-noise_ratio) makes it harder for honest researchers to do useful work.

Taken far enough it would eventually overwhelm any rational production of new ideas.

A.DIM
2013-Oct-09, 02:15 AM
Thanks for the input, all.

The "retraction watch" website mentioned in the article is rather shocking (but not really), even while it is the biomed/psych/anthro sciences which figure most prominently.

And to compound the problem: Scientist are "bad at judging peers..." (http://phys.org/news/2013-10-scientists-bad-peers-published.html).

I've no idea how to "fix" Science but I guess identifying some of the problems is a start... :D

Don Blazys
2013-Oct-09, 09:40 AM
Those are probably 1 sigma values, which makes your point invalid
Unfortunately, you are wrong. My point is perfectly valid. Most scientists would certainly agree that the current CODATA value of the fine structure constant which is: \alpha^{-1}=137.035999074(44) and the so called "best experimental value" of the fine structure constant which is: \alpha^{-1}=137.035999173(35) do indeed stand in disagreement with each other.

For instance, this paper by Charles Rhodes Ph.D. (Albert A. Michelson Professor Emeritus of Physics, University of Illinois):

http://arxiv.org/ftp/arxiv/papers/1212/1212.5941.pdf

says exactly that, gives other experimental values of \alpha^{-1} that also stand in disagreement with the "best one", and even offers his opinion that this disagreement "points to the failure of the conventional QED picture at sufficiently high precision".

Determining the true and exact value of the fine structure constant is of the utmost importance to science because that number will play a key role in the newly proposed International System of Units and the redefinition of the Kilogram (which has already been delayed for several years). Can you imagine how science will suffer if this is done by averaging out a bunch of experimental results, or in some other unsatisfactory manner?

I like the Cosmoquest philosophy of the "citizen scientist" getting involved and actually participating in the advancement of science. That's why I propose that, if there are any really good coders here, then they should help investigate this counting function:

http://www.mathisfunforum.com/viewtopic.php?pid=222133

by determining higher values of \varpi(x). You see, if this counting function proves to be sufficiently accurate, then we will have a value for \alpha^{-1} that not only stands in agreement with the experimental evidence, but also has a definite mathematical purpose! In other words, we will have demonstrated that, just as \pi and e are used in both math and physics, \alpha^{-1} is also both a mathematical and a physical constant!

That, in my opinion, would be a truly logical foundation for redefining the Kilogram.

Shaula
2013-Oct-09, 11:31 AM
Unfortunately, you are wrong.
Frequently, but not in this case. That paper compares eight order approximations modelling for one measurement to tenth order loops for another and is surprised that they are different in the last few digits. That alone makes be doubt the fairness of his analysis. The paper has also not been reviewed or submitted to a peer reviewed journal. In fact none of his three papers on this seem to have been peer reviewed. So it is not that useful in making your point that most scientists would agree. As I said, different assumptions (or in this case number of loops) lead to different results. And as they are sigma values the disagreement is not a binary thing, it is probabilistic and not very bad.

Don Blazys
2013-Oct-10, 11:21 AM
Frequently, but not in this case.
Unfortunately, you are wrong in this case as well.


That paper compares eight order approximations modelling for one measurement to tenth order loops for another and is surprised that they are different in the last few digits. That alone makes be doubt the fairness of his analysis.

His analysis is fair.

Those two values are \alpha^{-1}=137.035999084(51) and \alpha^{-1}=137.035999173(35) respectively, and the difference between them is more than twice the average standard error. Most rational and reasonable people would view that as a statistically significant disagreement.


The paper has also not been reviewed or submitted to a peer reviewed journal. In fact none of his three papers on this seem to have been peer reviewed.
Well, that's a "shoot yourself in the foot" argument if ever I heard one because that paper on "tenth order loops" touting the value \alpha^{-1}=137.035999173(35) doesn't seem to have been peer reviewed either! However the paper on "eight order approximations modelling" touting the value \alpha^{-1}=137.035999084(51) has been peer reviewed, and in a most prestigious journal at that.

By the way, that peer reviewed value: \alpha^{-1}=137.035999084(51) is exactly the value that seems to give the best approximations of the density of "regular figurative numbers" in my Polygonal Number Counting Function which, for your convenience, can be found here:

http://www.mathisfunforum.com/viewtopic.php?pid=222133

Shaula
2013-Oct-10, 02:16 PM
His analysis is fair.
Sorry, would rather see peer review. Your assertions don't really cut it for me. You don't like my assertions that you are wrong, I don't like your assertions that he is right... Hmm, if only there was some way to settle this. Some sort of accepted review process by a body of knowledgeable people that would serve to reduce the pointless back and forth I can sense looming in which you say "RIGHT!" and I say "WRONG!" over and over... Oh yes, there is. Only your sources don't seem to be willing to put their work through it.


Those two values are \alpha^{-1}=137.035999084(51) and \alpha^{-1}=137.035999173(35) respectively, and the difference between them is more than twice the average standard error. Most rational and reasonable people would view that as a statistically significant disagreement.
Hmm, let me see - take the upper one sigma bound for one to get 137.035999131. Take the lower one sigma bound for the other to get 137.035999133. Suddenly they look a lot closer. And that is just the one sigma value. For values derived using different approximations. That looks like very good agreement to me.


Well, that's a "shoot yourself in the foot" argument if ever I heard one because that paper on "tenth order loops" touting the value \alpha^{-1}=137.035999173(35) doesn't seem to have been peer reviewed either! However the paper on "eight order approximations modelling" touting the value \alpha^{-1}=137.035999084(51) has been peer reviewed, and in a most prestigious journal at that.
Actually that would support my point more strongly. Really only your foot got perforated in this exchange. So you are comparing 8th and 10th order approximations, only one of which has been peer reviewed. In a non-peer reviewed paper. And think you can claim that this throws experimental physics into disarray?

noncryptic
2013-Oct-10, 02:54 PM
Shame on phys.org for such a broad sweeping title, in spite of the fact that the crisis is limited to a few aptly called "money sciences".


Those two values are 137,... and 137,... respectively, and the difference between them is more than twice the average standard error.

The article does not even mention physics in relation to this reproducibility crisis.


Most rational and reasonable people would view that as a statistically significant disagreement.

Statistically significant, yes. But not evidence of a reproducibility crisis in physics. It is the stuff that science is made of, it is hard and gritty.
If it would be an indication that physics is in a reproducibility crisis, it has always been in a reproducibility crisis. And yet here we are, with our GPS enabled smartphones that are testimony to physics generally being plenty accurate for practical applications - and being the process that it is, ever improving.

tusenfem
2013-Oct-10, 04:06 PM
Most rational and reasonable people would view that as a statistically significant disagreement.



Do not use this kind of argumentation, where you sideways claim that your opponent is irrational and unreasonable for not agreeing with you.

Buttercup
2013-Oct-10, 04:12 PM
Do not use this kind of argumentation, where you sideways claim that your opponent is irrational and unreasonable for not agreeing with you.


:clap:

steve_bnk
2013-Oct-10, 06:55 PM
If it ain't broke don't fix it.


Science is populated with humans...not dispassionate Spocks from Vulcan.

What sets science apart is that in the long run though a competitive process falsehoods become visible.


Maxwell's prediction of electromagnetic wave propagation was not demonstratable until after he penned the idea when instruments were developed.

A good example was cold fusion. It took about 48 hours from the claim of achieving cold fusion for it to be globally shown not independently reproducible.

The thing is we are al familiar with the success of figures Einstein and Maxwell, but most never see all the theories that came and went. The process works.

And of course a mass of writers who write controversial articles and books about science....to get published.

ShinAce
2013-Oct-10, 07:06 PM
Don:
Agreement in a scientific context is usually that one value is within 2 sigma of the other. Shaula showed one way to look at that, which is standard, and it indicates there is agreement with ALL of the fine structure constants shown. There are no outliers beyond 3 sigma and they're all in the 1-2 sigma range, which is normal. It's plain old stats: the normal distribution and error analysis.

This means you could take a weighted average and get the more precise answer you seek. But it doesn't change that there are no signs that the fine structure constant is the wrong value. It also doesn't change that the paper you linked really had little to do with alpha and more to do with the cosmological constant(omega).

Van Rijn
2013-Oct-11, 12:01 AM
It can sometimes take a long time and at a terrible cost.

It took nearly 50years to stop the ideas of Trofim Lysenko (http://en.wikipedia.org/wiki/Trofim_Lysenko) from being applied to genetics and agriculture in many Marxist countries; millions may have starved as a result.

That had nothing to do with the scientific process. It was never taken very seriously by scientists except where it was dangerous to suggest it was wrong. Lysenko was politically favored by Stalin, and it would have been very bad to disagree with him.

Quoting from the article you linked to:

In 1964, physicist Andrei Sakharov spoke out against Lysenko in the General Assembly of the Academy of Sciences:

"He is responsible for the shameful backwardness of Soviet biology and of genetics in particular, for the dissemination of pseudo-scientific views, for adventurism, for the degradation of learning, and for the defamation, firing, arrest, even death, of many genuine scientists."


After Stalin died in 1953, Lysenko lost most of his clout, and you can see from the quote that within a decade he was very much politically unfavored and scientists there could publically discuss what he had been doing.

Don Blazys
2013-Oct-11, 10:34 AM
Agreement in a scientific context is usually that one value is within 2 sigma of the other.

I agree.

So let's make this simple.

Here again is the Wikipedia article on the fine structure constant:

http://en.wikipedia.org/wiki/Fine-structure_constant

As we all can see, it says that the currently accepted CODATA value is: \alpha^{-1}=137.035999074(44),

and that the "best" experimental value is: \alpha^{-1}=137.035999173(35)

whose 2 sigma values are .000000088 and .00000007 respectively.

Now, 137.035999173-137.035999074=.000000099,

and it is perfectly clear that the difference between the CODATA value and the "best" experimental value is substantially greater than 2 sigma regardless of which value of 2 sigma we choose. Thus, the above two values of the fine structure constant do indeed stand in disagreement with each other and thus "ShinAce", "Shaula" and everyone who agreed with them on this issue are wrong.


Statistically significant, yes.

Thank you for your courage and for being smart enough to agree with me on that particular issue.


But not evidence of a reproducibility crisis in physics.

I found no evidence that the so called "best" experimental value of the fine structure constant was ever even peer reviewed, so I doubt that it was ever reproduced by an independent researcher. Thus, Wikipedia may be reporting that so called "best" experimental value of the fine structure constant prematurely or even inadvertently disseminating misinformation. This is a serious issue because if we don't determine the correct value of this extraordinarily important number, then the Kilogram will be poorly redefined which will, in turn, adversely effect all of science, which requires ever increasing accuracy and precision in its measurements. I'm no alarmist and maybe "crisis" is a tad too strong a word to describe this situation, but I do hate to see anything slow down the advancement of science, which I believe is mankind's only hope.


If it would be an indication that physics is in a reproducibility crisis, it has always been in a reproducibility crisis. And yet here we are, with our GPS enabled smartphones that are testimony to physics generally being plenty accurate for practical applications - and being the process that it is, ever improving.

I'm a 63 year old grandfather who's lived an incredibly interesting life, so I hope for the sake of you younger folks that it's improving fast enough.


Science is populated with humans...not dispassionate Spocks from Vulcan.

You don't know that! This universe is an awfully big place and a lot of very reputable scientists believe that the probability of there being intelligent life forms out there is 1.:lol:

Swift
2013-Oct-11, 01:50 PM
Enough about the wikipedia article on the Fine Structure Constant

As a point of discussion for this thread, it was fine, but I do not want to see this thread turned into a debate on that topic. If you wish to have that discussion, please start your own thread in Astronomy.

prefixus
2013-Oct-24, 11:15 PM
"So how do we fix Science?"

Now that is the multi-billbion dollar question that needs to be address at CERN. Thanks for bringing this article to my attention.