PDA

View Full Version : Modus tollens



upriver
2005-Sep-09, 08:55 PM
"One study found that the vast majority of scientists
drawn from a national sample showed a strong preference
for "confirmatory" experiments. Over half of these
scientists did not even recognize disconfirmation (modus
tollens) as a valid reasoning form! In another study the
logical reasoning skills of 30 scientists were compared
to those of 15 relatively uneducated Protestant
ministers. Where there were performance differences, they
tended to favor the ministers. Confirmatory bias was
prevalent in both groups, but the ministers used
disconfirmatory logic almost twice as often as the
scientists did." ~Michael J. Mahoney, Publication
Prejudices: An Experimental Study of Confirmatory Bias in
the Peer Review System Cognitive Therapy and Research,
Vol. 1, No. 2, 1977, pp. 161-175.
http://confirmatory-bias.behaviouralfinance.net/Publication%20Prejudices%20An%20Experimental%20Stu dy%20of%20Confirmatory%20Bias%20in%20the%20Peer%20 Review%20System.htm

Modus Tollens
http://en.wikipedia.org/wiki/Modus_tollens

antoniseb
2005-Sep-09, 09:08 PM
This is one study done in 1976. I'm not certain where they found fifteen relatively uneducated protestant ministers. Most of the protestant ministers I know have Masters degrees, so right away, I'm suspecting something odd is going on here.

More importantly however, this is not an alternative theory, it is not even about astronomy or astronomers. You seem to be simply exploring a way of bashing mainstream scientists.

Did you have some other goal? Is there a reason I shouldn't move this to Off-Topic BABBling?

upriver
2005-Sep-09, 09:45 PM
Most of the protestant ministers I know have Masters degrees, so right away, I'm suspecting something odd is going on here.

More importantly however, this is not an alternative theory, it is not even about astronomy or astronomers. You seem to be simply exploring a way of bashing mainstream scientists.



You have just proven this paper to be correct.
You have allowed your bias to color a discussion of the premise of this paper.

This is posted in the context of "bashing" ATM theories. Also standard science says there is nothing wrong with the peer review system, so this is definately an ATM topic. As the paper says, scientists introduce bias variables into the peer review system.

Duane
2005-Sep-09, 09:51 PM
You have just proven this paper to be correct.
You have allowed your bias to color a discussion of the premise of this paper.

This is posted in the context of "bashing" ATM theories. Also standard science says there is nothing wrong with the peer review system, so this is definately an ATM topic. As the paper says, scientists introduce bias variables into the peer review system.

Upriver, you seem to have missed the question.

How is this paper related to astronomy, the universe, stars, etc? Looks pretty off-topic to me. :confused:

iantresman
2005-Sep-09, 11:26 PM
How is this paper related to astronomy, the universe, stars, etc? Looks pretty off-topic to me. :confused:

Perhaps its relevance is indirect: Peer review and the scientific method are both used by astronomy. The paper also reminds me of the following:


Why Most Published Research Findings Are False (http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124) (2005)
British scientists exclude 'maverick' colleagues, says report (http://www.eurekalert.org/pub_releases/2004-08/cu-bse081204.php) (2004)
Peer review is stifling for scientists on fringe (http://www.edifyingspectacle.org/gullibility/blog/archives/pierremarie_robitaille_an.php) (2002)
Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System (http://confirmatory-bias.behaviouralfinance.net/Publication%20Prejudices%20An%20Experimental%20Stu dy%20of%20Confirmatory%20Bias%20in%20the%20Peer%20 Review%20System.htm) (1997)
Trial by peers comes up short (http://www.guardian.co.uk/print/0,3858,4583809-111019,00.html) (2003)
Rejecting Nobel class papers (http://www2.uah.es/jmc/nobel.html) (2003)
Suppression Stories (http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/) (1997)
Challenging dominant physics paradigms (http://www.uow.edu.au/arts/sts/bmartin/pubs/04jse.html) (2004)
Publications on whistleblowing and suppression of dissent (http://www.uow.edu.au/arts/sts/bmartin/pubs/supp.html) by Dr Brian Martin


Regards,
Ian Tresman

Van Rijn
2005-Sep-09, 11:45 PM
I suppose you might make the argument that this could go in the "General Science" section.

But, I look at this and ask ... your point is?

We've gone over this before. Scientists aren't Vulcans. They are human like anyone else, and will have opinions. Also, some fields are more political than others, especially in biology/genetics, ecology/climate, and anything that touches on real-world nuclear applications, civilian and military.

So fine, it ain't perfect. But the ultimate goal is science, so whatever the faults of the mainstream may be, it doesn't give against-the-mainstream folks a free pass. They still have to do the science. They still have to show the math.

So, again, what's the point? Especially when discussing a subject like astronomy, where the political/religious controversy is minimal.

Nereid
2005-Sep-10, 12:16 AM
Unless and until someone can make clear that this is specifically about astronomy (etc), I think the home for this is 'General Science'. And I've also decided to be decisive (a mod, or admin, can always move it back, if warranted).

Ricimer
2005-Sep-10, 12:19 AM
I must be atypical then...as the work I've done that is enroute to being published (one is actually in review now) tends to be just such work. I really do prefer checking to see if people are wrong.

Example of something I've done: The literature says various types of quasars vary more or less than other types, the predictions are based upon the models used to explain the quasars behavior.

The work I've done found that specific quasars that are clearly defined types, vary in ways contrary to these predictions (basically they all vary abou the same amount)...as such the common wisdom is incomplete (if not just wrong).

Part of the problem with Modus Tollens is that many scientific experiments and predictions aren't very clearcut. The prediction may be incomplete, to general, and other aspects may be true. Just that particular manifestation is wrong.

Take that quasar thing as an example. It doesn't mean the models are completely wrong, perhaps they are only misapplied, or a couple of models can be replaced by a single model.

Ricimer
2005-Sep-10, 12:21 AM
Also, most people I talk to admit there are problems with the peer review system. Specifically the peers are human. They are opinionated, have motives, and have differing levels of experience in various fields.

However, nobody has really figured out a better way of handling the publishing of scientific papers.

upriver
2005-Sep-10, 03:05 AM
Take that quasar thing as an example. It doesn't mean the models are completely wrong, perhaps they are only misapplied, or a couple of models can be replaced by a single model.




Also, most people I talk to admit there are problems with the peer review system. Specifically the peers are human. They are opinionated, have motives, and have differing levels of experience in various fields.

However, nobody has really figured out a better way of handling the publishing of scientific papers.


Thank you. That all I wanted. I think this also applies to us as we try to figure out whether someones theory has merit but is incomplete.

JohnnyW
2005-Sep-10, 05:33 AM
I only monitor the "Against the Mainsteam" (formerly Alternate Theories). I felt the original post and the links provided by iantresman were excellent -- it is really something to worry about if you EVER expect to take a new theory INTO the Mainstream. I am sorry to see this moved, but I guess I will add General Science to my watchlist unless it becomes too time consuming. upriver and ian -- many thanks.

iantresman
2005-Sep-10, 01:05 PM
However, nobody has really figured out a better way of handling the publishing of scientific papers.
I think there have been several good suggestions about improving the system, but few have been taken up because (a) there is no incentive (b) mainstream looses its "power" (hence money).

How many people are even aware of suggestions that might improve peer review? And I think that sums it up. People don't care because they are content with the status quo.

I think it is interesting that although "science" aims to be impartial, open, and having a number of checks and balances, it doesn't apply the same criteria to the very system that publishes science. Peers are often impartial (because they have their own competing theories), closed (peers are anonymous and unaccountable), and there is no opportunity for checks and balances.

Regards,
Ian Tresman

Fram
2005-Sep-10, 01:23 PM
Well, present a suggestion for improving it that you think is good, and we can discuss it.

Nereid
2005-Sep-10, 02:18 PM
Perhaps its relevance is indirect: Peer review and the scientific method are both used by astronomy. The paper also reminds me of the following:

Why Most Published Research Findings Are False (http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0020124) (2005)
British scientists exclude 'maverick' colleagues, says report (http://www.eurekalert.org/pub_releases/2004-08/cu-bse081204.php) (2004)
Peer review is stifling for scientists on fringe (http://www.edifyingspectacle.org/gullibility/blog/archives/pierremarie_robitaille_an.php) (2002)
Publication Prejudices: An Experimental Study of Confirmatory Bias in the Peer Review System (http://confirmatory-bias.behaviouralfinance.net/Publication%20Prejudices%20An%20Experimental%20Stu dy%20of%20Confirmatory%20Bias%20in%20the%20Peer%20 Review%20System.htm) (1997)
Trial by peers comes up short (http://www.guardian.co.uk/print/0,3858,4583809-111019,00.html) (2003)
Rejecting Nobel class papers (http://www2.uah.es/jmc/nobel.html) (2003)
Suppression Stories (http://www.uow.edu.au/arts/sts/bmartin/dissent/documents/ss/) (1997)
Challenging dominant physics paradigms (http://www.uow.edu.au/arts/sts/bmartin/pubs/04jse.html) (2004)
Publications on whistleblowing and suppression of dissent (http://www.uow.edu.au/arts/sts/bmartin/pubs/supp.html) by Dr Brian Martin

Regards,
Ian Tresman
As you seem to have researched this Ian, do you have the following:

- any rebuttal papers to any of the above?

- quantitative analyses (I note that at least three of the works you cite do have at least some quantitative basis)?

- an understanding of why all the quantitative studies (in the works you cite) are in 'human' fields (cognitive psychology, biomedical research, and medicine)?

I note that, of the specifics I checked (in Campanario and Martin 2004 paper; there may be some in the four links I haven't checked yet), only two in the field of astronomy are cited ("A sample of well-qualified challengers to orthodox physics"):
- Arp
- Van Flandern
(there are three in hard-core physics: Assis, Marmet, and Spenser).

If this is the strongest case which you can present, then I for one would conclude that there would indeed seem to be little incentive to work for change.

Why? Because, in astronomy at least, it 'ain't broken' (in any measurable way).

Didn't we have a thread, in either BA or UT, where we discussed this topic, from a purely planetary sciences, astronomy, astrophysics, cosmology perspective? Why not revive that, to continue our discussion, focussing on what we members are most interested in (and, likely, knowledgable about)?

Ricimer
2005-Sep-10, 09:52 PM
If you think there are suggestions out there, lets hear them. They tend not to work out to well.

upriver
2005-Sep-10, 10:44 PM
Why? Because, in astronomy at least, it 'ain't broken' (in any measurable way).


What makes astronomers holier than thou?
Maybe you have not tested it. Put together a fake study
using an ATM theory and then one using a mainstream theory
and see which has the greater % of rejections. You
could find an ATM theory that is true but not widely
accepted.
I'm not not bashing, I'm just stating the obvious.

Eta C
2005-Sep-10, 11:12 PM
What makes astronomers holier than thou?
Maybe you have not tested it. Put together a fake study
using an ATM theory and then one using a mainstream theory
and see which has the greater % of rejections. You
could find an ATM theory that is true but not widely
accepted.
I'm not not bashing, I'm just stating the obvious.

Because nature doesn't lie. People do. The "human" sciences are behind the physical sciences (such as physics, astronomy, and chemistry) in their quantitative rigor. Not a knock on these fields, that's just the nature of the beast so far.

And despite your disclaimer, you are bashing practicing scientists with your statements. And no, they are not obvious. If the ATM theory, as is true for most of them (especially electric universe ideas) makes no quantitative predications that can be compared to experiment and the mainstream theory does, and matches those observations well, why are you surprised the former is rejected. I have yet to see an ATM idea proposed on these boards that would meet even the loosest requirements for acceptance.

VanderL
2005-Sep-11, 07:56 PM
If you think there are suggestions out there, lets hear them. They tend not to work out to well.

Would Crichton's idea be useful of making the peer system "public", complete with names and comments of the reviewers? It could get rid of some of the more "politically" motivated comments (maybe Crichton wasn't the first to suggest this, but I first read it in his awful novel State of Fear).

Cheers.

Arneb
2005-Sep-11, 08:04 PM
Might be a problem, too: Suppose two researchers are friends or, worse, reaearcher No. 1 is in some position of dependence on researcher No. 2 (wants him to give a favourable opinion on a grant; would like No. 2 to accept him as postdoc; No. 2 is on the board which decides about No. 1's tenure; etc.). Now No.2 submits a paper which is junk, and No. 1 is chosen as a referee. Only if he remains anonymous can he dare reject the junk paper. In an open system he would shoot himself in the foot by writing a rejection.

So the main danger with open peer review is massive nepotism - the greater danger, if you ask me. Of course, anonymity can be abused, too, but it is the oly environment in which one can be frank without the fear of recrimination.

Ricimer
2005-Sep-12, 12:57 AM
In astronomy at least, it's possible to "figure out" who's the referee just by the comments they make.

Then again, astronomy is a suprisingly small field in terms of researchers. So it's likely you know 90% of the people in your particular field.

papageno
2005-Sep-12, 09:37 AM
Would Crichton's idea be useful of making the peer system "public", complete with names and comments of the reviewers? It could get rid of some of the more "politically" motivated comments (maybe Crichton wasn't the first to suggest this, but I first read it in his awful novel State of Fear).

Would you put your name on your vote in the next elections?

VanderL
2005-Sep-12, 05:32 PM
Would you put your name on your vote in the next elections?

Sure. I even did some refereeing a while back. I wouldn't have minded to send my comments straight to the authors, and if there was a possibility for a backlash because of my refereeing, I could have declined, being a referee is not mandatory after all.

Cheers.

papageno
2005-Sep-12, 05:54 PM
Would you put your name on your vote in the next elections?
Sure. I even did some refereeing a while back. I wouldn't have minded to send my comments straight to the authors, and if there was a possibility for a backlash because of my refereeing, I could have declined, being a referee is not mandatory after all.
Anonymity helps avoiding pressure on referees, helping them to give freely their opinion on a paper.
If the referees are not anonymous, pressures might stifle disagreement with the authors.

Do you really expect free elections, if the vote were not anonymous?

So the main danger with open peer review is massive nepotism - the greater danger, if you ask me. Of course, anonymity can be abused, too, but it is the oly environment in which one can be frank without the fear of recrimination.

VanderL
2005-Sep-13, 05:24 PM
What kind of pressure do you think is on the referee if it was done publicly? I don't see this pressure on the referee, as I said refereeing is not mandatory, you can always decline.

And on second thought, the analogy with voting is also poor, as in some countries the loss of anonimity could mean the loss of life. This is totally opposite to science, where everybody has the same goal: "to further our knowledge". Many authors even thank referees for their helpful suggestions.

Cheers.

Manekineko
2005-Sep-14, 08:58 AM
On the other hand, maybe submitted papers could be "anonymized" before sending them to the reviewers: bias strikes both ways, and a reviewer might refuse (or worse ;) accept) a paper simply on the strength of the author, and not the paper itself.

pghnative
2005-Sep-15, 05:58 PM
Perhaps a combination of the two could work. Two groups of reviewers, one public and one anonymous. If the opinions are split, then the journal's senior editor (is there such a thing?) reviews all of the reviews and makes a final decision.

If a public reviewer consistently reviews papers differently than the anonymous ones, ultimately their reputation would suffer.

JohnD
2005-Sep-15, 10:39 PM
Eta C,
It's not just the 'human' sciences that are prone to human bias and even fraud. How about that guy who was faking his results in the behaviour of transistor materials a while back?

The 'human' sciences (biology, medicine?) are indeed less rigorous than their physical siblings, as liable to personal ambition, but very much more sensitive to commercial pressure. When the development of a new medication can mean wild success or bankruptcy for a pharmceutical company, there is a strong tendency for studies that show negative results (Vioxx?) or even just a nul result when compared with existing drugs to be shelved rather than published.

The Public Libraray of Science is one way to deal with this, though how that deals with studies and researchers for whom publication is at the whim of commerce I don't know.

John

Eta C
2005-Sep-16, 01:04 PM
Eta C,
It's not just the 'human' sciences that are prone to human bias and even fraud. How about that guy who was faking his results in the behaviour of transistor materials a while back?

The 'human' sciences (biology, medicine?) are indeed less rigorous than their physical siblings, as liable to personal ambition, but very much more sensitive to commercial pressure. When the development of a new medication can mean wild success or bankruptcy for a pharmceutical company, there is a strong tendency for studies that show negative results (Vioxx?) or even just a nul result when compared with existing drugs to be shelved rather than published.

The Public Libraray of Science is one way to deal with this, though how that deals with studies and researchers for whom publication is at the whim of commerce I don't know.

John

My intent wasn't to say that the physical sciences are immune to deception. Just take a look at the whole cold fusion fiasco for an example of how a large group of scientists deluded themselves. One reason may have been the obvious commercial boon that would have come had they been right. One difference, however, is that in most physical sciences such a deception is hard to pull off. Someone else will be sure to do the experiment and if there is no "there there" it will be found out.

The main issue is how large is the effect one is trying to observe, how high is the background level, and how large are the systematic uncertainties. If the effect is small and the background and systematics large it is easier to fudge the results should one be so inclined. In my opionion these conditions are more prevalent in biological and human sciences than they are in the physical sciences. Therefore the opportunity for fudged results is higher. When you add in the commercial stakes, the motive for acting on the opportunity is higher. Of course when the same conditions occur in a physical science (as in cold "fusion") the same sad result can occur.

jkmccrann
2005-Dec-27, 03:33 PM
I think the fact is that with any system used to validate scientific works, there are inevitably going to be political dimensions brought into the whole process as a matter of course. There is simply no way to escape that.

For me, and as many of you would well be aware, I am certainly no scientist, but peer-review strikes me as an eminently sensible way to go about judging new works and new theories brought up for consideration. Of course there are going to be entrenched interests in any field, some of whom will try to defend their patch of turf and their vested interests as much as possible, but anonymous peer review is as far as I can see it the best way to deal with it. Doing away with the anonymity is certainly not the path to go down to make things better, that introduces too many other destructive variables and certainly increases the whole politicisation of the process which is surely something that should strive to be marginalised?

snarkophilus
2005-Dec-27, 04:06 PM
In astronomy at least, it's possible to "figure out" who's the referee just by the comments they make.

Then again, astronomy is a suprisingly small field in terms of researchers. So it's likely you know 90% of the people in your particular field.

For Journal of Physical Chemistry papers (and some other ACS publications), you get to recommend reviewers for your papers. Of course, it's up to the editor to decide whether or not those reviewers are used, but I bet there's usually a good chance you'll get those particular people.