PDA

View Full Version : The Carter Catastrophe - A Statistical Doomsday Argument



Madalone
2005-Nov-08, 09:04 AM
I am currently re-reading Stephen Baxter's "Time". In this book, he uses (not necessarily endorses) the so called Carter Catastrophe. As Baxter is an author who usually has his science right (if on the speculative side), his use of this doomsday argument - even as a plot device - is heckling me.

The argument runs as follows (extract from "The Doomsday Argument" on www.anthropic-principle.com (http://www.anthropic-principle.com)):


Imagine that two big urns are put in front of you, and you know that one of them contains ten balls and the other a million, but you are ignorant as to which is which. You know the balls in each urn are numbered 1, 2, 3, 4 ... etc. Now you take a ball at random from the left urn, and it is number 7. Clearly, this is a strong indication that that urn contains only ten balls. [...]

But now consider the case where instead of the urns you have two possible human races, and instead of balls you have individuals, ranked according to birth order. As a matter of fact, you happen to find that your rank is about sixty billion. Now, say Carter and Leslie, we should reason in the same way as we did with the urns. That you should have a rank of sixty billion or so is much more likely if only 100 billion persons will ever have lived than if there will be many trillion persons. Therefore, by Bayes' theorem, you should update your beliefs about humankind’s prospects and realize that an impending doomsday is much more probable than you have hitherto thought.

In Baxter's book, one character calculates based on this argument that Doomsday is 150, maximum 200 years away.

Deep in my gut I have the feeling that this argument is fundamentally flawed (read: rubbish), but I can't come up with a clean and neat rebuttal. The best I can come up with: As the human population has grown roughly exponentially in the known past, exactly the same argument has been valid for each and every generation before us - for instance also for the generation that lived, say, 300 years ago. And yet we are here, 300 years after an imminent doom.

What are your takes on this argument?

Fram
2005-Nov-08, 09:23 AM
Yep, a nonsense argument. It is not because something has started only recently that it will end soon, or vice versa. You could just as well argue that the human race has an inherent life span of 1 quintillion people, and that because we are only at number 12 billion now, the impending dommsday is still far far away. This argument is equally rubbish.
A better way would be to calculate the average lifespan of a species on Earth that has survuved for at least 100,000 years (I mean from evolution till extinction, not the lifespan of individuals), and compare that to ours. It would still be only a statistic, not a certainty by far, but it might give a better idea.

01101001
2005-Nov-08, 09:49 AM
Shark number 7 must have been even more certain that the sharks were about to pass from this planet. Here the sharks are 200 million years later, wondering what the heck number 7 was thinking.

Ken G
2005-Nov-08, 10:09 AM
I've also wondered about this argument, though I didn't know it had a name. I thought of it when I heard it mentioned that it is "natural" for the Sun to be about halfway through its main-sequence lifetime, since it would be unlikely to be very near the beginning or very near the end. I don't know the relative merits of the two different applications of this general idea, but I would agree that Madalone's argument completely shatters the Carter catastrophe idea. (Fram's point is also well taken, but there is already nothing left of the argument that survives the OP!). It's a little like asking someone who wins a grand prize in a lottery what they would estimate their chances of winning were. It would boggle their mind if they hadn't really thought about it before. Yet somebody had to win, and that person is in such a special position that you can't ask them to apply the same logic as a "normal" person.

That's in essence what Madalone has said, and has shown that the Carter catastrophe hypothesis has already been disproven. The only thing I would add is that this is an example of the subtleties presented by the anthropic principle-- you can't ask the people who are in a special place to use probability arguments that assume they are not in a special place. Otherwise, why could I not turn the catastrophe argument around, and conclude that humans will survive for a significant fraction of the duration of the universe, otherwise if "today" is a date chosen at random, what would have been the likelihood that humans would be here today?

Madalone
2005-Nov-08, 11:44 AM
Good point, I like this one.

To retranslate it into the urn/ball thougt experiment:

"Imagine that two big urns are put in front of you, each containing a million red and blue balls. You know that one of them contains red and blue balls fifty-fifty and the other a thousand red balls in just under a million blue ones, but you are ignorant as to which is which. Now you take a ball at random from the left urn, and it turns out to be red. Clearly, this is a strong indication that that urn contains red and blue balls fifty-fifty.

Now consider the balls depicting millenia, where red balls stand for the time humanity exists, it is easiy to see that humankind must exist for a seizable portion of the lifespan of the universe."

But as I am writing this, I see a catch: Arguing from Carter's perspective, it is now obvious that doom is imminent not only for mankind, but for the universe as a whole: The fewer blue balls are in the second urn (i.e. the shorter the livespan of the universe), the better is the probability to draw a red one. Hence, the probability of being alive today is maximized if the universe ends tomorrow.

I am starting to like this line of reasoning. I guess one can prove everything (and its opposite) applying probabilities "ex post" on preexisting facts... ;)

grant hutchison
2005-Nov-08, 05:46 PM
Simply noting that people in the past would have been wrong when they used the Carter argument to predict their own imminent demise doesn't get you out of Carter's bind. It's inevitable that from any vantage point in time we'll see a group preceding us who "would have been wrong". But because of the exponential growth in population, that group is always small compared to the mass of people living today and in our future.
Carter's saying that a small number of people will be wrong to use his argument, but a large number will be right, because they'll be close to the extinction event, or the start of the exponential decline in population - they'll be in the population bulge, where the area under the population curve is high.
So if we choose a life at random from under the population curve, we have a small chance of ending up in the group who are wrong (those in the early days of population growth), and a large chance of ending up in the group who are right (those in the population bulge before the decline).
The way out of Carter's argument is to point out that exponential growth is unsustainable, and if we are to survive we are going to have to achieve population equilibrium. Once the population curve becomes a level line leading off to infinity, we're just as likely to find ourselves anywhere under that line (because there's no bulge to "concentrate lives" in one temporal region), and so we can't infer anything about survival time in the future.
Either we have exponential growth, and Carter's right, and we're likely to be living in the End of Days; or we level out our population, and Carter can't make any deductions about the future - the areal midpoint under an infinite level curve is anywhere.

Grant Hutchison

Edited for clarity, by changing "to use" to "when they used" in sentence 1

eburacum45
2005-Nov-08, 06:29 PM
I see the Carter Doomsday argument is not receiving much credence here at this point in time; funny, because last time it was discussed it did have a few supporters.
The most debatable point about this argument is the fact that it assumes that a person living now has no special place in the sample of all humans who have ever lived and will ever live; a kind of argument of Copernican mediocrity principle with respect to our position in a population of unknown size.

But our position in that population is not chosen at random; we are alive at the exact period when statistical maths develops to the point where predictions like Carter's can be made for the first time. This coincides with the period when the population is rising out of the long slow growth of the pre-industrial age towards a much more densely populated developed civilisation; a change which is still in progress. I think it is safe to say that these two periods are more likely to coincide than not; an increase in mathematical knowledge is bound to accompany a successful indiustrial civilisation.
In other words, as Robin Hanson has said,
All else is not equal; we have good reasons for thinking we are not randomly selected humans from all who will ever live.

See this wiki page about the arguments for and against this concept;
http://en.wikipedia.org/wiki/Doomsday_argument

grant hutchison
2005-Nov-08, 11:11 PM
All else is not equal; we have good reasons for thinking we are not randomly selected humans from all who will ever live.I'm not sure I buy this one, at least not without seeing some elaboration. It seems that the "privileged viewpoint" card being played here is irrelevant to the particular kind of mediocrity relevant to the argument. Carter's maths apply whether or not the population is aware of it. We're interested only in the area under the curve, not which bit of that curve is self-aware.
A bunch of superintelligent aliens roaming the galaxy might stop at a mediaeval planet and say: "There's a 95% chance they'll all be dead before x more births." Then they see us, and say "There's a 95% chance they'll all be dead before y more births. Then they move on to a star-spanning civilization that has been aware of Carter's calculation for many millennia: "There's a 95% chance they'll all be dead before z more births." If Carter's maths apply, 5% of such predictions will be wrong, the other 95% will be correct.

Grant Hutchison

Edit:If anything, Hanson's suggestion serves to undermine one of the counterarguments to Carter: that external evidence indicates species survive for many more million years than we have done so far, so we are more likely to be at the start of our career as a species. But our privileged viewpoint as a self-aware, mathematical, industrial civilization means that we are not like other species, and so their data cannot be applied to us.

Ken G
2005-Nov-09, 02:36 AM
Simply noting that people in the past would have been wrong when they used the Carter argument to predict their own imminent demise doesn't get you out of Carter's bind.

I don't agree. The point is, anyone who makes the Carter argument will always come to the conclusion that the end is fairly near. Exponential growth is not a crucial logical component, it just means that you infer the end is very near. But the conclusion is totally unsupportable either way, because everyone will reach a different conclusion for the likely survival time, depending on when they are alive. For example, if Cain and Abel had used the argument, regardless of whether or not the growth is exponential at the time, they would have found it totally unbelievable that humanity would survive them by thousands of years. I say that an argument that is wrong when used by them is also wrong when used by us, as there is nothing to distinguish us. The other arguments against the Carter hypothesis are sound, but unnecessary-- this idea is logically DOA (even though it may be right, sadly, but for totally different reasons).

Put differently, if exponential growth is a crucial element, then the Carter hypothesis would argue that either humanity will die, or it will cease exponential growth forever. But if we populate the stars, of course we will see the return of exponential growth. We will have to have exponential growth again at some point, or we will die out (in millions of years, let's say).

eburacum45
2005-Nov-09, 09:46 AM
If anything, Hanson's suggestion serves to undermine one of the counterarguments to Carter: that external evidence indicates species survive for many more million years than we have done so far, so we are more likely to be at the start of our career as a species. But our privileged viewpoint as a self-aware, mathematical, industrial civilization means that we are not like other species, and so their data cannot be applied to us.
I am pretty sure that we are not likely to show the same pattern as other species, so arguments from sharks are probably irrelevant. For one thing, we are likely to consciously take control of our own evolution before too many millenia have passed; this autoevolution hasn't happened in other species as far as I am aware. Whether that would signal the end of the human race and the collapse of the Homo sapiens population I wouldn't really know; on this page (warning-sci fi) (http://www.orionsarm.com/topics/Metasoft_Baseline_Reserves.html) I have suggested that the population of H. sap sap might continue to increase even when the rest of our civilisation has autoevolved into something else. So the future population curve may easily change into something that is unlike that of any other species.

A bunch of superintelligent aliens roaming the galaxy might stop at a mediaeval planet and say: "There's a 95% chance they'll all be dead before x more births." Then they see us, and say "There's a 95% chance they'll all be dead before y more births. Then they move on to a star-spanning civilization that has been aware of Carter's calculation for many millennia: "There's a 95% chance they'll all be dead before z more births." If Carter's maths apply, 5% of such predictions will be wrong, the other 95% will be correct.

If such an observer were to see the Palaeolithic human population, with its slow growth, then it would calculate that our species had a long future ahead; similarly an observer meeting the star-spanning culture would note the long, slow growth of population dictated by interstellar distances and calculate a long, if not indefinite future.
It is only today, when our population is growing very fast - almost but not quite exponentiating - that the calculations show a quick die-off. But this is the exact period when our culture has discovered the Bayesian interpretation of probability, so it is the only example of such a calculation we are aware of.

If we could talk to our far future descendants they would tell us that the period of time when the Carter-Leslie argument predicts a near-future doomsday is a brief and anomalous one. If we could also talk to other long-lived alien civilisations they are likely to tell us that they passed through this stage of rapid growth too, and many of them probably discovered statistical arguments of a similar nature during this same stage. By comparing all the values from all the calculations, from palaeolithic, (hypothetical) far-future and (hypothetical) extraterrestrial sources, we would see that almost all indicate long term survival, because they all show much slower growth than our own current situation.

Only the values calculated during the period of rapid industrialisation, an anomalous phase, show a near-future doomsday; exactly the same period when such statistical tools become available. Keep making the same prediction using the same tools once the world has become technologically advanced,and the population has stabilised, then the date of doomsday-with-95%-certainty recedes into a remote future once again.

grant hutchison
2005-Nov-09, 09:47 AM
I don't agree. The point is, anyone who makes the Carter argument will always come to the conclusion that the end is fairly near.That's right. And Carter says that some of them will be wrong, and most of them will be right, the exact ratio depending on the confidence interval chosen.
So the fact that people were wrong in the past doesn't make it any more or less likely that we are wrong now. They're independent coin tosses of a heavily weighted coin.

Grant Hutchison

Fram
2005-Nov-09, 10:20 AM
Look at it from another perspective, namely your own.
You are "ball number 7". When you are pulled out of the barrel, what is the chance that you will be number 7? Why, 100% of course. What is the chance you will be taken out of the barrel? Well, that chance will be much higher when your barrel has only 7 or 8 balls then when it has 100 billion, the Carter scenario says (paraphrasing of course). So probably you are in a small barrel (i.e. extinction will occur soon).
This is where the scenario has gone wrong, in my view. We don't know and have no reason to assume that only one ball is taken out of the barrel. If all balls are taken out, ours has to be taken out as well, and the fact that ball number 7 has been taken says nothing at all about the size of the barrel.

Let's look at it again, but from a completely different angle.
I am filling a barrel with numbered balls slowly and in order. One ball drops every second. After ten seconds, you pick a ball. It's number 7. What does this say about the total number of balls that will end up in the barrel?
The Carter scenario says that 95% certain, this will be only 20 or so. I guess it is plain for everyone to see that we have no information whatsoever to make such a statement. The only thing we know is that a) there will be at least 7 balls, as I have picked ball seven, and b) there will even be at least 10 balls, as so much seconds had passed (the observer can of course see the balls dropping). Nothing more.

Relating this to the original presentation of the scenario: you have two of those slowly filling barrels. One will stop at 100, the other at 10000. After ten seconds, you pull out ball 7. Does this help in any way to know which one will stop at 100 and which at 10000? No.

Madalone
2005-Nov-09, 10:52 AM
I see your point, but I don't quite agree...


Once the population curve becomes a level line leading off to infinity, we're just as likely to find ourselves anywhere under that line (because there's no bulge to "concentrate lives" in one temporal region), and so we can't infer anything about survival time in the future.
Either we have exponential growth, and Carter's right, and we're likely to be living in the End of Days; or we level out our population, and Carter can't make any deductions about the future - the areal midpoint under an infinite level curve is anywhere.

First, as the urn/ball example shows, exponential grow is not a necessary prerequisite for the basic argument; it just lets The End draw closer to the present point in time.

Second, the population curve is finite at least at one end (the past), so we have an asymmetry which makes a populated future "unlikely".

I just think trying to beat the Carter argument on its own premises is missing the point; my basic gripe ist still that the argument makes a statistical statement based on a sample size of precisely one...

But, heck, why am I arguing with you anyway? Probably you dont even exist! I know that Switzerland is roughly 200km across. The fact that I'm Swiss would be very, very unlikely if the World was seizably bigger than, say, 1000km. So, in all likelyhood, anything beyond the English Channel must be a figment of my Imagination! ;)

Madalone

grant hutchison
2005-Nov-09, 11:57 AM
This is where the scenario has gone wrong, in my view. We don't know and have no reason to assume that only one ball is taken out of the barrel.We know that only one ball is extracted, in this analogy. You live only once, and therefore you represent one ball.


One ball drops every second. After ten seconds, you pick a ball. It's number 7.You have no control over the time at which the ball is selected. You simply come into existence, and check your position in the birth order. If there are a very large number of balls, then you're unlikely to find yourself in the first few.

Grant Hutchison

grant hutchison
2005-Nov-09, 12:04 PM
First, as the urn/ball example shows, exponential grow is not a necessary prerequisite for the basic argument; it just lets The End draw closer to the present point in time.You're right. I was trying to explain the particular urgency in Baxter's scenario, and didn't make that properly clear.


But, heck, why am I arguing with you anyway? Probably you dont even exist! I know that Switzerland is roughly 200km across. The fact that I'm Swiss would be very, very unlikely if the World was seizably bigger than, say, 1000km.:lol: :lol:
But it's not a great argument against the Carter calculation, since you do have access to counter-information. Whereas we don't have any knowledge of "how big the world really is" with reference to the total number of humans that will ever exist. Carter's calculation is therefore probabilistic from limited information, and can't be undermined by analogy with other situations in which information is more complete.

Grant Hutchison

Fram
2005-Nov-09, 12:21 PM
We know that only one ball is extracted, in this analogy. You live only once, and therefore you represent one ball.

You have no control over the time at which the ball is selected. You simply come into existence, and check your position in the birth order. If there are a very large number of balls, then you're unlikely to find yourself in the first few.

Grant Hutchison

If you are number 7, then you are very likely to be in the first few :D This is a variation on the strong vs. weak anthropogenic principle. You have to make the distinction between an outsider who has the choice from all balls and picks out a low number, or a ball itself, which has a number, period. I am ball number 7, and that says nothing at all about the chances for survival of humanity in the next 100, 1000 or billion years. I can not pick any ball I want, and so the "choice" of a ball is no choice at all.

To expand my slowly filling barrel analogy to make it more resemble reality. The barrel is slowly filling and even more slowly emptying again. You are allowed one moment to pick one ball. If you get a low number, then this only proves that the barrel didn't exist very long yet. This says nothing at all about how long it will exist in the future.
It's like madalone says:

my basic gripe ist still that the argument makes a statistical statement based on a sample size of precisely one...

grant hutchison
2005-Nov-09, 12:39 PM
I can not pick any ball I want, and so the "choice" of a ball is no choice at all.Exactly. So it would be odd to find yourself among one of the first balls. The number you find yourself to be therefore does allow you to make a statistical inference about how many balls there are.

Grant Hutchison

eburacum45
2005-Nov-09, 12:50 PM
it would be odd to find yourself among one of the first balls.
I think (but I am not sure) that my argument can explain away that oddness. We are a special case, so not representative.

grant hutchison
2005-Nov-09, 12:58 PM
Only the values calculated during the period of rapid industrialisation, an anomalous phase, show a near-future doomsday; exactly the same period when such statistical tools become available.No argument from me. A population with no growth at all would judge that they are likely to survive into the future as long as they have existed in the past. But they would still see the same Carter limit to the number of likely future lives. I agree the urgency has gone, but Carter's argument isn't undone.

Grant Hutchison

Fram
2005-Nov-09, 12:59 PM
Exactly. So it would be odd to find yourself among one of the first balls. The number you find yourself to be therefore does allow you to make a statistical inference about how many balls there are.

Grant Hutchison

Yes, but it would be meaningless. It focuses on one point and ignores many others. It would be odd if I was the first human. It would be odd if I was the last human. It would be odd if I were the middlest (?) human. It would be odd if my 'birthnumber' would be exactly a prime number. Yet, someone has to be (or will be or has been) one of these. For a species to exist 100 years, a specimen has to exist after ten years. For a species to exist 100 billion years, a specimen has to exist after ten years. As long as we don't know what the average lifetime of a species (or an intelligent species, or whatever distinction you like to make) is, we can't make any useful statistics.

Using 'your' statistics, the best chance for a long future of humanity would be to kill all humans except a hundred or so, and keep humanity at that number. That would mean that if my number 9 billion is an average number (which wouldn't be that odd compared to it being an early number), we could have 90 million generations more after this one. Reduce it to a couple that has two children and so on, and you'll have 4.5 billion of those couples before homo sapiens gets extinct! I think you'll see that this shows that statistics can 'prove' anything and that the basic premisse (of the peculiarity of us living now unless we are doomed fairly soon) is flawed.

Ken G
2005-Nov-09, 01:56 PM
I am ball number 7

"I am not a number, I am a free man!"

Fram
2005-Nov-09, 02:17 PM
"I am not a number, I am a free man!"

"You are all individuals!"
"Yes, we are all individuals!"
"No, I'm not."

grant hutchison
2005-Nov-09, 02:31 PM
It would be odd if I was the first human. It would be odd if I was the last human. It would be odd if I were the middlest (?) human. It would be odd if my 'birthnumber' would be exactly a prime number.But it would not be odd if you were in the last 90% of humans ever to live. That's all Carter's talking about, not the specific number on your ball. And the population bulge of exponential growth crowds that last 90% into the last few years of humanity's existence.

Grant Hutchison

Ken G
2005-Nov-09, 02:51 PM
What we are basically arguing are the "rules of the game", not probability or logic per se. My original support of Madalone's was simply that it can't be logic if it leads even a minority to the wrong conclusion. Grant's point is that who said it had to be logic, it is merely an argument that leads most to the correct conclusion (that they are close to the end times, for exponential growth). So it's probability. Fram is saying that this assumes we could have come at any time, when in fact we are what we are, number 10 billion or whatever. It's part of who we are. It's a question of what is known. One could say that either it is the end times or it isn't, so without any information, it's a 50/50 chance!

Fram
2005-Nov-09, 02:57 PM
But it would not be odd if you were in the last 90% of humans ever to live. That's all Carter's talking about, not the specific number on your ball. And the population bulge of exponential growth crowds that last 90% into the last few years of humanity's existence.

Grant Hutchison

So my scenario for extending the lifetime of the human race by eliminating all but a few specimen should work?

Ken G
2005-Nov-09, 04:26 PM
Grant knows that killing most of humanity wouldn't help things, because you would know that you did it and therefore mucked with the probabilities. He is saying that if you did such a heinous act, and the survivors had no knowledge of it other than where they fell in the order of humans born, they would in fact conclude that humanity's "clock" has been extended. It's a little like playing poker. If your opponent has four of a kind, he/she is pretty sure they'll win and will bet in that expectation. But if you're sitting on a full house, you say, "bring it on". It's not until they see how you are betting that they begin to get an uncomfortable sensation.... The issue is, probability is not an absolute thing, except in quantum mechanics. Everywhere else, it is a matter of information. So the question really is, what information do we possess that can give us a better estimate of our "chances" than the Carter hypothesis, which is what I meant by, in the complete lack of all information, the odds are always 50/50, but that's a pretty meaningless way to inform your decisions! This is related to Fram's initial point that there may be better ways of estimating our longevity than using the "default" thinking of the Carter hypothesis.

grant hutchison
2005-Nov-09, 04:35 PM
So my scenario for extending the lifetime of the human race by eliminating all but a few specimen should work?Well, all the people you killed would already have lived, so that wouldn't reduce the total count.
So you might more humanely achieve the same effect by simply reducing the birthrate very dramatically.
Either way, you would neatly and satisfactorily have enacted the abrupt catastrophic fall in human population that Carter's reasoning predicts. Q.E.D. :)

Grant Hutchison

Ken G
2005-Nov-09, 07:26 PM
Either way, you would neatly and satisfactorily have enacted the abrupt catastrophic fall in human population that Carter's reasoning predicts. Q.E.D. :)

But you are in effect agreeing that the timetable for human extinction could be extended by reducing population levels. That's not a reasonable hypothesis in and of itself, although it may get support from completely unrelated issues like the rate at which we use resources, etc. Note that any argument of the latter type is what I mean by the poker game-- doing a better job of inferring probabilities than a "default" thinking mode (in short, if Carter really thinks that way, I'd love to play some high-stakes poker with him!).

Ken G
2005-Nov-09, 07:31 PM
Let me clarify this point, since it is crucial. If you don't know anything about gravity, you use probabilistic arguments to reason that the Earth's atmosphere should be spread evenly over the whole solar system and beyond. Then someone explains gravity, and suddenly you have it tightly stuck to the surface of our planet. A little information goes a long way when using probability arguments, and the Carter hypothesis is only valid in the absence of any information at all other than birth order. It would be very foolish to use so little information in making a probability argument, along the lines of the 50/50 argument I presented.

grant hutchison
2005-Nov-09, 10:12 PM
But you are in effect agreeing that the timetable for human extinction could be extended by reducing population levels.No I'm not. I'm teasing Fram for creating an illustrative case to undermine Carter which in fact would do Carter's work for him. But surely Fram's point was that this imagined situation is ridiculous?
You can't manipulate a priori probability this way. If an actuary tells you and five friends that, as a group, your various life expectancies sum to 180 years, you would not prolong your own life by killing your five friends. You'd just change the rules for the calculation, and the estimate would need to be redone.
In Carter's calculation, one of the vexed points is: Which lives count towards the total? When do we start counting entities and when do we stop counting entities? Some would argue that a near-extinction bottleneck merited the start of a new count, so that the survivors of the Framocaust would perhaps see themselves as having rather short future prospects. (Though they might argue that they were undoubtedly in the first 5% of the New Count humans, and so fend off Carter's calculations for a few generations.)

Grant Hutchison

Gullible Jones
2005-Nov-10, 12:36 AM
Sorry if I'm repeating anyone here but...

To put it simply, the argument behind the Carter Catastrophy is a logical fallacy because someone has to draw the short straw.

grant hutchison
2005-Nov-10, 01:02 AM
To put it simply, the argument behind the Carter Catastrophy is a logical fallacy because someone has to draw the short straw.I don't understand, sorry. Can you elaborate?

Grant Hutchison

Gullible Jones
2005-Nov-10, 01:36 AM
If there's a future human civilization, it has to have a history. Some of the total number of people born in its history would have to be born relatively early during that history - they'd "draw the short straw", as it were.

Ken G
2005-Nov-10, 03:37 AM
I agree that it is a logical fallacy, because logic has to be 100% correct. But I think Grant is right that depending on how the "game" is set up, there are situations where the Carter catastrophe analysis is correct-- the majority will be correct that the end is relatively near. So the question is still, do we have any additional knowledge that allows us to infer we are not a part of the "majority"? I would say, certainly, there's no need to argue from a standpoint of zero information, which is what the Carter hypothesis does. If anything, there is plenty of empirical evidence that we could be near the end times even if human population had been 10 trillion in the past! (And of course the Carter hypothesis would indicate that any time you have a population of 10 trillion, and it winnows down to 5 billion at some point, those living would conclude that the end times could not be anywhere near. Would that have been a reasonable conclusion for the dinosaurs?)

snarkophilus
2005-Nov-10, 03:41 AM
Imagine that two big urns are put in front of you, and you know that one of them contains ten balls and the other a million, but you are ignorant as to which is which. You know the balls in each urn are numbered 1, 2, 3, 4 ... etc. Now you take a ball at random from the left urn, and it is number 7. Clearly, this is a strong indication that that urn contains only ten balls. [...]


This isn't true. The fact that you chose ball number 7 doesn't say anything about which urn is which.

It is true that before you choose a ball, you are more likely to pick 7 if you pick from the urn with 10 balls. Reasoning the other way doesn't work, though.

Look at it this way: before you chose a ball, you had to choose an urn. At that point, you had a 50% chance of choosing the urn with 10 balls. Then you pull out ball #7. That gives you no new information about which urn you have selected! Sure, it is unlikely a priori that you would choose #7 from the million urn, but the fact is that you did pull out number 7. If you'd pulled out numbers 11-1000000, you'd have new information, but you did not. Your odds of having chosen the ten ball urn are still only 50%.

Ken G
2005-Nov-10, 03:50 AM
This isn't true. The fact that you chose ball number 7 doesn't say anything about which urn is which.
Answers like this always depend on the "rules". I'm reminded of the great Monte Hall puzzle, in which you are shown three doors, one of which is a grand prize and the other two are goats. After you choose one door, Monte reveals another door has a goat behind it. Then he asks you if you want to change your choice to the remaining door. Is there a reason to do it?

Depending on the "rules" governing what Monte does, the chance of the new door being the grand prize will either be 1/2 or 2/3. It's pointless to argue about which is correct until you know Monte's rules. (And given the way Monte normally works, the rules indicate the answer will be 2/3).

snarkophilus
2005-Nov-10, 04:07 AM
Answers like this always depend on the "rules". I'm reminded of the great Monte Hall puzzle, in which you are shown three doors, one of which is a grand prize and the other two are goats. After you choose one door, Monte reveals another door has a goat behind it. Then he asks you if you want to change your choice to the remaining door. Is there a reason to do it?

Depending on the "rules" governing what Monte does, the chance of the new door being the grand prize will either be 1/2 or 2/3. It's pointless to argue about which is correct until you know Monte's rules. (And given the way Monte normally works, the rules indicate the answer will be 2/3).

It's a similar sort of problem, yes, but not the same. In the Monty Hall problem, Monty has extra information that he indirectly gives you by ALWAYS opening a door without the grand prize. In the choosing balls from a tub problem, no such decision is made -- it's purely random. You don't get any extra information from having chosen the number 7. Your odds of having chosen 7 were small, sure... but once you've picked it, you have pruned your decision tree:

prior to selection:
Root (1.0)
---ten ball urn (0.5)
------1 (0.5 * 0.1)
------2 (0.5 * 0.1)
------3 (0.5 * 0.1)
------etc
---million ball urn (0.5)
------1 (0.5 * 10^-6)
------2 (0.5 * 10^-6)
------3 (0.5 * 10^-6)
------etc

post-selection:
Root (1.0)
---ten ball urn (0.5)
------7 (1.0)
---million ball urn (0.5)
------7 (1.0)

Actually, it might help to think about it this way. The problem is formulated in a deceptive way. They could pick any number from 1-10 and still make their point. What they could not do is pick 11-1 000 000. That means that the selection was not truly random, and given a random choice of numbers which would seem to make their point, there are ten options from urn #1 and ten options from urn #2. Even odds for each possibility, no matter which number is picked.

Or, this way: there is an urn with one ball, and an urn with 9 balls. You reach in and pull out a number... any number. The chance that it is a 1 is 20%. However, given that it is going to be a 1, the chance that it is a 1 is 100%, and you can flip a coin to determine which urn you're going to pull it from: it's equally likely that each urn is the one with the lone ball.

If you still doubt, try it! Put nine pennies and a dime on a table. Turn all the pennies but one to heads, and the dime and the other penny to tails. Now, mix them about with your eyes shut and poke a coin at random. If it's a heads coin, discard the measurement (you are not interested in selections that are not the 1 ball!), and if it's tails, record whether it's a dime or a penny. Except for the difference in coin sizes, you should end up with about 50% of each.

Ken G
2005-Nov-10, 04:10 AM
The issue is, if you set up the experiment as described, and repeated it millions of times, you'll find that the vast majority of the times that a 7 was chosen, the selection came from the small urn, if you have a 50/50 chance of choosing either urn. Of course, if you have a much greater chance of choosing the bigger urn, say proportional to its size, then you can have the 7 come equally often from either urn. It's all a question of the rules.

snarkophilus
2005-Nov-10, 04:27 AM
The issue is, if you set up the experiment as described, and repeated it millions of times, you'll find that the vast majority of the times that a 7 was chosen, the selection came from the small urn, if you have a 50/50 chance of choosing either urn.

That's not the same thing, though. Saying you perform the experiment and then check the value is not the same as fixing the value and then performing the experiment!

Ken G
2005-Nov-10, 04:55 AM
Saying you perform the experiment and then check the value is not the same as fixing the value and then performing the experiment!

Exactly. So the question is, which set of rules, of the two you contrast, is more appropriately applied to the situation of birth order of an intelligent species? Here's a way to set up the rules where the Carter hypothesis seems at first to be correct, in the absence of any other information. Imagine one morning you wake up and are visited by an alien time traveller. The time traveller tells you that he has observed the extinction of a million intelligent species on a million worlds. He amuses himself by picking lives completely at random, and paying them a visit, and he chose you. And he mentions, by the way, that of course by these rules, 90% of the time he is talking to someone whose species does not outnumber that individual's birth order by more than a factor of 10. Why would you not conclude that your chances of being in that group are 90%, if you know nothing that distinguishes the species in question? You may have a hard time sleeping that night as you ponder the Carter catastrophe.

But you wake up feeling better, because it has occurred to you that he did not tell you the 90% figure applied to the subgroup of beings who were in around the 10 billionth in birth order! And indeed, there's no way to know that the 90% figure would apply to that subgroup without more information. (Bringing us back to Fram's original point.) So you ask yourself, what more information do I have? And you begin to wonder if the 90% overall average goes up or down among the 10 billionth beings from species similar to your own....

snarkophilus
2005-Nov-10, 06:08 AM
Exactly. So the question is, which set of rules, of the two you contrast, is more appropriately applied to the situation of birth order of an intelligent species?

That's not the question, really. Well, perhaps it is. The answer is "the correct set of rules." In their argument, they make a false assumption about the probability of which urn is more likely...



Now, say Carter and Leslie, we should reason in the same way as we did with the urns.


...and then they use that false assumption. And that's the problem. You are reasoning with a false assumption, so you can't logically say anything about the correctness of your conclusion.

The truth in this case is that, reasoning in the same way that I do with urns (which I hope is correct), there is an equal chance of the urn containing any number of balls greater or equal to x, provided that I pulled x out of an urn. So the distribution is that there is 0 chance of the species going extinct before now (which makes sense!) because the ball did not come from an urn with fewer than x balls, and equal (and, as it turns out, infinitely small, if there is no upper bound to time/population) chances for any two times in the future, represented by any two urns with more than x balls.

Then, of course, even that reasoning is overly simplistic, because it doesn't apply any weighting to the urns based on social factors, when our star is due to blow up, science, etc. But it's the correct logical conclusion to an unbiased version of the urn example.

Ken G
2005-Nov-10, 06:50 AM
You have used a different set of rules to derive the "snarkophilus anti-catastrophe", namely, that humanity will never become extinct, because each total number of humans greater than our present birth number is equally likely, and there are a virtually infinite number of possibilities! Interesting. This seems as justifiable as the Carter catastrophe if yours are in fact the right rules, but really neither set of rules are very plausible, just as the "everything has a 50/50 chance, it either happens or it doesn't" is also not a very useful rule. What do you say to the alien time traveller argument, in relation to your picture of the right rules?

snarkophilus
2005-Nov-10, 10:12 AM
You have used a different set of rules to derive the "snarkophilus anti-catastrophe", namely, that humanity will never become extinct, because each total number of humans greater than our present birth number is equally likely, and there are a virtually infinite number of possibilities! Interesting. This seems as justifiable as the Carter catastrophe if yours are in fact the right rules, but really neither set of rules are very plausible, just as the "everything has a 50/50 chance, it either happens or it doesn't" is also not a very useful rule. What do you say to the alien time traveller argument, in relation to your picture of the right rules?

I'm not entirely sure what you mean by saying that there are different sets of rules. That idea came up in response to



"Saying you perform the experiment and then check the value is not the same as fixing the value and then performing the experiment!"


The question posed by Carter is, "given that you fix the result of the experiment beforehand, then perform the experiment and discover that the answer is 7, what can you say about which urn was selected?"

My answer is "nothing," because the experiment is fixed. The probabilities of having chosen each urn are the same as they were before the experiment. (I arbitrarily decided that there was a 50/50 chance of picking a particular urn. It doesn't have to be that way. That's partly why you get the "anti-catastrophe" scenario. Choosing a different distribution of urns gives a different set of probabilities.)

I suppose that another set of rules is not fixing the experiment, but performing it and seeing what happens. I've already explained what conclusion you can draw from that experiment, what you call the anti-catastrophe. And, as I mentioned, that conclusion isn't really valid in real life because there's no reason to think that each urn carries equal weight. But it's still better than the other version, because it is based upon sound logic.

To resolve the anti-catastrophe, do the following: pick a number. An integer. Any positive integer. Now, amongst the integers, the probability that you chose that number is essentially 0. (There is a discussion about surreal numbers kicking around here somewhere that is almost pertinent...) The fact is, however, that you chose an integer. That integer is finite, even though there are an infinite number of possible choices. And so it is with the anti-catastrophe. Even though you have an infinite number of choices of urns, in the end you have to pick one, and it has a finite number of balls in it.

There's still the possibility that the anti-catastrophe occurs, of course, but you can't decide how probable that is compared to an integral count (catastrophe after n people) with additional information.

*pause for breath*

And that leads in to the alien time traveller.



The time traveller tells you that he has observed the extinction of a million intelligent species on a million worlds. He amuses himself by picking lives completely at random, and paying them a visit, and he chose you. And he mentions, by the way, that of course by these rules, 90% of the time he is talking to someone whose species does not outnumber that individual's birth order by more than a factor of 10.


He has chosen a life at random from the species, so you have a 90% chance of being one of the last 90% of people to be alive. That makes perfect sense, and doesn't worry me in the slightest. That's because you have to add up all the possibilities. If there are only ever going to be 10 billion people, and I'm number 10 billion, that's bad. Same for if there are only going to be 11 billion. But what about if there are going to be a trillion? Two trillion? There are a lot of numbers up there, and in each of those scenarios, you're in the first 10%. It's increasingly improbable that he would choose you in particular as species number increases, but the fact is that he had to choose someone. In fact, if there's truly an equal chance of your species number being any number, your birth number is statistically so small (compared to infinity) that there's no point in worrying: you're most likely at 0%.

You need a realistic distribution as to the probabilities of each species number being correct to get any information out of your extraterrestrial visitor.

And, of course, there's the possibility that you'll never be extinct. He's witnessed a lot of extinctions, sure. But he didn't say how many species go on forever. You have no data whatsoever about that (except the laws of thermodynamics, but I digress).

Ken G
2005-Nov-10, 02:29 PM
You need a realistic distribution as to the probabilities of each species number being correct to get any information out of your extraterrestrial visitor.

This is what I am saying as well. The 90% number applies over the full sample, not necessarily the subsample of comparable beings (i.e., a subsample of species in comparable situations where the time traveller happened to choose the ten billionth being, or so). There might be a correlation between overall longevity and making it to ten billion at all, and there might be elements in place that make a particular species more or less vulnerable to extinction by the time they get to the 10 billionth being (which are we, do you suppose?). Thus the "rules" you need to make the Carter hypothesis work categorically is that the total number of beings before extinction must be a random variable that receives no inputs from anything about the species, and no correlations with making it to 10 billion in the first place (beyond the obvious). The rules to make the snarkophilus anti-catastrophe I'm not sure I can think of, but there probably are some that could be applied in the context of this thought experiment. But I would argue that in any event, neither of those rules are plausible. As soon as you look at anything that relates to extinction potential, the odds change dramatically from the Carter "default" argument. The real question is, which way?

Grey
2005-Nov-10, 03:51 PM
It also strikes me that the Carter hypothesis assumes that there must be a finite number of humans born a priori. In the version we're talking about here, it also assumes that there are only two options, the small urn and the big one. Even if we limited ourselves to a finite number of humans, why just these two choices? What happens to the odds if you have a billion urns, with a number of balls ranging from one to a billion? What if you have a trillion urns? Sure, by Bayesian reasoning, drawing number 7 means that the probability you drew it from the urn with 10 balls is greater than the probability you drew it from the urn with a million balls. But the overall probability that it was from a small urn goes steadily down as you increase the number of urns. And how do you know when to stop adding urns, unless you make an arbitrary decision ahead of time?

Ken G
2005-Nov-10, 04:49 PM
That speaks to limitations in the urn analogy, which is why I prefer the alien time traveller approach. Since the alien chose you at random, it would not seem to alter your situation vis a vis the validity of the Carter catastrophe.

grant hutchison
2005-Nov-10, 08:19 PM
If there's a future human civilization, it has to have a history. Some of the total number of people born in its history would have to be born relatively early during that history - they'd "draw the short straw", as it were.Ah, OK. This doesn't work, because (as Ken says) Carter's argument is probabilistic: it states that there's a 95% chance of extinction after x number of lives. Such a prediction accepts, indeed stipulates, that it will be wrong on 5% of occasions (ie, for those born early in human history).



Imagine that two big urns are put in front of you, and you know that one of them contains ten balls and the other a million, but you are ignorant as to which is which. You know the balls in each urn are numbered 1, 2, 3, 4 ... etc. Now you take a ball at random from the left urn, and it is number 7. Clearly, this is a strong indication that that urn contains only ten balls. [...]
This isn't true. The fact that you chose ball number 7 doesn't say anything about which urn is which.Unfortunately, this "illustration" seems to have been written by someone who doesn't understand Carter, or doesn't understand how to construct a good illustrative analogy: it's utterly misleading about how Carter's argument works.
There's no choice of urns. There's a single urn, which contains an unspecified quantity of consecutively numbered balls - anything from 10 to a million. We blindly select a ball, and find it's number 7. This is more likely to occur if the urn contains a small number of balls than a large number of balls. In fact, we can deduce a 95% confidence interval for how many balls the urn contains. Implicit in that calculation is the prediction that 5% of the time the urn will actually contain more balls than the upper limit of our confidence interval (assuming I've constructed a one-tailed confidence interval).

Grant Hutchison

grant hutchison
2005-Nov-10, 08:29 PM
It also strikes me that the Carter hypothesis assumes that there must be a finite number of humans born a priori.Yes. This was my point, early in the thread, when I pointed out that the possibility of infinite human lives blows Carter out of the water. Whether we draw the 7th ball or the googleplexth ball, we're equally unsurprised to find ourselves close to the start of human existence.
Freeman Dyson certainly did some work in which he suggested that the number of "processor cycles" possible in a finite, expanding, cooling Universe approached infinity as time went to infinity. This would suggest that there would be room for an infinite number of human consciousnesses (implemented in some suitable form left as an exercise for the student) in the future Universe.
A Big Rip scenario might well undermine his calculations, however: I don't know.

Grant Hutchison

Ken G
2005-Nov-10, 09:17 PM
Yes, I think that everything Grant has been arguing has been correct, except the basic assumption that every number of balls in the urn must be equally likely. Imagine instead that the urns are stuffed using an algorithm that makes it 1% probable that there are 10 balls, and 99% probably that there are 1000 balls. If you pick a 7, what do you conclude is now the probability that the urn contains 1000 balls? The relative probability for 10 balls is .01 times .1, or .001, compared to the relative probability for 1000 balls, which would be .99 times .001, or about .001. This is the same relative probability-- choosing a 7 in this case gives you a 50/50 chance the urn was the 1000 ball or the 10 ball variety! The key point is that to make a meaningful probability argument, you need to know something about how the probabilities are distributed. Probability is about information, it's not something absolute (except in quantum mechanics), and the assumptions you make about the things you don't know (the "rules") are everything.

If this is still unclear, realize that the Carter argument is made from a position of no information, outside of birth order. It is like a person who has just learned how to play chess, entering a chess tournament. This person has no knowledge of how chess tournaments work, other than that there will be 1000 competitors. They don't even know the level of anyone else in the tournament. But they figure, hey, out of 1000, I'll probably end up no worse than 900th place, with 90% confidence. But that's only true if all 1000 competitors also just learned, and are a random cross section from the same population. If instead, this tournament happens to be the World Championships, then it might be quite likely that the newcomer will place dead last, despite their probabilistic argument based on no information. When making a probability argument, the meaningfulness of the result depends on the reliability of the assumptions. Thus the Carter argument is correct as far as it goes, but is of very limited meaning, like the chess entrant's conclusion that he/she will not finish worse than 900th place.

ZaphodBeeblebrox
2005-Nov-10, 10:06 PM
The BIG Problem Is, How Do you Determine the Confidence Interval?

Without ANY Starting Information, It'd Be Impossible!!!!

:think:

Ken G
2005-Nov-10, 10:44 PM
Right. It's like the problems with the "Drake equation"-- what do these probabilities really mean when the numbers change every time a scientist sneezes?

grant hutchison
2005-Nov-10, 11:22 PM
The BIG Problem Is, How Do you Determine the Confidence Interval?A ball chosen at random has only a 5% chance of coming from the lowest-numbered 5% of balls, and a 95% chance of coming from the other, higher-numbered balls. We are therefore 95% certain that our ball, number 7, has a higher number than the lowest 5% of balls. So we are 95% certain that the lowest-numbered 5% consists of fewer than 7 balls. If there are only 6 or fewer balls in a 5% sample, then the total number of balls must be (6*20)=120 or fewer. When we draw ball number 7, we are therefore immediately 95% confident that there are 120 or fewer balls in the urn.

Grant Hutchison

grant hutchison
2005-Nov-10, 11:37 PM
Yes, I think that everything Grant has been arguing has been correct, except the basic assumption that every number of balls in the urn must be equally likely.Well, I don't think I've ever argued in favour of that - Carter just assumes it. To make my position clear, here, I've simply been trying to explain why it's rather harder to make Carter go away than some folk imagine - he's been vexatiously discussed for 20 years, after all, and that's not a common characteristic for an easily-dismissed logical error.

So you're saying that information external to Carter's assumption may constrain the lifetime of humanity in some way. By analogy, we might imagine a community of human liver cells trying to use Carter's reasoning to predict their community's future existence. Cells in a 7-year-old child would predict a relatively short future, while cells in a centenarian would predict a long future. Oops: the information they lack is that there is a characteristic lifespan for a human.
However, to use this argument to properly blow Carter away, you'd have to tell us which specific bit of information you have that gives you a more reliable estimate of humanity's future existence than Carter does.

Grant Hutchison

Gullible Jones
2005-Nov-10, 11:44 PM
Okay, I'll put things very simply and bluntly:

From a point in the history of any civilization, you can't predict with any accuracy how long that civilization will last. Period. For all we know, we could all be wiped out by a NEO in a few decades... Or we could be at the beginning of a civilization that survives billions of years into the future, and colonizes the whole galaxy. We don't know, and furthermore cannot find out until something actually happens. So y'all quit yer catastrophizing and do something useful, and we might actually have a chance of surviving whatever comes up!

grant hutchison
2005-Nov-11, 12:25 AM
Okay, I'll put things very simply and bluntly:

From a point in the history of any civilization, you can't predict with any accuracy how long that civilization will last. Period.Ah, the Argument from Repeated Protestations of Increasing Loudness. One of my favourites. :)

Sorry. Couldn't resist. Just teasing.

Seriously: Carter's argument would say that your NEO is just the sort of catastrophe needed to explain our apparent location near the End of Days. And that the billion-year civilization is of course possible ... just unlikely, given how close to the start of it all we find ourselves.

Grant Hutchison

snarkophilus
2005-Nov-11, 12:57 AM
There's no choice of urns. There's a single urn, which contains an unspecified quantity of consecutively numbered balls - anything from 10 to a million. We blindly select a ball, and find it's number 7. This is more likely to occur if the urn contains a small number of balls than a large number of balls.

No, because you don't know the size of the single urn. In order to properly predict which urn you have (which is the point of the whole exercise) you need to consider all possibilities for urns.

In the case of 7, there's a 1/7 chance of picking the 7 ball if n=7, 1/8 if n=8, 1/9 if n=9, etc. Add up all those possibilities, as n tends to infinity. It's a divergent series, which means your 1/7 chance is singularly unlikely: you will never be the last person in your species. The only ways to resolve this are to change the series by weighting the odds to get a convergent series or to accept that the species will never die.

grant hutchison
2005-Nov-11, 01:18 AM
In the case of 7, there's a 1/7 chance of picking the 7 ball if n=7, 1/8 if n=8, 1/9 if n=9, etc. Add up all those possibilities, as n tends to infinity.But why are you adding up these probabilities? What does this sum measure? By summing probabilities, you're producing a probability greater than one - what does that mean?

Grant Hutchison

Major Tom
2005-Nov-11, 02:39 AM
Ah, the Argument from Repeated Protestations of Increasing Loudness. One of my favourites. :)

Sorry. Couldn't resist. Just teasing.

Seriously: Carter's argument would say that your NEO is just the sort of catastrophe needed to explain our apparent location near the End of Days. And that the billion-year civilization is of course possible ... just unlikely, given how close to the start of it all we find ourselves.

Grant Hutchison

So a NEO or other global disasters would be more unlikely if fewer people had lived? Personally Im more confident about humanitys chances of a continued long existence with six billion individuals than if there were only, say, a hundred thousand. The more people, the more likely someone will pull through a major disaster and be able to continue reproducing.

Sam5
2005-Nov-11, 03:01 AM
In Baxter's book, one character calculates based on this argument that Doomsday is 150, maximum 200 years away.

Deep in my gut I have the feeling that this argument is fundamentally flawed (read: rubbish), but I can't come up with a clean and neat rebuttal. The best I can come up with: As the human population has grown roughly exponentially in the known past, exactly the same argument has been valid for each and every generation before us - for instance also for the generation that lived, say, 300 years ago. And yet we are here, 300 years after an imminent doom.

What are your takes on this argument?


Hmm, I don’t get it. Maybe I’m just too stupid to understand the puzzle, but it seems to me that based on the basic premise of the story, every generation would be just as much “near the end”, if we think that “this” generation is “near the end.” Or to put it another way, every generation would have just as much a chance of being “near the end” as any other.

Why is there any reason to think that we are more “near the end” than Europeans in the middle-ages who were dying by the millions of the plague, or Africans 50,000 years ago who lost whole tribes during gigantic floods, earthquakes, tsunamis, and massive volcanic eruptions?

Seems to me that humans would have been closer to becoming extinct half a million years ago, when there were so few of them and they had no medical protection from diseases. It wouldn’t have taken much to wipe them all out, just as thousands of other species were wiped out in the past. As a matter of fact, maybe a really superior species of being -- above humans -- did exist on earth for a while, but did die out because of some natural disaster, and we are merely the lowly intellectually-inferior survivors.

snarkophilus
2005-Nov-11, 10:26 AM
But why are you adding up these probabilities? What does this sum measure? By summing probabilities, you're producing a probability greater than one - what does that mean?

Grant Hutchison

The problem ultimately reduces to this: given a chosen ball X, find the probability of each urn with n balls being the one that represents the universe. If you had a limited number of urns, you'd add up the probabilities and multiply by a normalization factor to account for the fact that at least one urn must be chosen. Even with an infinite number of urns, if you have a finite sum, you can still normalize with a finite factor. However, because the series is divergent as the upper bound of n tends to infinity, the probability of each urn being chosen is essentially zero.

Carter claims that a 10 ball urn is more likely than a million ball urn. That may be true, but it is not more likely than an urn with between a million and fifty million balls. And so it goes. You can always find a range of urns that is more likely than the last. And that's where his error lies.

snarkophilus
2005-Nov-11, 10:36 AM
Hmm, I don’t get it. Maybe I’m just too stupid to understand the puzzle, but it seems to me that based on the basic premise of the story, every generation would be just as much “near the end”, if we think that “this” generation is “near the end.” Or to put it another way, every generation would have just as much a chance of being “near the end” as any other.


It's not that you're stupid. It's that you have good intuition. :) It's pretty clear that the conclusion can not be correct. The reason is that the premise of the puzzle is flawed, so the conclusion is flawed. You've stated in words what I've stated mathematically above.

grant hutchison
2005-Nov-11, 12:00 PM
Carter claims that a 10 ball urn is more likely than a million ball urn. That may be true, but it is not more likely than an urn with between a million and fifty million balls. And so it goes. You can always find a range of urns that is more likely than the last. And that's where his error lies.
Two problems:
1) If your purpose is to find a 7, then of course sampling from a large number of large populations will give you more chance than sampling from a single small population. But that's irrelevant to our problem: it's one ball, and one bite at the cherry, whereas your summing of independent probabilities implies that you're sitting with a big range of urns, pulling a ball out of each, and counting success if any of those balls is a 7.
2) Carter's not interested in the unlikelihood of pulling a particular number, but in the unlikelihood of our selected number (whatever it is) coming from the lowest 5% of the numbered population.
I've already rehearsed Carter's calculation earlier in the thread:
A ball chosen at random has only a 5% chance of coming from the lowest-numbered 5% of balls, and a 95% chance of coming from the other, higher-numbered balls. We are therefore 95% certain that our ball, number 7, has a higher number than the lowest 5% of balls. So we are 95% certain that the lowest-numbered 5% consists of fewer than 7 balls. If there are only 6 or fewer balls in a 5% sample, then the total number of balls must be (6*20)=120 or fewer. When we draw ball number 7, we are therefore immediately 95% confident that there are 120 or fewer balls in the urn.

Grant Hutchison

Edit: Slight expansion for clarity.

grant hutchison
2005-Nov-11, 12:18 PM
... it seems to me that based on the basic premise of the story, every generation would be just as much “near the end”, if we think that “this” generation is “near the end.” Or to put it another way, every generation would have just as much a chance of being “near the end” as any other.
That's exactly right. If the population curve were exponential from the very start, each generation would assume, with 95% certainty, that they were close to the Carter Catastrophe. That level of certainty implies that 5% of those who made that deduction would be wrong, and 95% would be right. But because of the exponential growth in population, the first 5% (the ones who're wrong) are strung out over most of history, and the last 95% crowd into the last few generations before the extinction.
So (like everyone else who's ever lived) we can use Carter's argument to deduce with 95% certainty that we're close to the End of Days. But there's a 5% chance that we're wrong, and in fact ahead of us is a vast bulge containing more than 95% of all the people who'll ever live. They'll be the 95% who're right, and we'll be in the 5% who're wrong, along with everyone else in history so far.

Grant Hutchison

grant hutchison
2005-Nov-11, 03:17 PM
So a NEO or other global disasters would be more unlikely if fewer people had lived?
Sorry, I missed this one on the first run through.
Not at all. If a small population has existed for a small period of time, only a few people have ever lived, and Carter's reasoning predicts that that population is likely to die out after only a few more people have lived. A "small disaster" might be sufficient to achieve that, given that the population is small. If a large population has survived for a very long time, then Carter predicts a large number of future lives before that population dies out. Probably a "large disaster" would be required. Since small disasters are more frequent than large disasters, this is all internally consistent: frequent small disaster eliminating small, short lived populations, and occasional large disasters wiping out the infrequent large populations who've been lucky enough to slip through the "small disaster" winnowing. (There's certainly genetic evidence that humans have suffered a near-extinction event when our population was small: we're genetically much more similar than we should be, given the length of time we've been around on the planet as a species.)

I think the exponential growth of the human population is what makes Carter's argument counterintuitive to some folk. If the human population were stable, then Carter's reasoning would reduce to:
"Hey, we've been around for a couple of million years without being wiped out. Chances are we can survive another couple of million. It's a 50:50 chance whether we last for a longer or shorter time than that."
Probably most people would find this an unexceptional bit of informal reasoning.
But Carter's calculation applies to human lives, rather than elapsed time, and if the distribution of human lives is skewed rightwards along the time axis, giving us a long pastward tail and a short futureward tail, people begin to become uneasy.

Well, it's been fun. I'm about to disappear into the Scottish mountains for a few days. I'll try to pick up this thread when I get back (assuming we're all still here). :)

Grant Hutchison

eburacum45
2005-Nov-11, 06:54 PM
I still think my argument kills Carter; in order for a statistical argument to apply, the sample has to be taken at random; but we are not a random sample, and so statistical arguments cannot apply. We exist now, when an apparent exponential growth is occurring in the population; we did not exist in the Palaeolithic, when slow growth would indicate a distant doomsday, and we do not exist in the future, when a slowly rising interplanetary or interstellar population would indicate a distant doomsday.

We can only sample the population now, when the rapid rise in population seems to indicate a near-future doomsday, because we have only just developed the right statistical tools.
We did not have the statistical tools in the Palaolithic, so we were not concerned about Carter (and it would have indicated a distant doomsday back then anyway) and we do not yet exist in the future.
If we did exist in the future we would no longer be worried about Carter, because the population would be growing slowly again and this would extend the Doomsday ever further into the far future; additionally we would have a historical record of the period long ago when the Carter argument was first discovered, and would know that it loses its accuracy during periods of rapid growth.
In most cases Carter's Argument predicts a long, long existence for the human species; only now for a relatively brief and anomalous period, does the argument seem to indicate a near-future doomsday.

And what do you know? That is the exact same period that we discover the argument itself! This is not a coincidence; we discovered the benefits of civilisation, of science, mathematics and statistics all within a short period - this explosion in technology and knowledge has produced an explosion in population numbers, and also has produced the Carter-Leslie argument.


To recap; this statistical method requires that we are chosen from random from all humans that ever lived. That is not the case- we are self-selected and could not exist at any other period of time, not in the past or in the future.

Fram
2005-Nov-11, 10:01 PM
That's exactly right. If the population curve were exponential from the very start, each generation would assume, with 95% certainty, that they were close to the Carter Catastrophe. That level of certainty implies that 5% of those who made that deduction would be wrong, and 95% would be right. But because of the exponential growth in population, the first 5% (the ones who're wrong) are strung out over most of history, and the last 95% crowd into the last few generations before the extinction.
So (like everyone else who's ever lived) we can use Carter's argument to deduce with 95% certainty that we're close to the End of Days. But there's a 5% chance that we're wrong, and in fact ahead of us is a vast bulge containing more than 95% of all the people who'll ever live. They'll be the 95% who're right, and we'll be in the 5% who're wrong, along with everyone else in history so far.

Grant Hutchison

And what is the chance that every generation of humans belonged to that 5 percent instead of the 95 percent? Well, that must be way smaller than the 5 % chance that we will be a long living species.
To say it in another way: the chance that we will be a long existing species is much bigger than the chance that we have made it thus far yet.
Just to put all this statistic spielerei into perspective...

Ken G
2005-Nov-12, 10:34 PM
Folks, there's been a lot said on this thread to dispute Grant's claims, but you've been missing the target (myself included, early on) because all Grant is saying is that only 5% of any species will live in the first 5% of birth order, and it's unlikely that we are in that 5%. That is completely correct in the absence of any other information. The mistake is simply that this 5% probability is being treated as some kind of absolute probability, like the chance a coin will flip "heads". But it isn't, it's a conditional probability, conditional on your knowledge. Here's an example of the difference. If you sit down to a game of flipping coins, you have a 50% chance of winning, no matter who you are. That's absolute probability. But if you sit down to a game of chess, you can't say, there are 2 players so I have a 50% chance. That's true, however, in the complete absence of any other information! But if you have any information at all, the chances change. If, for example, you know that your opponent is the World Champion, and you just learned how to play yesterday, obviously your chances are less than 1 in a million. But if you also know that your opponent wants to lose because he or she is very generous, then your odds could reverse. It's a conditional probability, it's all about what you know. So the real question that should be debated is, do we really have no other knowledge, in which case Carter is the best we can do no matter how many "urn analogies" we trot out, or do we have knowledge that completely changes the probabilities? This kind of amounts to saying, are you an optimist or a pessimist? But the optimists can certainly take comfort in the fact that the Carter argument is not an absolute probability.

grant hutchison
2005-Nov-14, 07:20 PM
... in order for a statistical argument to apply, the sample has to be taken at random; but we are not a random sample, and so statistical arguments cannot apply.I think there's a problem with this. None of the components of the "privileged viewpoint" you invoke have any influence on Carter's argument: it therefore doesn't matter whether or not sampling is random with regard to these components.
1) Carter's argument applies whatever the shape of the population curve (flat, exponential, declining, bell-shaped).
2) Carter's argument applies whether or not the population under consideration knows about it (we could apply it to a population of lemmings, for instance).
3) Carter's argument applies whether or not the population under consideration is technological.

You may well be correct that it's no accident we find ourselves understanding Carter at a time when population growth turns his prediction into a "Catastrophe". But Carter's argument is merely a statistical observation, and it is no less true during periods of of stable population, when it predicts a long future life-span for the population.
So I think you're indicating why the "Catastrophe" aspect maybe shouldn't surprise us, but I don't think you're undermining Carter's reasoning.

Grant Hutchison

grant hutchison
2005-Nov-14, 07:22 PM
And what is the chance that every generation of humans belonged to that 5 percent instead of the 95 percent? Well, that must be way smaller than the 5 % chance that we will be a long living species.No, the low probabilities don't multiply as you're suggesting.
Each new generation has a new piece of information (their own existence) and therefore redoes the calculation, coming up with a new 5% and 95% confidence interval.

Grant Hutchison

Fram
2005-Nov-14, 08:57 PM
No, the low probabilities don't multiply as you're suggesting.
Each new generation has a new piece of information (their own existence) and therefore redoes the calculation, coming up with a new 5% and 95% confidence interval.

Grant Hutchison

But for the Carter catastrophe to be correct, you could take an imaginary 1 million species to start with. After 1 generation, only X (let's say 100,000) would still exist. Of those, 20,000 gets to a third generation, and perhaps 5,000 to a fourth one, and so on. So the chance that we have reached here is of course 100%, but the chance that a first generation species has survived as long as we did is 1 in 1,000,000 (random number, again). So it is much more amazing that we exist, according to the same hypothesis, than it is to suppose that we will exist for a very long time afterwards...


Another criticism / point of view. An alien visits the earth, and sees just one generation of people. The chance that it sees us is bigger if only a few more generations will survive than if we will live for millioŕns of generations (thus far, standard Carter). Now take the opposite view. An alien comes to the earth, and sees another generation than us. The chance of this happening is much bigger the more generations there are, and much smaller if only a few generations are to come. So when you take the starting position that the alien has picked another generation (and this one is equally valid as the supposition that he has picked us), the chance of a long survival is actually bigger than the chance of a short survival.
A nice statistical paradox!

Ken G
2005-Nov-14, 09:33 PM
There is no point in arguing that the Carter hypothesis is wrong, unless you are bringing in additional information that bears on whether we are more or less likely to face extinction than purely random chance with no information at all. The only criticism of the Carter thinking that makes any sense, to me anyway, is the criticism that it draws on so little information that its conclusions are meaningless. It's like betting a poker hand before anyone else has bet, and you haven't looked at your cards.

grant hutchison
2005-Nov-14, 11:26 PM
But for the Carter catastrophe to be correct, you could take an imaginary 1 million species to start with. After 1 generation, only X (let's say 100,000) would still exist. Of those, 20,000 gets to a third generation, and perhaps 5,000 to a fourth one, and so on. So the chance that we have reached here is of course 100%, but the chance that a first generation species has survived as long as we did is 1 in 1,000,000 (random number, again). So it is much more amazing that we exist, according to the same hypothesis, than it is to suppose that we will exist for a very long time afterwards...Carter's calculation necessarily applies to a single species, the species of the observer. Otherwise the observer would not be a random sample from all possible individuals in the population.


Another criticism / point of view. An alien visits the earth, and sees just one generation of people. The chance that it sees us is bigger if only a few more generations will survive than if we will live for millioàns of generations (thus far, standard Carter).Carter doesn't say anything about this, and it doesn't seem to make sense, probabilistically.
Carter says only that an alien who did encounter us might be able to make some prediction about how long we will survive in the future, based on how many individuals of our species have lived so far. An alien passing at random is always more likely to encounter a long-lived species than a short-lived one, and to encounter a species somewhere in the middle 90% of its existence rather than at its start or finish.

Note: I say "might be able to make some prediction", because for the alien's random sampling in time to strictly correspond to Carter's random sampling in lives, the population would have to be stable across time, with the same number of individuals living at any given moment.
I've realized I didn't think this through properly the first time, when I introduced my visiting alien in discussion with eburacum. The same proviso would pertain when I suggested we might be able to apply Carter's reasoning to lemmings ... lemmings were a very bad example indeed, since their population is notoriously unstable.

Grant Hutchison

Ken G
2005-Nov-15, 12:59 AM
The lemmings problem can be rectified by using the Carter reasoning on the number that will likely yet live, rather than on the time the species will remain. The time always requires further reasoning, but the number comes right from the probability analysis. But I reiterate, knowing what we know about species here on Earth, it would be silly to not try and do better than the estimate that comes from the Carter reasoning from zero information (which goes back to Fram's initial post).

grant hutchison
2005-Nov-15, 01:20 AM
The lemmings problem can be rectified by using the Carter reasoning on the number that will likely yet live, rather than on the time the species will remain.I don't think so. Carter requires that any life is as likely as any other life to be sampled. Since we take a random snapshot of our lemmings in time, we're as likely to sample a bulge with lots of lemmings as a trough with few lemmings; but a random lemming life is more likely to sample a bulge than a trough. So we can't sample the population in a "lemming-life" way without knowing how it's distributed in time - but if we know that, we don't need Carter.

But I reiterate, knowing what we know about species here on Earth, it would be silly to not try and do better than the estimate that comes from the Carter reasoning from zero information (which goes back to Fram's initial post).I completely agree that Carter is best undermined by examining his simple assumption, rather than his simple reasoning. Simple assumptions are very often wrong, whereas simple reasoning is very often right. Nevertheless, it's my experience that people are often convinced there's a flaw in his reasoning, if they just pick away at it for long enough.

Grant Hutchison

Ken G
2005-Nov-15, 01:50 AM
Carter requires that any life is as likely as any other life to be sampled. Since we take a random snapshot of our lemmings in time, we're as likely to sample a bulge with lots of lemmings as a trough with few lemmings; but a random lemming life is more likely to sample a bulge than a trough.

Ironic that I now find myself to be the Defender of Carter! The point is, it makes no difference what the current population is, only the total birth number. If you think there might be correlations between extinction and present population, join the crowd. My point is, to use information of any type (outside of birth number) will break you from the Carter zero-information analysis, it makes no difference what the information is.



I completely agree that Carter is best undermined by examining his simple assumption, rather than his simple reasoning.
Yes, it is the assumption that we have no information to go on, when in fact it's not hard to find something that could correlate with extinction. The problem is, I can't help wondering if Carter's assumption is optimistic!

grant hutchison
2005-Nov-15, 10:49 AM
Ironic that I now find myself to be the Defender of Carter! The point is, it makes no difference what the current population is, only the total birth number.Ah, but if we sample at a random point in time with respect to an exponentially growing population (like lemmings or humans), then we are more than 5% likely to sample from the initial 5% of lives, simply because those lives are spread over a long time period.
We therefore can't move on to state that the birth number we find on sampling has only a 5% chance of existing in the first 5% of lives, and we've crippled Carter's reasoning. In fact, if we sample at a random time, we can't deduce a 95% confidence interval without knowledge of the shape of the population curve ... and if we have that, Carter is superfluous.

The elegance of Carter's insight is that sampling a random life (simply by living), allows you to forget the shape of the curve and derive a confidence interval with no other information but your birth number. Random sampling in time will only give equivalent information if lives are uniformly distributed along the time axis.

Grant Hutchison

Ken G
2005-Nov-15, 11:50 AM
Ah, but if we sample at a random point in time with respect to an exponentially growing population (like lemmings or humans), then we are more than 5% likely to sample from the initial 5% of lives, simply because those lives are spread over a long time period.

But why are you discussing sampling in time? We are sampling a life, under the assumption that all lives are equally likely to be sampled. Other constructions are certainly possible, such as, always selecting the first life! But I agree with your main point, that the Carter approach assumes the life we lead is a randomly sampled life from all of human lives, equally weighted. Again, that is what you would assume in the lack of all information about what makes you, you.

grant hutchison
2005-Nov-15, 04:46 PM
But why are you discussing sampling in time?Because I introduced the topic of external sampling earlier in the thread (of ourselves by aliens, of lemmings by humans). In both cases these samples are necessarily randomized with respect to time, rather than lives. (Because we can't choose a life at random as observers outside the population unless we have a knowledge of the population profile against time, in which case we don't need Carter, because we have external information.)
Given that I'd inadvertently introduced a flawed example, I felt obliged to point out it was flawed and to explain the flaw. I can't get sniffy about other people's rotten explanatory analogies while allowing my own to persist!

Grant Hutchison

Ken G
2005-Nov-15, 06:57 PM
I see. Yes, I think a lot of the problems in this thread involved invoking analogies to help understand the point, but the point is so delicate that analogies are not likely to work! I think it's more important to get away from the informationless Carter argument and into areas where we can actually assess, and possibly alter, our survival chances. For example, I'd say that natural catastrophe is no longer a serious issue, as it would probably take a major extinction event to wipe out humanity, and those happen on timescales of tens of millions of years. But humanity's technological development is happening on timescales that are up to a million times shorter! So it is clear that our survival depends entirely on our responsible use of our own technological advancements. No timescales embedded in human history are relevant at all any more, nor is the Carter argument, because in point of fact our fate is in our own hands now.

astromark
2005-Nov-15, 07:26 PM
Yes ridiculous. ,and dangerous. To sagest the domsday of humanity becouse the probabilaty points to it is a nonsence. Its likened to a religiouse balief. Based on speculative prediction, and probabilaty calculations. I dont see any science here.

grant hutchison
2005-Nov-15, 11:57 PM
Yes, I think a lot of the problems in this thread involved invoking analogies to help understand the point, but the point is so delicate that analogies are not likely to work!Fortunately I wasn't attempting to construct an argument from analogy - just trying to think of some ways Carter's reasoning could be applied to a population that didn't understand Carter.

Grant Hutchison

Ken G
2005-Nov-16, 01:33 AM
I think I see what you're saying, you're saying that if you wanted to try and apply Carter to other species, perhaps go out and take a census of all the living species on Earth for example, right now, then you would not be able to conclude that only 5% of the species were in their first 5% in birth order. In your example, you say what if all the populations have exponential growth prior to total extinction, then the vast majority of the species you would encounter would be early in their growth because we are sampling at a random time. The only time you can know the population growth behavior but still apply the Carter conjecture is if you are applying it to your own species, such that you are a randomly chosen life no matter what information you have about what the distribution is doing. I think you are right about that.

Here is another wrinkle though, if we want to restrict ourselves to the thread that we have no useful information about our survival so we may as well apply Carter, probabilistically. To what extent can we count ourselves as a random sampling from all of humanity? Could our genes come at any point along the way? Indeed, what if humans start doing genetic engineering on the genome, such that you or I would be impossible 10 billion humans from now? So the Carter conjecture might not hold simply for extinction, in the case of intelligent life it may only hold for the time it will take to alter the genome such that you are I are no longer a randomly chosen life over all humanity. Put differently, there has to be some criterion for constraining what the selection is occuring over. If we can count future human progeny that is even minutely different from you and I, then where do we draw the line? How much similarity to us is required for it to count in the Carter selection process?

And here's yet another wrinkle. If the elegance of the Carter conjecture involves the sentience of the being doing the reasoning, which seems to be a necessary part of "selecting a life", then did you have to be a man to count? In other words, was your life selected from the lives of all humans, or just human men? This speaks to the question, could you really have been any human, one life selected at random, or did you have to be exactly you, a single individual who won an unbelievably unlikely lottery to even be here at all?

grant hutchison
2005-Nov-16, 10:56 AM
The only time you can know the population growth behavior but still apply the Carter conjecture is if you are applying it to your own species, such that you are a randomly chosen life no matter what information you have about what the distribution is doing.Yep, that's what I was saying. But here's an interesting paradox that seems to arise from that.
1) Suppose a visiting alien had observed humanity at the time of Aristotle. Seeing in us the potential for exponential growth, the alien realises that he cannot use Carter's reasoning to predict our extinction, because his visit is random in time, rather than in lives.
2) The alien explains Carter's reasoning to Aristotle. By the same token, Aristotle is not entitled to use it to predict human extinction, because the alien has delivered knowledge at a random time, rather than to a random life.
3) But suppose Aristotle had come up with Carter's reasoning for himself, without alien intervention? He would then assume himself to be a random sample from human lives, and could reason according to Carter.

It's the difference between 2) and 3) I find paradoxical, and it seems like there is some germ in there related to eburacum's earlier "privileged viewpoint" discussion.


So the Carter conjecture might not hold simply for extinction, in the case of intelligent life it may only hold for the time it will take to alter the genome such that you are I are no longer a randomly chosen life over all humanity?Yes, there's a lot of discussion about when you start counting, when you stop counting, and what you count.
Does the "Carter Catastrophe" merely mark a transition to some trans-human condition, as you suggest?
Do we count other species of early human, or do we start from what seems to be some sort of extinction bottleneck in early homo sap existence? I can't now remember exactly how Baxter came up with the very tight 150-200 years cut-off quoted at the start of this thread, but I seem to recall he used some argument to eliminate many humans from the count: perhaps using only post-industrial humans, or those who have existed since Carter's argument was first proposed.

Grant Hutchison

Ken G
2005-Nov-16, 09:56 PM
I think I have the answer that will clear a lot of this up. As we've discussed, the Carter conjecture requires using no information, or the probabilities could change. But you do have to use one bit of information to make it fly-- you have to know what your birth number is, say 10 billionth. But beware, as soon as you input any information it will change the odds, even if you are not sure how. Ultimately, the validity of the Carter argument therefore relies on an unjustifiable assumption about how the total population numbers of intelligent species are distributed.

Let me clarify. Lets assume we have an immortal alien, keeping a census on all intelligent populations in our galaxy, from the beginning. No matter how the populations are distributed, only 5% of those beings will live in the first 5% of their populations. But is this still true of the subclass of 10 billionth born? Very likely not! For example, what if the total number of an intelligent species is a random number evenly distributed from 1 to a trillion. Then it is clear that if we restrict to the subclass of 10-billionth borns, we have populations that range evenly from 10 billion to a trillion. Hence in that model, the 10 billionth born has about an 80% chance of being in the first 5%! The answer depends on the population distributions. So the Carter conjecture is internally inconsistent-- you are not allowed to use both your birth number and the 5% chance of being in the first 5% in the same calculation! The only time this would be valid is if the population distribution worked out such that 10 billionth borns really did work out to be in the first 5% of their populations in 5% of all species, which would require an amazing coincidence and we certainly have no reason to expect it.

So here's what I'm saying. It is correct (modulo all the other difficulties about counting) to say "as a randomly chosen life, I have only a 5% chance of being in the first 5% of my species". What you can not say is "as the 10 billionth born, this means there's only a 5% chance our species will outnumber 200 billion". You can't use the number from the second sentence in concert with the number in the first sentence, because you would definitely expect the information in the second sentence to screw up the truth of the first sentence, even if you don't know how!

grant hutchison
2005-Nov-17, 01:10 PM
Isn't this a variety of the sort of "external information" we've agreed would undermine Carter if it were reliable?
Carter says nothing about how a number of different populations will behave, only about your own population.
In your scenario, each of the 10-billionth souls would conclude, according to Carter, that there was only a 5% chance they were living in the first 5% of their population.
It would turn out that 80% of them were wrong. In their own populations, that simply means that they are in the 5% pastward tail, and that the 95% of their population who come after them are correct in using Carter's reasoning.
Carter would therefore still be entirely correct and consistent within each population, although the 10-billionth souls could have made a better estimate if they'd been party to the external information your immortal alien has collected about population numbers.

Grant Hutchison

Damburger
2005-Nov-17, 05:50 PM
Most of us wouldn't have come up with the Doomsday Argument.

Therefore, we are not a random selection of the human race. We are part of the subset of the human race that existed after the Doomsday Argument was raised.

Thus the initial assumption of the argument is incorrect.

grant hutchison
2005-Nov-17, 08:37 PM
Most of us wouldn't have come up with the Doomsday Argument.Trouble is, Carter's argument applies to you whether or not you know about it.
So the non-random distribution of "Doomsday knowledge" doesn't seem like it should have any more of an effect on Carter's "random life" stipulation than the non-random distribution of, say, knowledge of the rules of cricket.

Grant Hutchison

snarkophilus
2005-Nov-17, 09:47 PM
Trouble is, Carter's argument applies to you whether or not you know about it.
So the non-random distribution of "Doomsday knowledge" doesn't seem like it should have any more of an effect on Carter's "random life" stipulation than the non-random distribution of, say, knowledge of the rules of cricket.


The trouble is that the argument doesn't apply at all.

Look at it this way: if you can end up with a paradox, then you didn't state the rules of the problem correctly, or you've made a bad assumption. The world is consistent. So regardless of whether or not you accept that a specific criticism of Carter's idea is valid, the fact that you are coming up with paradoxes should lead you to believe that the idea is flawed somewhere.

Ken G
2005-Nov-17, 10:00 PM
Isn't this a variety of the sort of "external information" we've agreed would undermine Carter if it were reliable?

No, it's not external information, it's internal information. The point is, you cannot use the number 10 billion anywhere in your longevity calculation, and also say that there is a 5% chance you are in the first 5% of your population. The distribution I gave as an example proves this. There has to be some probability distribution, it's not external information until you specify it. But no matter what it is, Carter would not be correct in his 5% probability, except under specially chosen conditions.


In your scenario, each of the 10-billionth souls would conclude, according to Carter, that there was only a 5% chance they were living in the first 5% of their population.
It would turn out that 80% of them were wrong. In their own populations, that simply means that they are in the 5% pastward tail, and that the 95% of their population who come after them are correct in using Carter's reasoning.

You understand what I'm saying, yes, but follow up your reasoning. As a function of birth number, given the distribution I assumed, the actual probability of being in the first 5% is completely a variable. For some it's less than 5%, for some more. This is my point-- there is automatically correlation between birth number and longevity probability. Carter thinks that because you don't know this correlation, you can ignore it and still use both the 10 billion number and the 5% number in the same calculation. In fact, you may only use one or the other, not both, even if you don't know the correlation between them. There are many other examples of this phenomenon in probability. I'll start a new thread about two envelopes containing money to demonstrate the point. It is very subtle, and I think it is the real reason that the Carter reasoning is not just of limited value, as I argued before, but it's downright wrong. The other arguments are working from an intuitive place where Carter can't be right, but they haven't really hit the mark because correlations in probability have a very subtle effect sometimes.

grant hutchison
2005-Nov-17, 10:14 PM
The trouble is that the argument doesn't apply at all.

Look at it this way: if you can end up with a paradox, then you didn't state the rules of the problem correctly, or you've made a bad assumption.For sure. Are you referring to any specific paradox?

Grant Hutchison

grant hutchison
2005-Nov-17, 10:46 PM
I'll start a new thread about two envelopes containing money to demonstrate the point.OK, I'll look at that.

Grant Hutchison

grant hutchison
2005-Nov-19, 07:00 PM
Back from that thread, with a quote lifted from there:


Concentrating on a subset of repeated trials with (birth-number = ten billion) for Carter is, to my way of thinking, missing the point in the same way as concentrating on a subset with (y=$10): that's not the way the world works, and the overall truth of the prediction comes out only when the full range of values is explored.I agree completely with this-- you are making my argument precisely. The last part of your sentence is exactly why Carter is wrong to say that there is only a 5% chance humanity will outlive its 200 billionth member! Because the first number, 5%, cannot be used in concert with the 10 billion to come up with 200 billion.
I disagree. I successfully used both bits of information in your envelopes puzzle to come up with a betting strategy which is successful over repeated trials. The fair price is struck if the buyer offers a price equal to the value revealed in the first envelope, whatever that price may be.
Carter uses his bits of information to come up with a probabilistic betting strategy which is successful over repeated trials (5% are wrong, 95% are right). This success is attained if the "bettor" bets on a total population equal to 20 times his birth number, whatever that birth number may be.
Both approaches go around any concern about the shape of the distribution. (And of course both approaches can also immediately be improved if we are provided with additional information about the distribution.)
By singling out an inappropriate subset of cases which share a specific value (of revealed content, of birth number) I think you're actually creating an apparent problem that doesn't exist if the whole picture is observed.

Grant Hutchison

Ken G
2005-Nov-19, 07:25 PM
I disagree. I successfully used both bits of information in your envelopes puzzle to come up with a betting strategy which is successful over repeated trials.

Actually, your betting strategy never used the $10 number in determining your formula. What you really did was use logic to show that a betting strategy that uses the first number (whatever it is, as you say) to buy the second envelope will work out in the long run. There's no $10 in that, except that your formula tells you that when you encounter the $10 subset, you should pay $10. But as soon as it does so, you no longer have any expectation of breaking even in that particular subset of trials! This last remark is crucial.



Carter uses his bits of information to come up with a probabilistic betting strategy which is successful over repeated trials (5% are wrong, 95% are right). This success is attained if the "bettor" bets on a total population equal to 20 times his birth number, whatever that birth number may be.

I see the issue. The key point is, you do not expect to have a breakeven strategy in any given subset, such as, 10 billion births!

Here's a better way to see it. Imagine that every intelligent being that has ever lived meets in the restaurant at the end of the universe. Someone says, "let's have all the people who were among the first 5% born to one corner of the room." Of course there's going to 1/20 of the attendees in the corner. Now they say "OK, can I have every being that was the 10 billionth born to raise their hand". Why would you expect 1/20 of those people to be in the corner? Generally, that would not be the case! The reason is, if you imagine that total populations are generated by some probability algorithm, then that algorithm must have some scale, a "median" birth number. The number 10 billion has some unknown relation to that scale, which generates the unknown correlations, just as the number $10 has some relationship to the envelope-stuffers scale! We used to think that because the relationship was unknown, Carter was an OK probabilistic argument. But what I'm now saying is, just because you don't know the correlation does not make the argument meaningful. You still don't pay $12.50 for the other envelope, and you still don't expect there to be 1/20 of the 10-billiionth-borns in the corner of the room! So Carter is just plain wrong, since it is an argument that completely involves that bunch of 10-billionth-borns. This is the subset I've been referrering to-- it's us!

grant hutchison
2005-Nov-19, 08:27 PM
... your formula tells you that when you encounter the $10 subset, you should pay $10. But as soon as it does so, you no longer have any expectation of breaking even in that particular subset of trials! This last remark is crucial.But only if you believe that this subset of trials has any relevance to the real world. I say it hasn't.


Someone says, "let's have all the people who were among the first 5% born to one corner of the room." Of course there's going to 1/20 of the attendees in the corner. Now they say "OK, can I have every being that was the 10 billionth born to raise their hand". Why would you expect 1/20 of those people to be in the corner? Generally, that would not be the case!I'm very happy with that. I can't see why there should be any correlation between these two groups. If most populations are very long-lived, then a lot of those with the 10-billionth birth number are going to be in the corner. Carter doesn't care, so long as 5% of each population was wrong in its assertion that it was in the last 95% of all births.
I think you're chasing some sort of symmetry that doesn't have to be there for Carter to work.

Grant Hutchison

Ken G
2005-Nov-19, 08:43 PM
I think you are missing the key point that Carter says more than just that we have a 5% chance of being in the first 5%, it also tries to use the fact that we are the 10 billionth born to say that this says something about our total populations. I'm saying that you cannot use both of those numbers to generate a third one, like an "expected longevity" of humanity. That is exactly the same mistake as using $10 to get an expected value for the other envelope, by any method. You're saying you don't care about the 10 billionth-born subset, all you care about is the 1/20 of the total attendees who are in the corner. But if that were true, you would have to be able to state Carter's claims without applying it to us, because we're in that subset of 10 billion borns, and there are not 1/20 of us in the corner. Carter's error is in the number 200 billion, not in the number 5%. We are 5% likely to be in the first 5% until you use the information that we are 10 billionth born. You can't use both those numbers and apply it to us.

grant hutchison
2005-Nov-19, 08:52 PM
I think you are missing the key point that Carter says more than just that we have a 5% chance of being in the first 5%, it also tries to use the fact that we are the 10 billionth born to say that this says something about our total populations.No, I understand that.


I'm saying that you cannot use both of those numbers to generate a third one, like an "expected longevity" of humanity.And I'm saying we can, and that the absence of correlation between the "10-billionth group" and the "first 5% group" is entirely to be expected.


you would have to be able to state Carter's claims without applying it to us, because we're in that subset of 10 billion borns, and there are not 1/20 of us in the corner.And that's fine, because we don't care about where we're situated among the superset of all populations everywhere, only about where we're situated in our own population. And we accept the possibility we might be wrong in our prediction.

Grant Hutchison

Ken G
2005-Nov-19, 09:22 PM
And that's fine, because we don't care about where we're situated among the superset of all populations everywhere, only about where we're situated in our own population. And we accept the possibility we might be wrong in our prediction.

I see that your approach is to restrict to the human in the "restaurant". But here's the problem with doing that. There's only one 10 billionth born, let's say it's you. You're either in the corner, or you're not. But whether you are or not, on what are you going to base your conclusion that there was a 5% that you would be? What criterion can you use other than looking at the other 10 billionth borns in their own similar intelligent species? Let me go through the steps of Carter's argument. First all, we assert that you and I have only a 5% chance of being in the first 5% of humans. We've talked about potential difficulties with that, but those are irrelevant compared to this more central issue, so let's assume that this much is true. OK, now we do some research and find we are the 10 billionth born. The last step is to combine these tidbits as if the second did not alter the first, and conclude that there is only a 5% chance that humans will outnumber 200 billion. That is the wrong step. A probability is a fraction out of many trials that are equally likely. It is fine if those trials are idealized, they don't ever have to actually take place. But what is the set of trials that you are using to say that there is a 5% chance humanity will outnumber 200 billion, if it is not the trials represented by all the species in the "restaurant"? I can't see any basis for using a 5% probability in concert with a 200 billion total population. Can you explain any way whereby you have a set of (idealized) populations distributed such that 5% are more than 200 billlion and 95% are less? And if you can't, what does a 5% probability mean?

grant hutchison
2005-Nov-19, 10:20 PM
There's only one 10 billionth born, let's say it's you. You're either in the corner, or you're not. But whether you are or not, on what are you going to base your conclusion that there was a 5% that you would be?The fact that I am declaring with 95% confidence that I'm in the latter 95% of all humans who'll ever live: that requires that there's a 5% chance I'm in the corner.

What criterion can you use other than looking at the other 10 billionth borns in their own similar intelligent species?OK, that's fine, but that's bringing in a bit of external evidence. I suggested this was what you were doing when you first introduced this argument, and you denied it at that time.

A probability is a fraction out of many trials that are equally likely. It is fine if those trials are idealized, they don't ever have to actually take place. But what is the set of trials that you are using to say that there is a 5% chance humanity will outnumber 200 billion, if it is not the trials represented by all the species in the "restaurant"?I am not performing these trials. The 200 billion result comes only from the 10-billionth human. Other humans perform the same calculation and produce different results. 5% of those various different human results will be wrong, not 5% of every 10-billionth soul who makes the 200 billion calculation. I've no idea how many of those will be wrong.

If we go back to your restaurant, with the first 5% of every species cluster in one corner: the 10-billionth soul of some species will be in the corner, the 10-billionth soul of other species will still be in the general room. All the folk in the corner will be saying "Oops, we got our calculations wrong - that's 95% confidence for you." All the folk in the general room will be saying "Yep, we were right to assert we were in the last 95%." So some 10-billionth souls will have been right, and some will have been wrong. We can't say what proportion without introducing external information. It doesn't make the calculation invalid at the 95% level for any given population.

Grant Hutchison

01101001
2005-Nov-19, 10:22 PM
And if you can't, what does a 5% probability mean?

There's the rub. I feel Carter's argument is information that does not inform. What can we do with it? How should we behave? What should we do different? What can we do with it? What does it mean?

grant hutchison
2005-Nov-19, 11:06 PM
What can we do with it? How should we behave? What should we do different? What can we do with it? What does it mean?I dunno.

Grant Hutchison

Ken G
2005-Nov-19, 11:39 PM
OK, that's fine, but that's bringing in a bit of external evidence.

What external information? All the species are identical. Think of them as all the possible ways humanity could have gone, if a butterfly had flapped its wings. How is this not exactly the issue involved in a confidence interval? You are clinging to the 95% confidence without recognizing that being the 10 billionth changes that confidence irreparably, even if you don't know how it changes it. You may only maintain that confidence if you never use the information that you are the ten billionth. That is the extra information.


The 200 billion result comes only from the 10-billionth human. Other humans perform the same calculation and produce different results.

But that's not how confidence intervals work. Where is your idealized set, all who conclude their species won't outlive 200 million, and only 5% are wrong? That is the set you have not produced.

grant hutchison
2005-Nov-20, 12:31 AM
What external information? All the species are identical. Think of them as all the possible ways humanity could have gone, if a butterfly had flapped its wings.Well, you originally suggested we look at:
... the other 10 billionth borns in their own similar intelligent speciesThat certainly seemed like you were looking for external information about the likely lifespan of intelligent races.

You are clinging to the 95% confidence without recognizing that being the 10 billionth changes that confidence irreparably, even if you don't know how it changes it."Clinging"? Nice choice of word.
But 95% confidence here doesn't require all those other imaginary species in their crowded restaurant - just us humans.
Let's say everyone who ever lived announced at some time in their lives: "I'm one of the last 95% of people who'll ever lived." 95% of them would be right. If they were statistically astute, they'd have better phrased it "I'm 95% confident I'm one of the last 95% of people who'll ever lived." The 10-billionth person might be one of those in the top 95% or one of those in the bottom 5%: it doesn't matter, his/her statement is still valid.



Where is your idealized set, all who conclude their species won't outlive 200 million, and only 5% are wrong? That is the set you have not produced.There is no such set. The 200 million figure is the product of a single person, the 10-billionth, who may be right (with 95% probability) or may be wrong (with 5% probability). His rightness or wrongness is not a function of the figure 200 million, but of his/her actual position in the total number of humans, something which (s)he cannot know, but can certainly estimate with 95% confidence.
In some of your collection of species, the person who predicts 200 million will be correct; in others, that person will be wrong. We have no way of estimating the proportion of rights and wrongs in that set, but that proportion has no mathematical linkage to the first 5% of humans who ever lived, or whether there were more or less than 10 billion of them.
I think you're confusing within species probability and across species probability.

Grant Hutchison

Ken G
2005-Nov-20, 03:46 AM
"Clinging"? Nice choice of word.

No offense intended, your insights have been razor sharp. Indeed, on several occasions I've thought, "wait, he's right... no wait, I'm right..." But I see it clearly now, and there is only one point that I am having trouble making to you. It looks like you and I are all that remain in this debate for the time being, so it's up to us to get to the bottom of it.



Let's say everyone who ever lived announced at some time in their lives: "I'm one of the last 95% of people who'll ever lived." 95% of them would be right. If they were statistically astute, they'd have better phrased it "I'm 95% confident I'm one of the last 95% of people who'll ever lived."

Agreed.


The 10-billionth person might be one of those in the top 95% or one of those in the bottom 5%: it doesn't matter, his/her statement is still valid.

Also agreed. As long as they never use the information that they are the 10 billionth, your statement is completely correct. But there's something we must agree on. The only meaning to the statement that "I am 95% confident" of anything is that I have in mind a collection of cases, or trials, or individuals, that are effectively indistinguishable, except for some random stirring which is being examined in regard to the confidence interval. How are you going to even define your confidence interval, if you don't use this definition? You have in mind a set of all humans, and the stirring is the difference between you and someone else. So yes, of that set, 5% are in the first 5%, it's well defined. But the same must be true of any statement of confidence. Therefore, if you claim that "I am 95% confident that humanity will not outlive 200 billion", you must also produce such a set (even if idealized) or it is a meaningless statement. Note this statement sounds very similar to your statement above, but it is not the same, because it explicitly uses additional information-- I am the 10 billionth. The first statement has a set where 5% are wrong, the second set does not, not even in principle. This makes it an incorrect probability argument, much like the set of people who saw a $10 envelope and figured it would be the cheaper envelope half the time. How do you prove that claim wrong? By asking for a set of people seeing $10 and having it be the lower envelope half the time. Could the supporters of that probability view simply say...

There is no such set.

as if it saved their argument? The fact that there is no such set is exactly why the argument is wrong.



In some of your collection of species, the person who predicts 200 million will be correct; in others, that person will be wrong. We have no way of estimating the proportion of rights and wrongs in that set, but that proportion has no mathematical linkage to the first 5% of humans who ever lived, or whether there were more or less than 10 billion of them.

Exactly. This is why you simply cannot be 95% confident that the 200 billion limit applies.

I think you're confusing within species probability and across species probability.

The within species probability is not enough, because it is not a probability that tests a confidence interval about a population that makes it to 200 billion, it only tests a confidence interval about being in the first 5%. These are different tests! What do you mean by "I am 95% confident our species won't outlive 200 billion." Where is the 95% who are right about this exact statement, right from their own mouths, and the 5% who are wrong? Carter must change the statement to "I am 95% confident our species won't outlive 20*x, where x is my birth number". That statement only checks out if you range over all x, but then where is the 200 billion? This is just like you had to range over all y to say that the expectation of the second envelope is y, and you realized that isn't true for a given y=10. It is just the same thing here.

grant hutchison
2005-Nov-20, 02:29 PM
It looks like you and I are all that remain in this debate for the time being, so it's up to us to get to the bottom of it.Or to abandon it as an argument that's going nowhere, to be frank.

So yes, of that set, 5% are in the first 5%, it's well defined. But the same must be true of any statement of confidence. Therefore, if you claim that "I am 95% confident that humanity will not outlive 200 billion", you must also produce such a set (even if idealized) or it is a meaningless statement.The same set provides confidence for both claims. The mathematical linkage between the first claim and the second claim is trivial.
(A paediatrician tells you that your son's height is in the 98th centile for his age. You go home and measure your son's height - he's 120cm tall. That immediately tells you that 2% of kids your son's age are over 120cm tall. You don't need to doubt that proportion just because you've now measured the height. It's the same proportion, with or without the numerical knowledge.)

Could the supporters of that probability view simply say...
There is no such set.as if it saved their argument? The fact that there is no such set is exactly why the argument is wrong.Or this imaginary set has no relevance to the argument, which is my position. The set of relevance is the set of humans who make a claim about the likely total number of human lives, using their birth-number to make the calculation. 5% of them are wrong.
The set of people who make the specific claim that there will be 200 million humans, of which 5% are wrong, doesn't exist, as I said, for exactly the reasons you've set out. But it's just a non-issue for this problem.
Are you going to claim that there is a set of children 120cm high who are not in the 98th centile for their age, and so measuring your child's height invalidates the paediatrician's assessment?

Here's another way of looking at it. Saying you're 95% percent certain you're not in the first 5% of human lives is just shorthand for saying "If only 100 humans lived, I'm 95% certain I wouldn't be in the first 5." Inserting your birthnumber merely introduces a constant of proportionality into that claim. Each individual inserts their own birth number, and comes up with their own estimate of the total number. The claim stays the same because the proportion is the same, just as it would have done if you'd decided to express your claim as a fraction or a per-mil, rather than a percent.

Grant Hutchison

Ken G
2005-Nov-20, 03:46 PM
Or to abandon it as an argument that's going nowhere, to be frank.

I hope we won't have to do that, the answer is out there!



(A paediatrician tells you that your son's height is in the 98th centile for his age. You go home and measure your son's height - he's 120cm tall. That immediately tells you that 2% of kids your son's age are over 120cm tall. You don't need to doubt that proportion just because you've now measured the height. It's the same proportion, with or without the numerical knowledge.)

Your example is perfect, it shows us exactly what we need! But you have to change it into a probability argument. Let's say my son's height is 120cm. Can I assert with 95% confidence that he is above the 5% rank with no additional information? Yes I can, because I never used the 120cm, I can always make that assertion. The real question is, can I put limits on the median height at this same confidence, by using that 120cm? No, that is exactly what I can not do! Here is how it would go. I'd assume height was a Gaussian variable, and I'd say I was 95% confident my son was above the 5% point. Let's say that means he is above half the median height (we'd need the standard deviation to allow this to be done, I'm picking numbers to come out simple). Can I now put a 95% confidence that the median height is below 240cm? There is a testable prediction, a calculation we can actually do and see who is right. I will wager anything reasonable that if we do this calculation, we will find that if we randomly select heights, and range over all possibilities, we will not find that 95% of the parents in our hypothetical study will be correct in the 95% limit they place on the median height! I don't know if it will be above or below 95%, but there's no reason to expect 95% as you must be claiming. (Unless, that is, there is something very special about Gaussian distributions, so I wouldn't wager too much until I'd thought that part out!)


The set of relevance is the set of humans who make a claim about the likely total number of human lives, using their birth-number to make the calculation. 5% of them are wrong.

Here you are ranging over all the people who make the calculation, so you are correct that 5% will be wrong. But we are not using the 5% number after ranging over all humans, we are using it just for ourselves, in saying that humanity is 5% likely to outlive 200 billion. The argument cannot be stated without the 200 billion number, so it has not ranged over all humans.



Are you going to claim that there is a set of children 120cm high who are not in the 98th centile for their age, and so measuring your child's height invalidates the paediatrician's assessment?

No, I'm not bringing in another parameter like age. I'm saying that the median height limit, for that age, cannot be calculated to 95% confidence using the single 120cm measurement (and the standard deviation information), because humans have an average height, and 120cm has some unkwown relationship to that height, so I can't make any confidence calculation about that average height.



Here's another way of looking at it. Saying you're 95% percent certain you're not in the first 5% of human lives is just shorthand for saying "If only 100 humans lived, I'm 95% certain I wouldn't be in the first 5." Inserting your birthnumber merely introduces a constant of proportionality into that claim. Each individual inserts their own birth number, and comes up with their own estimate of the total number. The claim stays the same because the proportion is the same, just as it would have done if you'd decided to express your claim as a fraction or a per-mil, rather than a percent.

The point is, since the number 200 billion appears in the Carter catastrophe argument, you have not ranged over all humans in presenting that argument, and so the 95% confidence does not apply.

grant hutchison
2005-Nov-20, 03:57 PM
OK.
We're now well past the stage at which you're just repeating your stuff and I'm just repeating my stuff.
Time to give up, I think. :)

Grant Hutchison

Ken G
2005-Nov-20, 04:16 PM
Wait! I've had a revelation. The reason that Carter's 95% confidence is in general not correct is the correlation between 10 billion and the actual "average" number of intelligent beings in any species. I've said that already. The only way Carter works is if there is no such correlation,and although you may not agree with that statement right now, perhaps you will when I point out that there is a way to eliminate that correlation, and that is if the population distribution is scale free. This gibes with your last point about being able to rescale the numbers, so that's why I think this might be a useful insight to this discussion. So what kind of distribution is scale free? A power law! So Carter only works if populations are distributed according to a power law. So it's not the Gaussian distribution, it's power laws, that are special. So that's good news and bad news for Carter. The good news is, there are a lot of distributions in life that do come out power laws. The bad news is, they always break down at some point. If the true population-distribution power law is too small, then you have zero population expectation, and if it's too large, then you have infinite population expectation. You have to introduce a scale to avoid these problems, and as soon as you do, bingo, the correlation appears and Carter breaks down. I maintain that your intuition is correct, but it's working under the unrevealed assumption of a scale-free population distribution, which in practice is not possible. But if you can resuscitate the scale-free idea, then you have resuscitated Carter. Otherwise, RIP.

Ken G
2005-Nov-21, 03:25 AM
It looks like this thread has reached its end, so I'll summarize what was learned for those who might still peruse it, or be curious about the validity of the Carter catastrophe conjecture. The idea behind the catastrophe is that if each person has x humans born up to their own birth, then they can be 95% certain they are not in the first 5% of humans, in the absence of any other information. This would seem to suggest that the 95% confidence extended to the idea that humanity would not outlive a count of 20 times x, but the primary debate centered on whether or not it was allowed to set x=10 billion (or so, for us), and still expect the 95% confidence to apply to the number 20x = 200 billion. To have this work, most felt it would have to be argued that setting x=10 billion did not constitute any extra information. But since 10 billion must have some relationship to the actual expected number of humans (i.e. high or low in relation to it), in a situation of complete knowledge of all the contributing factors, so most felt that using any actual value of x (such as 10 billion) did constitute extra information, invalidating the Carter confidence interval. This was not a unanimous view. No doubt the Carter conjecture will continue to be debated, but probably not on this forum!

TheOrqwithVagrant
2005-Nov-26, 01:59 PM
Sorry if I'm resurrecting a dead thread and only using new words to reiterate arguments that have already been put forward, but it seems to me that the problem with the Carter Catastrophe is that it implies that the future or present probability of something is somehow altered by the probability of the chain of events leading up to it.
If I roll three six sided dice in a row, the chance of getting a 6 the last roll is still 1 in 6, not 1 in 216, no matter if I already rolled 6 twice in a row before.
Really... isn't any attempt to calculate the probability of any event or outcome based on anything but the probabilites of the direct physical factors that matter to the specific outcome/event in question inherently meaningless, and a matter of "confusing the map with the territory"?

Mister-E
2007-Nov-25, 08:53 AM
I read er... listened to "time" by stephen baxter, i found the "Carter Catastrophe" pretty interesting, you see it happen all the time in nature, except maybe for cockroaches, which apparently never die out. I'm just going to forget about the hysteria and note a couple possible holes in the theory, first of all, it just cites population, not other important factors like technology level, area occupied etc. Secondly, you can say you have sufficient data, but really, is looking at a few samples of data over 100 or so years going to suffice, we may have accurate data for humans at least, but for the animals you're comparing us to you may not have enough samples for the comparison.
For the theory, if u suspended ur disbelief long enough, it does make sense if you think about it a certain way, first of all, all species are terminated on a distinct timeline, no exceptions even cockroaches, mosquitos, and fruit flies, which we wish were all dead now.

Otherwise the theory makes sense, but as history has shown us, the worst thing that happens with isolated populations is they all die (easter island) which we have a lot of, Or the government collapses and a lot of people die, but not all.

HenrikOlsen
2007-Nov-25, 12:18 PM
Before starting on this again, please note that this has been a resurrection of a thread that died 2 years ago when the discussion reached it's natural end.
The resurrection hasn't added anything new to the discussion.

Chuck
2007-Nov-27, 03:02 PM
The probability of drawing a high numbered ball from the million ball urn is zero if the person filling the urn has had only enough time to put in the first ten balls. Drawing a seven tells us nothing about how many balls will eventually be placed in the urn. For the experiment to work, I'd have to wait until all of the balls are placed.

Even if the human race will be much larger, we haven't been around long enough to produce trillions of members so of course my birth order is low. No matter how large the population will become, everyone doing the experiment now is going to get a low birth number. It has nothing to do with how large the population will become just as the early drawing of a ball from an urn doesn't tell us how many balls will eventually be placed in it.

It may be true that most of the people from among everyone who will ever live will be correct in guessing that the human race will become extinct soon, but I have no reason to believe that I'm in that majority because we're not done producing more people. A ball can't be drawn from an urn if it hasn't been put in yet.

Ken G
2007-Nov-28, 04:04 PM
It may be true that most of the people from among everyone who will ever live will be correct in guessing that the human race will become extinct soon, but I have no reason to believe that I'm in that majority because we're not done producing more people.

The problem with your argument is that it gives a result that even though 90% of humans will be in the last 90% born, by your argument none of them will ever be able to conclude that they are likely to be in the last 90% born, on the simple grounds that they have no way of knowing if we are done producing more people. That doesn't make your argument wrong, it means that it still leaves something to explain.

Chuck
2007-Nov-28, 05:49 PM
If all I knew was that there were going to be a finite number of humans and I was one of them then I'd have to say that there's a 90% chance that I'm in the last 90%. But I have made other observations. As stated in the problem, I know that the population is increasing geometrically. With the possibility of truly vast numbers of new people being added, how is it possible for anyone to seriously think that he's now probably in the last 90%?

It would be different if the earth really had standing room only or a killer asteroid were already spotted. But as it is, the same geometric progression that suggests that most of the people who will ever live will be in the last few generations also suggests that we aren't in those generations.

Noclevername
2007-Nov-28, 05:58 PM
Just goes to show how little number games can actually mean sometimes.

Ilya
2007-Nov-28, 06:17 PM
Bayesian statistics (i.e. "Carter Catastrophe") only works when applied to finite sets. When applied to potentially infinite sets, such as total number of humans that ever will live, it produces meaningless results. In particular, it produces same result (we are nearing the end of the set) no matter when it is applied -- today, 1000 years ago, or 10,000 years ago.

Ken G
2007-Nov-29, 05:03 AM
If all I knew was that there were going to be a finite number of humans and I was one of them then I'd have to say that there's a 90% chance that I'm in the last 90%. But I have made other observations. As stated in the problem, I know that the population is increasing geometrically. With the possibility of truly vast numbers of new people being added, how is it possible for anyone to seriously think that he's now probably in the last 90%?
Yes, this is also my position on this-- that we are applying additional information that can compromise the idea that we are in the last 90% born. I feel that one can never use the number of people that have lived in the same argument as one that asserts we are 90% likely to be in the last 90%. So it's true that anyone can conclude they are 90% likely to be in the last 90% in the absence of all information, but they cannot look at how many people have been born (be it 10 or 20 billion or whatever it is), as then they are no longer acting in the absence of all information.


It would be different if the earth really had standing room only or a killer asteroid were already spotted. But as it is, the same geometric progression that suggests that most of the people who will ever live will be in the last few generations also suggests that we aren't in those generations.
The Carter hypothesis would require that the exponential phase is quite likely to be coming to a close, on the grounds that any random person is likely to be born near the close of such a phase, not near its inception. But as soon as we look at how many people have been born, so where we are in that exponential process, we can no longer use that argument because there can be correlations that we are simply unaware of that spoil the Carter argument. In other words, if you have no information that can interact with correlations, then you have a truly random sample, but as soon as you use information that might involve correlations, even if you don't know what the correlations are, you can no longer assume your sample is random. That's also the conclusion from the "envelope puzzle" described in a thread spawned by this one, I forget its title. (I never reached rapprochement with Grant on this issue, I wonder what his opinion is at this point.)

Ken G
2007-Nov-29, 05:06 AM
Bayesian statistics (i.e. "Carter Catastrophe") only works when applied to finite sets. When applied to potentially infinite sets, such as total number of humans that ever will live, it produces meaningless results. In particular, it produces same result (we are nearing the end of the set) no matter when it is applied -- today, 1000 years ago, or 10,000 years ago.But the number of humans that will be born is almost certainly a finite set. You're right though, this is all about Bayesian statistics.

Ilya
2007-Nov-29, 02:05 PM
I wrote potentially infinite set. It is undefined.

Ken G
2007-Nov-29, 02:53 PM
But I'm saying it's not potentially infinite, the expansion of the universe would seem to preclude that. I don't think the weakest aspect of the Carter catastrophe conjecture is the assumption of a finite total human population, it is the assumption of a lack of correlation between the current birth number and the expectation of future longevity. But perhaps one could merge the arguments and say that if the total number may be assumed to be finite, then correlations exist between current birth number and the possible totals, and if the total is potentially infinite, then no such correlation exists but then the Carter conjecture is invalid on the grounds you raise.

Chuck
2007-Dec-11, 02:57 AM
The entertaining aspect of this problem is that the theory predicts the opposite of what would seem like common sense. If a population is thriving then the theory predicts an early die off. If a population is declining then the theory predicts that they'll be around for a long time.

Since there's nothing special about me we can estimate that there will be about as many people born after me as there were born before me. Since population is increasing, those born after me will live in far less time so we face early extinction. Back in the sixties I bought myself a slide rule. I didn't really need one but wanted something to play with. There was nothing special about that slide rule so we can estimate that there will be about as many manufactured after it as there were manufactured before it. Since the calculator made the slide rule obsolete they're now being made only as novelty items so it will take a long time to manufacture as many as already existed when I bought mine. This reasoning leads to the conclusion that we'll still be making slide rules for millennia after we've become extinct.

Ken G
2007-Dec-11, 06:40 AM
The entertaining aspect of this problem is that the theory predicts the opposite of what would seem like common sense. If a population is thriving then the theory predicts an early die off. If a population is declining then the theory predicts that they'll be around for a long time.
I think that's another fallacy in the Carter argument. All you can say is that 90% of humans will be born in the last 90% of humans, so if you and I are randomly selected humans, then we are 90% likely to be in the last 90%. But that will no longer be true if we specify our birth number (say 15 billionth, I don't know), nor will it be true if we specify that the population is growing exponentially (and like you say, Carter used the exponential growth to convert birth totals to a timescale, which involves additional information that cannot be used if one assumes we are randomly selected humans unless one thinks a population grows exponentially right until it goes extinct).

Since the calculator made the slide rule obsolete they're now being made only as novelty items so it will take a long time to manufacture as many as already existed when I bought mine. This reasoning leads to the conclusion that we'll still be making slide rules for millennia after we've become extinct.Yes, that exposes the fallacy.

grant hutchison
2007-Dec-11, 02:55 PM
Yes, that exposes the fallacy.I think it exposes a fallacy, but not one in Carter's reasoning.
The fallacy here is in doing a post hoc analysis once the results are in and the distribution is known. "There was only one chance in 14 million that my lottery number would come up, and yet it came up: that's so amazing!"
At the time Chuck bought his slide-rule there was a 90% chance it would be in the last 90% manufactured, and a 1% chance it would be in the last 1% manufactured. Turns out, with hindsight, that it was unusual, in that it was bought at the cusp of the Carter catastrophe for slide-rules.

Grant Hutchison

Chuck
2007-Dec-11, 04:05 PM
I used the slide rule as an example to get the result that I wanted just as Carter or whoever used the increasing human population to get the result that he wanted. The fact that he knows that the population is increasing is similar to my knowledge the the slide rule is obsolete.

grant hutchison
2007-Dec-11, 06:51 PM
I used the slide rule as an example to get the result that I wanted just as Carter or whoever used the increasing human population to get the result that he wanted. The fact that he knows that the population is increasing is similar to my knowledge the the slide rule is obsolete.The difference is that you are observing the situation post hoc, with a strong indication that the bulge of sliderule manufacture has passed. You have excellent reasons to believe that our current situation is unusually, and that we have lived through the End of Days for sliderulekind.

Carter's argument holds only in the absence of a priori reasons to believe we are unusual with regard to our birth order. Either we are unusual in our birth order, or the current exponential growth in population will be strongly modified in the near future.
It's such a banal statement that I've never been quite sure why people get exercised about it. (I'm not suggesting that you are getting exercised, by the way; simply observing that emotions do seem to run high on this topic, quite often.) It's probably twenty years since I read Carter's original paper, and I can't seem to find it in the files, but I don't recall him saying much else apart from inviting us to come up with a convincing argument that we are in a privileged position with regard to birth order. It has since been overegged as some sort of inevitable, unavoidable doomsday scenario, but I don't recall that being Carter's stance.

Grant Hutchison

Chuck
2007-Dec-11, 07:54 PM
I do have reason to believe that my birth number is relatively low. Population has been increasing for millennia so it seems likely that there will be more people born after me than before me. It's certainly not a sure thing, but I do know more than just my birth number, just like in the slide rule situation. I'm not even certain about the slide rules. Some religious sect could take over the planet and ban electronics. Then slide rules would make a comeback. I don't expect that to happen just as I don't expect humanity suddenly die out.

I see little difference between the slide rule situation and the population situation.

grant hutchison
2007-Dec-11, 08:25 PM
I see little difference between the slide rule situation and the population situation.In one case you have knowledge; in the other case you have an assumption.

Grant Hutchison

Chuck
2007-Dec-11, 08:29 PM
I can only guess that slide rules won't make a comeback. I can't see the future in either case. I see slide rules decreasing and population increasing. I see no immediate reason for either trend to end. The situations are nearly identical.

JohnBStone
2007-Dec-11, 08:48 PM
Just musing about some factors here:

1) The human race is not a distinct set - it is a continuum with all ancestors that went before it way back to the first replicator (and there are probably some parallel near-human life branches that could interbreed with humans that are already extinct). "Species" is a term of convenience (endearment?). So shouldn't we apply the argument to all life leading up to this point rather than to a fuzzy set like humans?

2) The Earth's human population is predicted to peak by the end of the century anyway (source UN).

grant hutchison
2007-Dec-11, 08:49 PM
The situations are nearly identical.The similarity isn't really jumping out at me, I have to confess. So I'll call it a day on this one. :)

Grant Hutchison

grant hutchison
2007-Dec-11, 08:58 PM
So shouldn't we apply the argument to all life leading up to this point rather than to a fuzzy set like humans?Yes, that's one of the big debates among people who take Carter seriously. What is the population we're dealing with? What is the "start" and what is the "end"?
Do we start the clock with the last speciation event, the last near-extinction bottleneck, or just in 1983 when Carter first raised the idea?
At the other end, some of the arguments that "undo" Carter bring their own implicit catastrophe. What if the vast majority of humanity will exist in a form different from the one we currently have (as software, cyborgs, or something unimaginable)? Then we certainly occupy a privileged position in the first 10% of "humanity", and Carter is undone. But shouldn't Carter's argument then apply to "humans like us"? In which case, Carter may be both correct and wrong: "humans like us" disappear, while "humanity" in a different form persists. Similar reasoning applies for the "infinite lifespan" argument, in which we must necessarily stop being "us" in order to continue as "humanity".

Grant Hutchison

Chuck
2007-Dec-11, 09:02 PM
I see a trend in each of the quantities of two sets of items. I see no reason for either trend to change in spite of the U.N. prediction. If the population does peak I'd expect that to work against the doomsday prediction anyway.

Ken G
2007-Dec-12, 09:14 AM
Carter's argument holds only in the absence of a priori reasons to believe we are unusual with regard to our birth order. Either we are unusual in our birth order, or the current exponential growth in population will be strongly modified in the near future.Actually, there's two separate issues there, relating to whether one is talking about time or just birth order. If one is only speaking about birth number, then the exponential character of current growth plays no role. The "fallacy" I was referring to above was the connection with time, the idea that we only have a few more e-folds to go at the current growth rate. In my view, that confounds the initial fallacy by also connecting it to a temporal growth rate. But the real issue continues to be the initial mistake of connecting birth number to the generic character of our personal standing. Either may exist alone, but the two together are incompatible, they violate basic assumptions needed.


It's such a banal statement that I've never been quite sure why people get exercised about it.The statement that either we are in the last 90%, or we are unusual, is not controversial. It is the statement that the Earth's population has only a 10% chance of exceeding its current birth number by a factor of 10. That statement is true generically, but it is not true if one ever actually uses the current birth number. That number has unknown correlations that violate the basic assumption that we are generic. So what I mean is, anyone can say "I'm probably in the last 90% of the Earth's population", but if they actually look at their birth number, they cannot use the number 90% and the birth number in the same probability argument. It's just wrong to do so.

grant hutchison
2007-Dec-12, 09:32 AM
So what I mean is, anyone can say "I'm probably in the last 90% of the Earth's population", but if they actually look at their birth number, they cannot use the number 90% and the birth number in the same probability argument. It's just wrong to do so.And yet the whole of inferential statistics is based on taking actual numbers and plugging them in to normalized sampling distributions, in order to derive other actual numbers.

Grant Hutchison

Ken G
2007-Dec-12, 11:11 AM
I am not saying one must avoid actual numbers when doing probability and statistics, I'm saying one must keep careful track of one's assumptions to make sure they are not invalidated by those numbers. Here we have the assumption that we are "generic" humans, sampled randomly from the total (eventual) population. We also have our birth number. There are correlations there which we cannot assume are absent, or we are doing incorrect probability. The fact that we do not know the correlations does not allow us to ignore them.

Let's look at the situation with birth number. I don't remember what I said above, but this argument seems pretty direct. Let's imagine that at the end of our galaxy, a super-intelligent species looked at all the intelligent beings that ever lived in that galaxy (perhaps from careful archeology), and took stock of the total number of beings that lived in those species. There'll be some kind of distribution over total birth number. (Let's define "intelligent species" as "one that considered the Carter argument" at some point.) Can we say that 90% of those beings lived in the last 90% of their kind? Certainly yes. Can we say that 90% of those who, individually, considered the Carter argument, were in the last 90% of their kind? Who knows, but very probably not. It is quite possible that either this argument comes up long before the end of a species, maybe because it shows enough self-awareness to stave off extinction, and it is also entirely possible that it comes up about the same time as the species wipes itself out, out of suicidal angst of some kind. So it already violates the "generic" assumption, needed for 90% to live in the last 90%.

But even without that argument, the Carter hypothesis still fails, because even if we do not ask if those species asked the Carter question, if we instead ask if they discovered the wheel, or fire, there will still be some kind of probability distribution of total birth number per species. So true enough, 90% will live in the last 90%, but can they use their individual birth number as a predictor of the the total birth number? Let's look at some examples.

Imagine a game where a number is selected at random and with equal likelihood from a distribution from 1 to N, but you do not know N. Furthermore, you are told that first another distribution, that you know nothing about, is used to choose N, and then you get your number from 1 to N. Now you are asked, before you look, what is the probability that your number will be larger than N/10, whatever N is? Answer, 90%. We agree there. Now you look at your number, and it is 100. Then you are asked, what is the probability that N is less than 1000? Not 90%, that is the wrong answer, pure and simple. You simply have no way of answering the question meaningfully. If you doubt me, try using various distributions to choose N. Unless you choose a "rigged" distribution, you will see what I mean. (It suffices to choose a bimodal distribution of just two possible N, so that you are basically playing my game with the envelopes, so that's why I introduced that other thread.)

Chuck
2007-Dec-12, 02:43 PM
Let's suppose that the human race does continue to increase exponentially and then comes to a sudden end. Then a time traveling alien comes back and asks one of us if the end is near. If the alien chooses one of us at random with equal probability then the correct answer is probably yes. If the alien chooses a random year from among those in which we existed and then asks someone in that year then the correct answer is probably no.

If I'm asked that question now, how do I answer? Do I assume that I'm a typical human being or do I assume that this is a typical year? If the alien asks everyone and everyone guesses no, the end is not near, then most of the individual answers will be wrong but for most of the time it will be right. If I want to maximize my chance of being right, what should my answer be? If I assume that I'm a typical human being and answer yes, the end is near, then I'm assuming that this is not a typical year, it's a year very close to the end. Can I justify assuming that there's anything special about this year any more than I can assume that there's something special about me? It seems that I'd have to do one or the other.

Or maybe this statistical method can't be used to predict the future. Not that abusing a statistics isn't fun, of course.

Ken G
2007-Dec-12, 03:14 PM
1) The human race is not a distinct set - it is a continuum with all ancestors that went before it way back to the first replicator (and there are probably some parallel near-human life branches that could interbreed with humans that are already extinct). "Species" is a term of convenience (endearment?). So shouldn't we apply the argument to all life leading up to this point rather than to a fuzzy set like humans?
That is indeed a big problem, relating to "Bayesian statistics". How much do we already know about ourselves that we are allowed to use? In other words, one set you could take would be humans, as we are, but that does lead to all kinds of uncertainties. Or, it would be equally valid to look at the sum total of all beings in the entire universe who are capable of asking the Carter conjecture. Is that not also a valid set? We should conclude that we are most likely to be among the last 90%, and the first, of that set, as long as we don't use anything about ourselves other than that we asked the Carter question. As soon as we use one other single thing, anything about humanity, anything about birth number, anything about how we think-- we can not use that other thing and the 90% idea at the same time in any valid probabilistic calculation.

grant hutchison
2007-Dec-12, 03:17 PM
So true enough, 90% will live in the last 90%, but can they use their individual birth number as a predictor of the the total birth number? Let's look at some examples.But the "castastrophe" in Carter's reasoning isn't driven by the value of my birth number, or the total number of births. It's dependent on the observed shape of the curve (exponential), and the "true enough" reasoning that 90% will live in the last 90%. We need no other numbers at all.


Do I assume that I'm a typical human being or do I assume that this is a typical year?There is no such thing as a "typical year", since each year contains more humans than the previous year. So you are constrained to reason that you're a typical human. (Of course, you may produce arguments to suggest that you are not a typical human, as Ken G does above, and eburacum45 has done earlier; that's another matter. But I'd suggest that "typical year" doesn't fly at all in this context.)

Grant Hutchison

Ken G
2007-Dec-12, 03:51 PM
But the "castastrophe" in Carter's reasoning isn't driven by the value of my birth number, or the total number of births. It's dependent on the observed shape of the curve (exponential), and the "true enough" reasoning that 90% will live in the last 90%. We need no other numbers at all.It's not the number itself, it's the scale that matters. We do need the scale, or there's no "catastrophe". If all we say is that we are 90% likely to be among the last 90%, that's all well and good. What we cannot do is say that the total population N is 90% likely to be less than 10*M, where M is our current birth number. That does invoke that scale, even if we personally have no idea what M actually is.

Let's take an example. Let's say that intelligent species face a critical moment when they develop nuclear weapons. Half wipe themselves out at birth number around 10 billion, and half figure out how to deal with it and typically make it to birth number 10 trillion (just make believe this, as an example). Now, it is still true that over both those subsets, 90% of the people will be in the last 90%. But if you go to someone who has birth number 5 billion, that sets a scale for the "catastrophe". Now ask them, what is the chance their species' birth number will exceed 50 billion, Carter would say only 10%, but the correct answer is 50%. Now you might say that the additional information about the N distribution has enabled a more precise determination of the probability, but that's tantamount to saying that "everything that can either happen or not has a 50% chance if we know nothing else about it". What is actually true is that if we know nothing about it, we cannot assert a meaningful probability-- you have to be able to make assumptions about what you don't know or you cannot use probabilities, they mean nothing.

Note that we don't need to know our own birth number to apply the above logic. It is still just plain false to say that there's a 90% chance the total number of humans that will live is less than 10*M, if our birth number is M and we don't happen to know M (indeed, we don't). You still have to use M even if you don't know what it is (the scale must appear in the answer, or there's no "catastrophe" to worry about). Try some distributions for N and you can verify that it is not true that 90% of the total numbers will always be less than 10*M-- that will only be true if you average over M, which is very much begging the question of the Carter catastrophe. This is the point, if we say Carter is right only when we average over M, the "catastrophe" is gone-- there's no scale any more. It all says a lot about what probability is-- and what it isn't.

It's the same with the timescale of course-- one must invoke what the e-folding timescale is before one gets a sense of "catastrophe" in the time domain-- there always has to be a scale in your mind, even if you don't numerically specify it. Carter can say "we are 90% likely to be in the last 90%" of any set from which we are generically chosen, after averaging over all distinguishing subfeatures of that set (even in the entire universe as a whole, not just human). But that is all-- there can be no other information, no scale in time or number, that is not being averaged over and must not appear in the answer.

And that's true separately from the other big problem-- it's far from clear that "humanity" is the set from which we are generically chosen, such that "M" for humanity is what has been averaged over. What personal attribute of ours is really the thing that identifies our generic set, all the rest having been averaged over? Our height, age, expected lifespan? If 100 years from now, the life expectancy is 1000 years (somehow), does it mean that people who think about the Carter conjecture would be weighted toward those, as they have more opportunity to have this conversation? What if tomorrow you write a very influential book on the Carter conjecture, such that it becomes a household word for billions of people (I expect some royalties there). Are we generically sampled from the set of people who have ever thought about this, even though probably up to now only a few million people at most have? Either of these two main objections put the lie to the Carter catastrophe idea, it's simply an example of how probability can not be used. You have to know more about what you don't know, about what is being averaged over, before probability has any meaning.

Chuck
2007-Dec-12, 04:17 PM
But the "castastrophe" in Carter's reasoning isn't driven by the value of my birth number, or the total number of births. It's dependent on the observed shape of the curve (exponential), and the "true enough" reasoning that 90% will live in the last 90%. We need no other numbers at all.

There is no such thing as a "typical year", since each year contains more humans than the previous year. So you are constrained to reason that you're a typical human. (Of course, you may produce arguments to suggest that you are not a typical human, as Ken G does above, and eburacum45 has done earlier; that's another matter. But I'd suggest that "typical year" doesn't fly at all in this context.)

Grant Hutchison

But if every year contains more people than the previous year then they're all typical. I don't see how pointing out such an extreme similarity in any way shows that something is not typical. The only nontypical year would be one that has fewer people, such as just after a war or a plague.

grant hutchison
2007-Dec-12, 04:47 PM
But if every year contains more people than the previous year then they're all typical.Well, if you are a year, they're typical. But you're not a year: you're a person, sampled from the Sea of Souls, who is more likely to have been born in a year when lots of people were born than in a year when fewer people were born.
So what's a "typical" sort of year for people to be born in? 10000BCE? 2000CE?

Grant Hutchison

grant hutchison
2007-Dec-12, 04:51 PM
This is the point, if we say Carter is right only when we average over M, the "catastrophe" is gone-- there's no scale any more. It all says a lot about what probability is-- and what it isn't.OK. That's us back to where we ended up the last time. :)

Grant Hutchison

Chuck
2007-Dec-12, 09:55 PM
Well, if you are a year, they're typical. But you're not a year: you're a person, sampled from the Sea of Souls, who is more likely to have been born in a year when lots of people were born than in a year when fewer people were born.
So what's a "typical" sort of year for people to be born in? 10000BCE? 2000CE?

Grant Hutchison
The fact that I was born recently instead of later doesn't mean there's less chance of there actually being a lot more people later. It's like drawing a ball from a 100 ball urn or a million ball urn and throwing it back if its number is higher than 100 and drawing again. If very few higher birth numbers than mine can be chosen now then it tells us nothing about how long humanity will last.

grant hutchison
2007-Dec-13, 01:14 AM
The fact that I was born recently instead of later doesn't mean there's less chance of there actually being a lot more people later. It's like drawing a ball from a 100 ball urn or a million ball urn and throwing it back if its number is higher than 100 and drawing again. If very few higher birth numbers than mine can be chosen now then it tells us nothing about how long humanity will last.I'm pretty sure I don't understand what you're saying, here.
But I notice "typical year" is no longer mentioned. Have we reached a resolution on that? :)

Grant Hutchison

Chuck
2007-Dec-13, 02:34 AM
If I knew that human population would increase geometrically and then end suddenly and that a time traveling alien chose a year at random and asked the first person he met if the end were probably near, I think the correct answer would be no because most years aren't near the end. If the alien picked a human at random with equal probability then the answer would be yes because most people lived near the end. A typical year would be one in which I had no additional information concerning our probably extinction. If I were in a typical year and an alien appeared and asked me if the end were near but didn't mention how I'd been selected, how should I answer? He might have chosen me at random, chosen this year at random, or used some other method of selection. The best I could do is say that I don't know.

There is no alien here that I can see, so how was I selected? It appears that I chose myself, but that hardly seems random. Was I chosen by happening to read the question? That's somewhat random but I can't have been chosen at random from the set of everyone who will ever live. People who haven't been born yet could not have been selected. That's like not being able to draw a high numbered ball from the large urn. From my point of view I'm in front of the 100 ball urn or the million ball urn but the only balls I can reach are numbered 90 to 100 no matter which urn I'm drawing from. Drawing ball 95 doesn't tell me which urn I drew from.

If I consider myself to have already been selected at random by circumstances beyond my control and I want to answer the question accurately, which answer should I give? "The end is near" will ultimately be the correct answer for most people but it will have been the wrong answer most of the time. Knowing that the end will be near for most people some day doesn't seem to help me now since I might still be in the time in which the end is not near, which is most of the time. Since we're assuming that I have no indication other than my birth order, I don't see how I can make an accurate prediction.

I'm not sure what it means to think that I'd have had a greater chance to be born in the future because more people will be born then. It's unlikely that my genetic duplicate will ever exist again and even if one did, he'd grow up in a different environment and be a different person. I don't think I could exist anywhen else but now. My being here has nothing at all to do with how many people will exist in the future unless it's possible that I could have been one of them instead of being myself, in which case my being here means a lesser chance that they exist. But the existence or absence of anyone in the future did not change the probability of me being here in the slightest.

HenrikOlsen
2007-Dec-17, 01:19 AM
I am currently re-reading Stephen Baxter's "Time". In this book, he uses (not necessarily endorses) the so called Carter Catastrophe. As Baxter is an author who usually has his science right (if on the speculative side), his use of this doomsday argument - even as a plot device - is heckling me.

The argument runs as follows (extract from "The Doomsday Argument" on www.anthropic-principle.com (http://www.anthropic-principle.com)):


Imagine that two big urns are put in front of you, and you know that one of them contains ten balls and the other a million, but you are ignorant as to which is which. You know the balls in each urn are numbered 1, 2, 3, 4 ... etc. Now you take a ball at random from the left urn, and it is number 7. Clearly, this is a strong indication that that urn contains only ten balls. [...]

But now consider the case where instead of the urns you have two possible human races, and instead of balls you have individuals, ranked according to birth order. As a matter of fact, you happen to find that your rank is about sixty billion. Now, say Carter and Leslie, we should reason in the same way as we did with the urns. That you should have a rank of sixty billion or so is much more likely if only 100 billion persons will ever have lived than if there will be many trillion persons. Therefore, by Bayes' theorem, you should update your beliefs about humankind’s prospects and realize that an impending doomsday is much more probable than you have hitherto thought.

In Baxter's book, one character calculates based on this argument that Doomsday is 150, maximum 200 years away.

Deep in my gut I have the feeling that this argument is fundamentally flawed (read: rubbish), but I can't come up with a clean and neat rebuttal. The best I can come up with: As the human population has grown roughly exponentially in the known past, exactly the same argument has been valid for each and every generation before us - for instance also for the generation that lived, say, 300 years ago. And yet we are here, 300 years after an imminent doom.

What are your takes on this argument?
Is is nonsense because in the argument you use a guess for the total number of people which will live over the lifetime of the human race plus how many have already lived, to get a statistically expected number of people over the lifetime of the human race.

Essentially he's deriving a number from a guess at its value.

Ken G
2007-Dec-19, 07:32 AM
OK. That's us back to where we ended up the last time. :)

I believe so. But did you see my post about the game where first the gamemaster invokes some unknown distribution to set N, and then hands us a random number from 1 to N? Surely this is a canned enough problem with obvious similarities to the Carter conjecture, so my question is, will you agree that if you look at your number, and it is M, that it is false that there's a 50% chance that N < 2*M? The correct answer depends on the distribution over N, call it p(N), since
prob that N < 2*M = (sum over all N > M-1 and N < 2*M of p(N)/N) divided by the (sum over all N > M-1 of p(N)/N )
Agreed? So what's clear is that the probability is not 50% if p(N) is known, the question is, is it 50% if p(N) is not known? The answer depends on how the data is "sliced". If you look at every time the game is played, is there a way to slice the data such that it is not true that 50% of the M are > N/2? The answer is yes-- slice based on M. In other words, look at every trial where the same number M was chosen, and ask, is M > N/2 in half those trials? The answer will certainly be "no". It is only "yes" if you slice based on N (so average over M), or take all trials (so average over N and M). So the question boils down to, does the Carter catastrophe idea slice the data based on N or on M? It is sliced based on M, clearly. In other words, if you and I are birth number 15 billion or so, then we must group ourselves with all the other beings in the history and future of this galaxy whose birth number is also 15 billion. Then we must ask, will 50% of that group lie in the last 50% or their species? Answer: no. Agreed?

Ken G
2007-Dec-19, 09:02 AM
Or put differently, the flaw is already apparent in the association:

Imagine that two big urns are put in front of you, and you know that one of them contains ten balls and the other a million, but you are ignorant as to which is which. You know the balls in each urn are numbered 1, 2, 3, 4 ... etc. Now you take a ball at random from the left urn, and it is number 7. Clearly, this is a strong indication that that urn contains only ten balls. [...]

What is bogus here is imagining that the urns are similar to the population situation, because with the urns we have a clearly valid assumption that each urn choice is equally likely. Why can we assume this for the populations? Imagine that the urn containing a million balls is a billion times more likely to be chosen, for whatever reason. Can we still say that if we see the ball "7" that it is a "strong indication" the urn with ten balls was chosen? We cannot say that, without the implicit assumption about the urn likelihood-- and we cannot make that assumption for populations, we simply do not have the necessary information to do so. We can expect that our selection of "7" argues against the million-urn being vastly the more likely choice, but we cannot assess the likelihood that we have the 10-urn, without knowing more about the mechanics of the choice process. For example, let's say instead of choosing an urn, we mix all the balls together, and the balls from one urn are blue and from the other are green. Then we choose a blue "7" at random-- my question is, was it the 10-urn or the million-urn that had the blue balls? We have no way to say. So how is my specification not the valid way to look at the population issue-- why is the urn approach more valid?

BioSci
2007-Dec-19, 05:56 PM
I believe so. But did you see my post about the game where first the gamemaster invokes some unknown distribution to set N, and then hands us a random number from 1 to N? Surely this is a canned enough problem with obvious similarities to the Carter conjecture, so my question is, will you agree that if you look at your number, and it is M, that it is false that there's a 50% chance that N < 2*M? The correct answer depends on the distribution over N, call it p(N), since
prob that N < 2*M = (sum over all N > M-1 and N < 2*M of p(N)/N) divided by the (sum over all N > M-1 of p(N)/N )
Agreed?

I do not understand your argument - you first state that one selects a "random number from 1 to N" and then you argue that the distribution of such numbers is not simple or continuous but may be some unknown function. That would seem to conflict with the usual expectation of the meaning of selecting a "random number."

The strength of the Carter hypothesis is that its statistical conclusion is correct when you know nothing about the likely future distribution of a population and assume that one is taking a "random" sample.
The problem with the Carter hypothesis is that it is only valid if you know nothing about the likely future distribution of a population and therefore assume that your sample is a random selection.
So, if you have some valid argument or evidence regarding the likely future distribution of a population, then the simple Carter statistical argument no longer applies.

Ken G
2007-Dec-19, 06:23 PM
I do not understand your argument - you first state that one selects a "random number from 1 to N" and then you argue that the distribution of such numbers is not simple or continuous but may be some unknown function. That would seem to conflict with the usual expectation of the meaning of selecting a "random number."Let me clarify-- first N is selected by the gamemaster, and that selection is what has an unknown distribution for us. After N is chosen, then we get a random number from 1 to N. The latter is evenly distributed, so the whole game exactly mimics the Carter situation, but is more conducive to mathematical scrutiny.


The strength of the Carter hypothesis is that its statistical conclusion is correct when you know nothing about the likely future distribution of a population and assume that one is taking a "random" sample.There is no "strength" to the Carter hypothesis. It can only be one of two things depending on how far you take it: 1) the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen, and 2) the false probability argument that this says anything about the number of humans that will be born, given the number that have been. So the problem is not in the "random sampling", it is in the incorrect use of probability concepts-- and the mathematical game I described shows this, just take any interesting distribution over N that you like and ask if 50% of the times that a given specified M is chosen, will that M be > N/2. It will not-- unless you average over M, but then you can't use M to create the whole catastrophe concept.


The problem with the Carter hypothesis is that it is only valid if you know nothing about the likely future distribution of a population and therefore assume that your sample is a random selection.It is a common misconception about probability that it has something to do with your knowledge. The choice to include knowledge is yours when you do a probability calculation, there is nothing "automatic" about it. All probability calculations are subject to the assumptions you put in-- knowledge is irrelevant, except that it is normally assumed that you will use all your knowledge in building your assumptions. When people think that probabilities depend on knowledge, they get all confused about under what situations do the probabilities change, like if you forget what cards have been shown in poker, does your probability of winning change? Answer: there is no unique concept of a probability of winning, it all depends on the calculation you choose to make.


So, if you have some valid argument or evidence regarding the likely future distribution of a population, then the simple Carter statistical argument no longer applies.It still applies, if you choose not to use any argument or evidence-- on the grounds that it would be of suspect reliability (which it would). The problem with Carter is much more fundamental-- it is wrong probability if you are selecting on the basis of the value of M.

Chuck
2007-Dec-19, 08:25 PM
The reason that I likely drew ball number 7 from the ten ball urn rather than the million ball urn is that the million ball urn had a lot more balls that I could have drawn making it less likely for me to have selected a number less than 11. It works because the other 999,990 balls were available for drawing. If I could reach only balls 1 to 10 in either urn then drawing ball number 7 would tell me nothing about which urn I selected. Since people who haven't been born yet aren't available for drawing, the selection of me here and now tells us nothing about how many more people are likely to be born.

BioSci
2007-Dec-19, 08:26 PM
Let me clarify-- first N is selected by the gamemaster, and that selection is what has an unknown distribution for us. After N is chosen, then we get a random number from 1 to N. The latter is evenly distributed, so the whole game exactly mimics the Carter situation, but is more conducive to mathematical scrutiny.

Yes, but to then add the aspect of only looking at pre-selected values of M no longer matches the Carter situation.


There is no "strength" to the Carter hypothesis. It can only be one of two things depending on how far you take it: 1) the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen, and 2) the false probability argument that this says anything about the number of humans that will be born, given the number that have been. So the problem is not in the "random sampling", it is in the incorrect use of probability concepts-- and the mathematical game I described shows this, just take any interesting distribution over N that you like and ask if 50% of the times that a given specified M is chosen, will that M be > N/2. It will not-- unless you average over M, but then you can't use M to create the whole catastrophe concept.

The Carter hypothesis does not select a given "M" and so this argument does not apply.
The fact that "the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen" is the power of the Carter hypothesis when simply combined with an exponential population curve and assumption that all known species have a finite existence. That is the strength of the argument - that a statistical estimate can be made with so little input. It is also the weakness of the argument because it requires one to ignore any additional knowledge.



It is a common misconception about probability that it has something to do with your knowledge. The choice to include knowledge is yours when you do a probability calculation, there is nothing "automatic" about it. All probability calculations are subject to the assumptions you put in-- knowledge is irrelevant, except that it is normally assumed that you will use all your knowledge in building your assumptions. When people think that probabilities depend on knowledge, they get all confused about under what situations do the probabilities change, like if you forget what cards have been shown in poker, does your probability of winning change? Answer: there is no unique concept of a probability of winning, it all depends on the calculation you choose to make.

Well, it certainly is important to include knowledge into an probability calculation if such knowledge is available to improve the validity and reliability of the calculation. Certainly, one's calculation is only as good as the assumptions that went into the calculation. If knowledge is available but not used, then any such calculation is limited by the assumptions that were used (are the dice loaded?)


It still applies, if you choose not to use any argument or evidence-- on the grounds that it would be of suspect reliability (which it would). The problem with Carter is much more fundamental-- it is wrong probability if you are selecting on the basis of the value of M.

If you chose not to use any other information to help describe the likely population distribution (and have a value of M as a result of an essentially random selection {not-pre-selected} from a finite population), then the Carter estimate is the best you can do - but it is fatally weak because you are purposely not using important information that may have significant impact on the likely population curve and expected specie longevity.
(It is like expecting the normal odds of getting a 7 is a valid statistical argument even when you know that the dice are loaded but you just choose to ignore that information and calculate odds assuming that you don't know they are loaded).

Ken G
2007-Dec-20, 12:47 AM
Yes, but to then add the aspect of only looking at pre-selected values of M no longer matches the Carter situation.
I see it as precisely the Carter situation. The "pre-selected value" of M is our own birth number, the scale for the predicted continued longevity of humanity that leads to the "catastrophe". In my game, the selection of M is the same as the selection of "you", out of the N humans that will live. It is identical to the Carter argument, phrased in terms of birth number (which is the only valid way to phrase it without assuming strange things about exponential growth that never hold in any system).


The Carter hypothesis does not select a given "M" and so this argument does not apply.See above.


The fact that "the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen" is the power of the Carter hypothesis when simply combined with an exponential population curve and assumption that all known species have a finite existence.But that "combination" with an exponential curve is kind of ridiculous anyway. There are hosts of systems that undergo temporary exponential growth, then settle into some other pattern. To use that predictively is to be ignorant of a vast fraction of reality.

That is the strength of the argument - that a statistical estimate can be made with so little input. It is also the weakness of the argument because it requires one to ignore any additional knowledge.The argument falls completely flat even in the absence of any additional knowledge. It makes assumptions that are unjustified, and its own assumptions are its weakness.

Well, it certainly is important to include knowledge into an probability calculation if such knowledge is available to improve the validity and reliability of the calculation.Agreed, this is normally part of the value of using probability. But it doesn't "change the probability", it merely changes the inputs we choose to include. The calculation is always purely our own choosing, it has no "reality" to it, and so cannot "change" in some absolute kind of way.


Certainly, one's calculation is only as good as the assumptions that went into the calculation. If knowledge is available but not used, then any such calculation is limited by the assumptions that were used (are the dice loaded?)Yes, I agree.



If you chose not to use any other information to help describe the likely population distribution (and have a value of M as a result of an essentially random selection {not-pre-selected} from a finite population), then the Carter estimate is the best you can doNo, you cannot even do that well, it's just wrong probability to apply it to classes of similar M.

Chuck
2007-Dec-20, 01:25 AM
It would seem that the argument could never predict an early doomsday because you must use only the birth number as data and ignore any additional knowledge. But the fact that the population is increasing geometrically is additional knowledge and it can't be ignored because the early doomsday prediction depends on it.

BioSci
2007-Dec-20, 01:40 AM
No, you cannot even do that well, it's just wrong probability to apply it to classes of similar M.

I will repeat one last time - that is not what the Carter probability is doing. It does not apply probability to multiple values of a specific "M" to estimate total "N." It applies a statistical argument to any one individual. There is no dependence on the value "M" for the argument.

It simply uses the "the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen" and combines with our observed exponential growth to arrive at a simple statistical probability that the end is likely near.

If you use no other information, it is a valid statistical estimate. If you think there are other arguments for probable human life existence, that the shape of our population curve will change in the future, or that your specific birth order is somehow special (not generic), then the simple, Carter (no additional knowledge) estimate will not be accurate to the extent you believe your other knowledge is informative and/or likely to be correct.

Audios.

Ken G
2007-Dec-20, 08:22 AM
I will repeat one last time - that is not what the Carter probability is doing. It does not apply probability to multiple values of a specific "M" to estimate total "N." It applies a statistical argument to any one individual. There is no dependence on the value "M" for the argument.
What you are failing to recognize is that applying it to an individual is invoking an M value-- it is simply the birth number of that individual, and it is used to specify the entire catastrophe scenario. Let me put it another way. Let's say there are a million different intelligent species in the universe, all of whom think of the Carter catastrophe at some point. Let us go into the minds of the subclass of those individuals whose birth number is, say, 15 billion, which is kind of like our own. Now let us ask a simple question: do we expect that 90% of that subclass are living in the last 90% of the number of their kind?

The answer to this, if you understand probability, is: no, we simply have no idea what that fraction will be, which is easily verifiable by simply choosing some distributions of N for those million species and checking my claim that it will not yield 90% (try it, really, you probably can't understand what I'm saying until you do). Saying that 90% is therefore the best we can do is like saying that anything can either happen or not, so everything has a 50% chance of happening if we know nothing else. That's wrong use of probability, which is different from correct use of probability in the absence of much information. The Carter argument, if it is applied to us at this moment in human history, i.e., to this M, is simply wrong use of probability.


It simply uses the "the plainly obvious statement that 90% of beings live in the last 90% of any set from which they are generically chosen" and combines with our observed exponential growth to arrive at a simple statistical probability that the end is likely near.I already addressed the fallacy of "combining" that with any expectation of continued exponential growth. On what basis would we choose that combination, pray tell? But that's another issue, I prefer to think of the Carter conjecture in terms purely of birth number, for that requires less absurd assumptions and is therefore wrong for far more subtle reasons.


If you use no other information, it is a valid statistical estimate.That's like saying that if I think of a number, and you try to guess it, you can either be right or wrong. Since there are two possibilities, and you have no other information to go on, the best you can do is assume you have a 50% chance of being right. Shall we call that a valid statistical estimate, and blame our lack of information? No, when we lack all information, we also lack the very meaning of probability.


If you think there are other arguments for probable human life existence, that the shape of our population curve will change in the future, or that your specific birth order is somehow special (not generic), then the simple, Carter (no additional knowledge) estimate will not be accurate to the extent you believe your other knowledge is informative and/or likely to be correct.So Carter is using no information to say it will continue to be exponential, but I'm bringing in something new if I ask for a reason to accept that assumption? Not.

Ken G
2007-Dec-22, 12:13 AM
This thread appears to be returning to dormancy, so I'll summarize the interesting issues that have emerged. It is a point of some frustration to me that I have made the following mathematically bulletproof arguments yet objections have been raised that don't seem to understand the fundamental points I'm making, so I'll try one last time to condense them into the best and purest form I can, and the reader can take it or leave it as they like. I personally think there is a lot to be learned about what probability can be used for, and even more interestingly, what it can't.

Bottom line: there are two very different ways to state the "Carter catastrophe" conjecture, and they are both examples of bad probability, even in their purest forms with no consideration of whether or not our asking the question itself changes anything (which is a severe problem for the conjecture but I don't need it). Here are the two ways, and what is wrong with them:

1) the conjecture stated in terms of birth number: This goes something like this. 90% of humanity will have a birth number that will ultimately prove to be in the last 90% of humans born. Thus any human may expect with 90% certainty that they fall in the last 90%. That is true as stated, but it is no longer true if that human uses their own birth number (let's say ours is about 15 billion) to infer a coming "catastrophe" in the next 150 billion (to achieve the incorrect 90% likelihood). I have referred to this as "using the M value" to make predictions about N. Others have claimed that is not part of the "catastrophe" scenario, but they have not justified that claim with any suggestion of how there is a "catastrophe" without it. You see, the only way to not invoke M is for a human who has no idea what their birth number is, even vaguely, to then say "I'm 90% likely to be in the last 90% of humans". Note that is perfectly true, but see the important difference? Where's the catastrophe! If the person's birth number might be 100 trillion, for all they know, then humanity could easily live to a quadrillion. Or a million times that-- this is what it means to have no idea what your birth number is, and is clearly violated by the Carter logic. If any use is made of our actual birth number around 15 billion, all bets are off-- the probability argument is simply false at that point, as is easily verified by choosing any arbitrary distribution over N that you like.

2)the conjecture stated in terms of exponential growth rate: Here it goes something like, since humanity is growing exponentially with an e-folding time of (let's say) 50 years, then a version of the birth number argument states we are 90% likely to be within about 2.4 efolds of the end of humanity, say less than 120 years. This version does not require an M value, so takes advantage of the magnitude-free form of an exponential distribution, but it fails for far less subtle reasons. After all, who in their right mind would think it is a valid expectation that if humanity has been e-folding every 50 years for the last handful of centuries, that it should continue to do so for the next handful? In five minutes I could list 100 contradictions to that assumption from everyday experience, it's no better than the pseudoscientific arguments used by creationists (indeed, they do use invalid extrapolations all the time). All you can say is that any exponential distribution cannot be generically expected to extrapolate more than a few efolds into the future, to which I say: duh. But that's not a "catastrophe", merely an expectation that the magnitude-free form of human population growth should be expected to change fairly soon. Again, duh. (Note that the argument goes through perfectly well if the growth slows to, say, an efold of 100 years-- no catastrophe there. We can only be said to be generically chosen from the subclass that shares our growth rate-- unless we make specific reference to M to obtain a catastrophe prediction.)

Rydberg
2007-Dec-25, 12:17 AM
To me, the Carter Catastrophe reads like wishful Christian thinking wrapped in statistical pseudoscience. It's based on the assumption that we, ie: you or me or anyone alive today, has the equal (or in fact, much greater) probability of living in a time of greater population (hence, a possible distant future). Statistically speaking it is sound...if one completely ignores reality (as, unfortunately, so many of these type of arguments do).

The reality it is ignoring is simply this...I'm alive now (or you are alive now, or my wife is alive now...) because there is no other time that any of us could be alive.

That almost sounds circular or anthropic in nature, but I assure you it's not. Rather, I'll call it the Bio-historical rebuttal.

What I mean is this: who I am is a product of my precise biology--beginning, most fundamentally, with my genetics, which forms the foundation of 'me'. From that, of course, follows the environmental factors that fine-tuned my personality. But the genetics, the precise combination of that one sperm and egg are the ONLY combination that could have produced 'me'. Any other combination, even from my biological parents (as those of you with siblings clearly know) would not and could not produce 'me'. Obvious, my parents are also the results of similar events as were their's, back through time. From this it is trivial to see that, not only could I only be born to those parents through that particular combination of genetic material but, therefore, 'I' could only be born in this particular time.

With this understanding, the foundation of the Carter Catastrophe completely crumbles because there is 0% chance of 'me' (or anyone else alive today) being born at any other time, regardless of how many more generations yet may live.

Therefore, at best, the Carter Catastrophe is an amusing game of statistics, at worst...well, maybe that could be the topic for some interesting novels.

Grand_Marquis
2008-Nov-25, 01:01 AM
Goodness! Six pages of argument over something so simple to refute! Oy!!
Here, let me end this silliness once and for all.


Let's turn it into a simple logic problem. You have 2 urns: A and B which contain some numerical value within. There are two numerical values of (X) 10 and of (Y) 1 million, which you must connect to A and B in order to find out if A is greater than or less than B. You will do this by pulling a single numerical value (N) out of urn A.
IF N is greater than 10, THEN N falls outside the bounds of X, THEREFORE A must equal Y
ELSE IF N is less than 10, THEN N falls within the bounds of both X and Y, THEREFORE A = ?

if you suggest anything other than "inconclusive" you would be making an illogical statement. Regardless of probability or statistical likelihood, as long as A could be either X or Y, A cannot be either X or Y to the exclusion of the other. Without more information, there is no way around this. Sorry.


Nobody disputes that a small population is a population in danger. This does not mean you can take that population number, by itself, and suddenly start making predictions about extinction with it. The world doesn't work that way.

Chuck
2008-Nov-25, 02:19 PM
If I draw a ball numbered less than ten then I don't know which urn it came from, but that doesn't mean equal probability for either. The real question is whether or not drawing balls from urns is the same problem as selecting myself as a representative of all the people who will ever live. I think that it's not.

geonuc
2008-Nov-25, 02:33 PM
Goodness! Six pages of argument over something so simple to refute! Oy!!
Here, let me end this silliness once and for all.


Let's turn it into a simple logic problem. You have 2 urns: A and B which contain some numerical value within. There are two numerical values of (X) 10 and of (Y) 1 million, which you must connect to A and B in order to find out if A is greater than or less than B. You will do this by pulling a single numerical value (N) out of urn A.
IF N is greater than 10, THEN N falls outside the bounds of X, THEREFORE A must equal Y
ELSE IF N is less than 10, THEN N falls within the bounds of both X and Y, THEREFORE A = ?

if you suggest anything other than "inconclusive" you would be making an illogical statement. Regardless of probability or statistical likelihood, as long as A could be either X or Y, A cannot be either X or Y to the exclusion of the other. Without more information, there is no way around this. Sorry.
I believe this is called thread necromancy. :)

In any case, sorry, but your logic example is not clear so, at least to me, you haven't ended all this silliness once and for all.

Chuck
2008-Nov-25, 02:38 PM
There seems to be room for a lot more silliness to me.

Ken G
2008-Nov-30, 07:30 PM
Let us attempt to address the "silliness" using a very concrete construction. Imagine a supra-intelligent alien species that is the last surviving intelligence anywhere in the observable universe. A statistician from the species has access to all knowledge of every intelligent population that has come before. Specifically, they know the distribution N(n), where N is the number of populations that gave birth to n members before it went extinct.

Now, let us further stipulate that this statistician uses a random process to select one individual from all those populations, where each individual was equally likely to be the one selected. It seems to me this is precisely the situation at issue in the Carter reasoning. Now, the question is, if we look at the species that this individual was chosen from, it turns out they were individual number m from that population. The question is now, what is the probability that their species ever gave birth to more than 10*m individuals?

The Carter argument claims that probability is 10%. Hence, it must follow that if we repeat this experiment a billion times, in only a billion/10 times will n > 10*m. Unfortunately, that is a false conclusion, because the actual answer depends on the distribution N(n), no assumptions about which have been specified in the usual setup of the problem (essentially because nothing is known about that distribution). For example, if we take a simple example where N(n) corresponds to a 50% chance of n=1011, and a 50% chance of n=1015, then any individual with m < 1011 does indeed have a 50% chance of belonging to either population (as in Grand Marquis' scenario, made more concrete here). Ergo, for that N(m) distribution, when m=1010 (as it does for us), it is false that there is a 10% chance that n > 1011-- the chance is actually 50% for that m value.

Now, it could certainly be argued that in the distribution I am talking about, it is extremely improbable that m=1010 will be chosen. Nevertheless, in the spirit of "someone has to win the lottery", the key point is you cannot ask those people to make a standard probability argument-- they are in a special place. So in that is that actual N(n) in our universe, then we know we have m=1010, so we have no way at all of knowing that we are not special-- if that is really the N(n), then the m=1010 individuals are making exactly the Carter argument, and getting exactly the wrong answer.

The way I sum this up is, you cannot make any probability argument that invokes both the number m and the average n in the same argument-- because if you don't know N(n), you will be reaching a false conclusion if you do that. You can use the average n-- you can say that before you choose m, the chances are that m will not be too different from the average n (but that requires knowing what the average value of n is). Also, if the statistician did not know N(n), and was selecting m value to suss it out, that's also fine-- the first m chosen begins to give the statistician a sense of what the average n is. But nevertheless, when all is said and done, when you bin all the m=1010 individuals together, which is the bin that we are in, you find that 50% of them live in a species that will vastly outlive 10*m.

This binning proves that the Carter argument is simply wrong probability, even if you make the very same assumptions it does. Any group that uses their own m value in their calculations of expectation values is specifying something about themselves that invalidates any claim they would otherwise have of being "generic". This is quite a common error in probability that comes up in a lot of puzzles, some on this forum.