All of Nate Silver's State-Level Polling Predictions Proved True

This actually shows that Silver is poorly calibrated. if he were accurately calibrated, 80% of his 80%-confidence predictions would come true, 50% of his 50%-confidence predictions would come true, etc. But 100% of his >50%-confidence predictions came true. In the future, he should be more sure of his predictions.

Not necessarily. Most of the uncertainty in his predictions was due to the conditional probability of systematic bias in likely voter models. For example, Gallup was showing much better results for Romney and the Rac... err... Republicans across the board, which was probably due to how they screened people who responded.

Systematic error shows up as a conditional probability, so you are lumping together completely disjoint realities into your final result. In terms of discrete conditional probabilites, imagine that based on historical data you have three equally likley possible conditions: 5% Democratic bias, no bias, and 5% Republican bias. You run your simulations with each of these three biases, and you get a result that says senators D0, D1, D2, D3, D4, D5, D6, D7, D8 and D9 are all 80% likely to win. But 99% of that 20% chance they'll lose comes from the condition where there's a 5% Republican bias.

But remember: those three conditions describe completely disjoint realities. They are not sampling error, but statements of ignorance about the actual state of the world.

Now the world really is just one way (it may be ambiguous relative to some human categorization, but then that ambiguity is just part of the one unique way the world realy is.) So only one of the three conditions are true. If it happens that the no-bias case is the way the world really is, then 100% of those 80% chances will come true.

That said, in future elections Bayesian predictions of the kind Silver and everyone else in this space are making will lower the conditional probabilities of bias, because this election demonstrated good low-bias results, but so long as the ultimate uncertainty is dominated by the systematic error, Bayesian predictors will tend to appear either uncannily accurate or dismayingly inaccurate.

However, averaged over many, many election cycles (18 or more) you would expect to get statisics such that 80% of the 80% calls are correct, and so on. But within individual elections that use fixed likely-voter models that won't be the case.

Conditional probabilities are one of the most difficult things for humans to understand (the Monty Hall problem is a classic case where all the confusion comes from treating a conditional probability as if it was a total probability) so it's worth practicing the art of thinking carefully about these things, and the odds are still good I've said at least one confusing or incorrect thing in the above.

Source: http://rss.slashdot.org/~r/Slashdot/slashdotScience/~3/MZqESkX4Kys/all-of-nate-silvers-state-level-polling-predictions-proved-true

Jael Strauss Alison Pill gizmodo cnet britney spears sprint Sam Bacile