Most Read
    Image 01 Image 02 Image 03

    If Nate Silver cannot be wrong, how can he be right?

    If Nate Silver cannot be wrong, how can he be right?

    I have no implicit pro- or anti-Nate Silver bias. 

    He was a journolist and in 2008 got some secret help with his predictions from the Obama campaign, but since joining The NY Times I think he has raised his game.

    I find the whole focus on Silver and his presidential election “model” to be particularly annoying, in part because there seems to be a lack of transparency.  Is the methodology and how it is used to come up with specific percentages fully disclosed?

    Whatever, Silver-mania (both pro and con) seems to have invaded way too many bodies. 

    I was in an SUV last Wednesday night in Austin with a group of people at a conference heading to dinner.  A woman in the car got a beep on her phone, and she exclaimed words to the effect: “I just got an alert, Nate Silver has Obama’s chances of winning at 72.3 percent.”  (I may be off slightly on the percentage, but that’s pretty close.)

    At that moment, if I did not know it before, I understood that this Nate Silver thing had gone way too far.

    I’m not a numbers guy, never claimed to be.  My primary strength when it comes to numbers has been to reduce complicated damages calculations to an explanation so simple that even someone as statistically illiterate as me can understand it.  If I can understand the numbers, anyone can.

    But there’s something that doesn’t require you be a numbers guy or gal which bothers me about Silver-mania.

    Silver cannot be wrong because his model, whatever it is, merely puts a likelihood of success on the election.  Silver’s model could predict a 75% chance of an Obama win, but if Romney wins the model was not wrong because his model allowed for a 25% chance of a Romney win.

    If Silver cannot be wrong, how can he be right?  Heads he wins, tails you lose.  Calling all philosophy majors!

    I’ll leave you with this.  My first experience with Silver-mania was in his pre-NY Times days, when he was covering the Coakley-Brown race.  According to Wikipedia, Silver’s 538 model was pure genius:

    FiveThirtyEight writers Schaller, Gelman, and Silver also gave extensive coverage to the January 19, 2010 Massachusetts special election to the U.S. Senate. The “538 model” once again aggregated the disparate polls to correctly predict that the Republican Scott Brown would win.[40]

    The source in the footnote was Silver’s January 18, 2010 blog entry, 538 Model Posits Brown as 3:1 Favorite posted at 5:26 p.m. the day before the election.

    Well, yeah, duh, about 14 hours before the polls opened Silver joined the rest of the world in predicting a Brown win, but he was consistently predicting doom for Brown and was the last person to jump on board.

    Here’s the pertinent chronology from my post in February 2010, Beware Statisticians Bearing Advice:

    Nate Silver, January 5, 2010, Wicked Awesome Thoughts on Massachusetts Special Election:

    Rasmussen is supposed to have a poll out tomorrow (Tuesday) on the Massachusetts Senate Special Election, which will take place on the 19th. There’s been some speculation, mostly from Republican blogs but also from some Democratic analysts, that the Republican candidate, Scott Brown, might have a chance, which would potentially wreck the Democrats’ chances to pass health care reform.

    I’ll be curious to see what Rasmussen and the other pollsters (PPP? Suffolk?) have to say, and bears remembering that special elections are highly unpredictable affairs. But I’d be somewhat surprised if the election turns out to be especially competitive.

    Nate Silver, January 14, 2010, OK, It’s a Toss-Up:

    Earlier today I tweeted about how there wasn’t enough evidence to describe the Massachusetts special election as a “toss-up”, as some other forecasters have done, based on the information available to us at that time.

    Well, now there’s some new evidence. And it isn’t good for Martha Coakley.In particular, the evidence is a Suffolk University poll that shows the Republican, Scott Brown, ahead by 4 points, 50-46.

    Nate Silver, January 17, 2010, 538 Still Rates Massachusetts as Toss-up:

    Please don’t be too enthralled with/scared by the specific numbers below. They’re based a number of assumptions which may not be valid. However, I agree with the characterization that the model comes to, which is that the Massachusetts special election should continue to be regarded as a toss-up.

    Nate Silver, January 19, 2010, Defying Odds, Republican Brown Becomes Next Senator from Massachusetts:

    In an outcome that would have been unthinkable just weeks ago, Republican Scott Brown has become the Senator-elect from Massachusetts.

    Silver was as wrong as anyone could be in the Brown-Coakley race, yet came out a hero because his model predicted a 75% chance Brown would win late in the day before the election.

    I neither a Nate Silver lover nor hater be. 

    But when someone has a model which cannot be proven right or wrong, I’m not buying whatever it is he’s selling.


    Donations tax deductible
    to the full extent allowed by law.



    It seems to me that to really test Silver’s election eve predictions, you would want to look at a large number of them and see if the favored/non-favored split breaks down the way he says. In other words, if he says there is a 75% chance that Will Ferrell wins, then for Silver to be right, over time the Ferrells of the world should win 75% of the time and the Galafinackises should win 25%.

    Of course, even if you could set that up, and even assuming his model doesn’t change, that only tests his election eve predictions. His pre-election predictions are untestable.

    Evan3457 | October 30, 2012 at 3:28 pm

    There are really two problems with Silver’s model. Let’s stipulate for the moment that his weighting of various polls based on several factors (track record, number polled, how recent, and I’ll even let him slide on his fudge factor, which is whether the result of the poll seems in line with the way he or his system perceive how the race is going in a particular state at the moment the poll was taken). Even so, there are still two problems.

    The first is that he had access to the Obama internal polls last time. That helped his accuracy. Obama’s people think they’re leading all the battleground states according to their internal polling. What if their turnout model is wrong? What if they’re lying to Silver this time?

    The second is larger and more difficult for Silver’s model to handle. What if the media polls his model relies on, or a great majority of them, have bad turnout models this time? For example, they could be oversampling early voters (and they are) in states where their poll says Obama is way out in front with early voters (and they are, especially in Ohio). What if a poll organization he says was inaccurate, or has a “house bias” in past elections, is deliberately adjusting its model to try to eliminate that error?

    Let’s say a pollster is polling party ID and decides that although it was D+7 for the 2008 election, it’s now R+2. And maybe this pollster says, hmmm, last time, most of my state polls understated Obama’s support. Maybe I’ll adjust my turnout model to D+4, even though my own party ID poll says R+2. Maybe I’ll be closer to the mark this time. Let’s call this hypothetical pollster, oh, I dunno…Masrussen.

    Why, then, Mr. Silver’s model wouldn’t take that into account, would it? It would be stuck with the same weighting, because it doesn’t know that Masrussen is adjusting (and probably over-adjusting) to try to fix it’s problems from 2008.

    And then Mr. Silver’s model would give less weight to a pollster who probably is closer to the actual turnout than those who are forecasting the same turnout as last time (and in some cases, a bigger D advantage than last time, especially in Ohio).

    And then Mr. Silver’s model is in a position to fall flat on its face a week from today.

    Shorter version of the above 7 paragraphs: Garbage In = Garbage Out.

    Jeffrey | October 30, 2012 at 5:22 pm

    Nate silver’s projections consistently are wrong until the day before the election and suddenly they are then right.

    Sorry, but thats just cheerleading in disguise until the final moment.

    Move along

    The way to measure Nate Silver (and the other quants) isn’t the top-line number: it’s his projected EV count and state-wins. If he gets 48 or 49 states right (or, uh, 50) then he’s accurate. If he lands, say 47 or below then he’s probably in big trouble for 2016 in terms of credibility.

    You can do this for other sites as well. I recommend Election Projection for quants-from-the-right (the guy also does his version of unskewing which is a lot better than the general unskewing and comes with its own testable hypothesis).

    As someone who’s day job is in big data (but: not a quant) I do respect this kind of analysis so I generally like what the math guys are doing, even if I don’t always agree with the outcomes–but just keep in mind that what Silver is selling is NOT a 70% chance of victory–that’s just the top-line-people-demand it. It’s the individual state counts.

    Over 10 or so swing states he “should” hit his numbers more often than not so he’s reasonably testable there.

    WestLooper | November 8, 2012 at 7:15 pm

    This thread is priceless, given what actually happened. Silver got 50 out of 50 states correct, and also came quite close on many vote share projections.

    Leave a Comment

    Leave a Reply

    You must be logged in to post a comment.

    Notify me of followup comments via e-mail (or subscribe without commenting.)

    Font Resize
    Contrast Mode
    Send this to a friend