Canadian polling is notoriously bad as of late, and it’s fair to suspect that Thursday’s Ontario election will differ wildly from what Ekos or Ipsos are projecting. So let’s break this down, how do polls work, and why are they just so bad? Then, we’ll take a look at Ontarip’s numbers.
The Problem With Polling
You can pick up a phone and call 1,000 people right now, asking for whom they would vote. Many will hang up, many won’t be home. Of those that do answer your questions, they may be undecided, they can change their mind between now and the election, they could be straight up lying to you, or they may not vote at all. So the question becomes, of the people who happened to be home at that precise time (or happened to have land-lines), and happened to answer my questions, are they going to vote at all? And if they do vote, will they vote for the person for whom they said they’d vote? This is the difference between an “eligible voter metric” (everyone who gave you an answer) and a “likely voter metric” (people who will likely actually vote come election day).
We never know who is going to decide to stay home that day, but pollsters still try to predict it, often with contradictory results.
Canada: A Brief History of Bad Polling
The website ThreeHundredEight.com does an excellent job of collecting all the different polls that pollsters put out, and aggregating them (putting the data together). We’ll be using their numbers below. Polls are weighted according to how they were conducted (their methodology), how many people were contacted (the sample size), and how confident they are in their numbers (the margin of error). In recent years we’ve gotten it all wrong a few times, and here are just some examples:
Going into the election, ThreeHundredEight said the results would be 38% for Wildrose, 36% for the Progressive-Conservatives, 11% NDP, and 11% Liberal (38/36/11/11 for WLD/PC/NDP/LIB), and the seat distribution would be 43/39/5/0: a Wildrose minority.
The actual results were 34/44/10/10, and the seat count was 17/61/5/4. Look at those numbers: there was a massive PC majority! The PC count was off by 22 seats out of 83! Percentage wise there was an 8% gap!
Forecast: Parti Quebecois/Parti Liberal du Quebec/Coalition Avenir de Quebec/Quebec Solidaire (PQ/PLQ/CAQ/QS), percentage-wise was 34/28/26/7, and seat-wise 63/33/27/2, a PQ majority.
Actual result: PQ/PLQ/CAQ/QS, percentage-wise was 32/31/27/6, much closer, and seat-wise 54/50/19/2, a razor-thin PQ minority. Here the PQ and CAQ support collapsed with a sudden tidal wave to the Liberals.
British Columbia (2013):
Forecast: NDP/LIB/GRN/CON, percentage-wise was 46/38/8/5, and seat-wise 49/35/0/0, an NDP majority.
Actual result: NDP/LIB/GRN/CON, percentage-wise was 40/44/8/5, huh?, and seat-wise 34/49/1/0, a Liberal majority.
Here a 6% change in the polls led to a swap in the projected number of seats. A 15 point swing is massive when there are only 84 seats.
Forecast: PQ/PLQ/CAQ/QS, percentage-wise was 27/40/23/8, and seat-wise 45/69/9/2, a Liberal majority.
Actual result: PQ/PLQ/CAQ/QS, percentage-wise was 25/42/23/8, pretty much the same, and seat-wise 30/70/22/3, with the Liberals winning about the same number of seats, but a seismic change in the PQ/CAQ seats. The CAQ picked up 13 seats overnight, and the PQ lost 15?
These are all forecasts made the night (or a few days) before the election. We got the numbers wrong.
This has been described as a race with only bad choices. Kathleen Wynne’s Liberals have been beset by scandals, Tim Hudak’s PCs’ signature platform has seams exposed, and the NDP is pledging to reward job creators. Kathleen Wynne wrote an op-ed yesterday with the headline “A vote for Horwath is a vote for Hudak,” trying to convince NDP-loyalists to jump ship. Safe to say, it’s a mess, and trying to predict a winner when so many people may switch who they’re voting for (or just stay home) is a game just risky enough to be exciting, especially when only 47% of respondents said that they’re not changing their vote before election day. So let’s take a look at competing forecasts.
Forecast of all eligible voters: LIB/PC/NDP/GRN have 34.2/35.5/20.5/7.4 percent, respectively. So, the PCs are leading slightly, but it’s still anyone’s game.
Forecast of all likely voters: LIB/PC/NDP/GRN have 40.7/35.7/16.3/5.6 percent. Oh, so liberals are more likely to show up then? What a twist!
How “likely voters” were determined: A vote has more weight depending on a variety of factors including past voting behaviour, how strongly they feel about Wynne (equal weighting to hopeful, happy, and angry), intention, and knowing where their polling station is. The emotional component is unique, though it seems to be strangely tilted towards the Liberals.
Forecast of all eligible voters: LIB/PC/NDP/GRN have 39/37/17/6 percent, respectively. So, the Liberals are leading slightly, but it’s still anyone’s game.
There is no forecast of likely voters, so take these numbers as-is.
Forecast of all eligible voters: LIB/PC/NDP/GRN have 35/35/26/4 percent, respectively. So, it’s a dead heat. Wow!
Forecast of all likely voters: LIB/PC/NDP/GRN have 32/40/24/3 percent. Oh, so PCs are more likely to show up then? What a twist!
How “likely voters” were determined: Simply, they only considered the votes of “decided voters,” those who said that “nothing short of an unforeseen emergency could stop me from getting to the voting booth and casting my vote.” It’s a reasonable metric, but doesn’t take into account the demographics of those not polled.
None! Ekos says it’ll be 40/36 for the Liberals, Ipsos says it’ll be 40/32 for the PCs. The NDP can gain or lose support at any moment, and who knows if any of these people have really made up their mind? This isn’t to say that all polling is bad (far from it, there are plenty of essential uses for regional and demographics based polling!), what’s important to consider is that these should all be taken with a grain of salt, and that we shouldn’t be surprised in the event of a Green Party majority.
Want more Beginner’s Guides, breaking down hard to swallow topics into bite sized pieces? Then contribute to our crowdfunding campaign: igg.me/at/truenorthtimes!