Polls are reported with a margin of error that describes only one source of uncertainty, and the others are generally larger.
The sampling principle
A random sample from a population allows inference about the population, with quantifiable uncertainty.
Which is the mathematical foundation, and it requires that every member of the population has a known probability of selection.
That condition is essentially never met in modern polling.
The response rate problem
Response rates to telephone polls have fallen dramatically over decades, from majorities to low single-digit percentages in many cases.
Which means the sample consists of people willing to answer, who differ systematically from those who are not.
The margin of error assumes random sampling and therefore understates total uncertainty substantially.
Weighting
The adjustment that makes modern polling possible.
Samples are weighted so that demographic groups appear in the proportions they occupy in the population.
Which corrects for known imbalances and cannot correct for imbalances on unmeasured characteristics.
Choices about which variables to weight on are consequential — weighting by education became standard after it was identified as a source of error in specific elections.
Likely voters
Most election polls attempt to identify who will actually vote.
Which requires a model, generally based on stated intention, past behaviour and interest.
Different pollsters use different models, and the choice can move a result by several points.
It is among the largest sources of variation between polls of the same race at the same time.
House effects
Systematic differences between pollsters, arising from methodology rather than from sampling variation.
Which means comparing polls from different organisations conflates real change with methodological difference.
Aggregators adjust for measured house effects, which is one of the main services they provide.
Question wording and order
Affect responses measurably, and experiments varying wording produce substantially different results.
Which means the exact question matters and is frequently not reported alongside the result.
Order effects are real — earlier questions prime responses to later ones.
Aggregation
Averaging polls reduces random error and does not reduce systematic error.
Which means if all pollsters share a bias, the average shares it, and this has occurred.
Aggregators using historical error to estimate uncertainty produce wider intervals than the polls themselves imply, which is more honest.
Probabilistic forecasts
Convert polls into win probabilities using models incorporating historical error, correlations between regions and time to the election.
Which are frequently misread — a stated probability is not a prediction, and outcomes assigned meaningful probability occur at roughly that rate.
Communicating this has proven genuinely difficult, and forecasters have written about the problem repeatedly.
Reading a poll
Check the field dates, the population sampled, the sample size, the mode, and the exact question.
Treat differences within the margin as noise.
Prefer averages to single polls.
And expect error larger than the stated margin, since history indicates it.
Modes
Telephone, online panel, text message, mail and in-person all produce different samples and different results.
Which means mode is a substantial methodological variable, and the industry has shifted heavily toward online panels as telephone response collapsed.
Online panels are not random samples of the population, and they rely on weighting and modelling to produce population estimates.
Probability-based panels, recruited by random sampling and then surveyed online, sit between the two approaches.
Non-election polling
Issue polling faces additional problems since there is no eventual result to validate against.
Which means systematic errors in issue polling can persist undetected, unlike election polling where accuracy is checked.
Question wording effects are largest for issues where opinion is weakly held, which is most issues.
Exit polls
Surveys of voters after voting, used for analysis of who voted how.
Which face their own problems, including differential willingness to participate and, increasingly, the difficulty of sampling voters who voted early or by post.
Methodologies have been revised substantially as voting methods changed.
Sample size and subgroups
A poll adequate for a national estimate is inadequate for subgroups.
Which means reported findings about small demographic groups within a standard poll carry very wide uncertainty, frequently not stated.
Crosstabs are the most over-interpreted part of polling coverage.
Push polls
Not polls at all, but campaign contact disguised as research.
Which is identified by the questions being designed to convey information rather than to measure opinion, and by large call volumes.
Industry bodies condemn the practice, and it persists.
Transparency standards
Professional associations set disclosure standards covering methodology, sponsorship and question wording.
Which member organisations follow, and polls not meeting them should be treated with more caution.
Checking whether a pollster discloses its methods is a quick and useful filter.
Reporting standards
Broadcast and press codes in several countries set requirements for reporting polls, including disclosure of methodology and prohibitions on reporting during voting periods.
Which vary considerably, and blackout periods before elections exist in some jurisdictions and not others.