Confidence Intervals Part 1: Uncertainty in Estimation

Suppose you read this in the news: 23% of Canadian households experienced food insecurity in the past year.

You may even read that "This estimate is based on a random sample of 2000 Canadians, plus or minus 2 percentage points, 19 times out of 20." Let's break down what this statement actually tells us, because it's not just filler. It's how we communicate both what we know, and what we don't know.

Let's start with the point estimate. What's a point estimate you ask? Well, this is the estimate we make about a population based on the data we've collected from the sample.

For example, let's say we want to know how many Canadian households experienced food insecurity in the past year.

We survey a random sample of 2,000 households, and we find that 23% reported experiencing food insecurity.

That 23% is the point estimate, and it's our starting point.

Here's something important to remember:

The true value is fixed, but unknown. This is what we want to know about the population, a quantity or a characteristic. There is a real percentage out there.

We just can't know exactly what it is unless we survey every household in the country, which impossible. Instead, we estimate it using a random sample.

Let's look at it another way.

In this image, the blue dots represent the entire population of Canada. Because we can't survey every household in the country, we survey a random sample of 2,000 households, represented by the yellow stars.

With this sample, we obtain a point estimate of 23% of Canadian households having experienced food insecurity in the last year.

By taking another random sample, we could obtain a different estimate.

This time, the sample is shown in red and estimates that 21% of households in Canada experienced food insecurity in the past year.

This leads us to our Main Idea 2: the margin of error.

Because we used a sample instead of surveying every household in Canada, how much might that 23% estimate vary if we asked a totally different group or sample?

That's where the margin of error comes in. It tells us how much our point estimate might vary.

It's normal to have uncertainty when you use samples. The magnitude of the margin of error reflects how uncertain we are about the point estimate.

To achieve smaller margins of error—or more precision— a larger sample is usually needed.

In our example, the sample of 2,000 households resulted in a margin of error of plus or minus 2 percentage points around our estimated value of 23%.

But what if we wanted to achieve higher precision?

A way to do this is to take a larger random sample. Larger samples reduce the uncertainty associated with point estimates.

In this example using a larger sample size, the width of the interval around the estimated value is narrower.

Finally, let's talk about the confidence level. You'll often hear the phrase "19 times out of 20", or "95% confidence", associated with an estimate. But what does that really mean?

19 times out of 20 means we are 95% confident that the true value we're trying to estimate is inside our confidence interval.

So if we used different samples over and over, 95% of those confidence intervals would contain the true value but 5% wouldn't.

Remember that the true value is fixed. We are talking about the error rate of our method - it captures the true value 95% of the time but fails to do so 5% of the time.

Before we wrap up, let's take quick look at how you can spot confidence intervals in the real world.

This chart can be found on Statistics Canada's website, and if you look closely, you'll see little vertical and horizontal lines on each bar creating a buffer area around the estimates. That's the confidence interval. And it's showing you how much uncertainty is built into the estimate. For example, the first bar represents a point estimate of 30% for the year 2018, while its margin of error is plus or minus three percentage points.

This table here, also shows confidence intervals, this time as lower and upper bounds next to the estimate.

So now, when you are looking at StatCan data, or any survey reporting confidence intervals, you'll know what those ranges mean and why they matter.

So here are the three things to remember about confidence intervals:

• First, the Point Estimate is the value we get from the collected data. In our case, 23%.
• Second, the Margin of Error is the range around that value, reflecting the uncertainty due to sampling. In this video, it's plus or minus 2 percentage points, which gives us a range of 21% to 25%.
• Finally, the Confidence Level is how often this method gives us an interval that covers the true population value. Remember, at 95%, we're saying this approach works MOST of the time, but not ALL the time.

Remember…the truth is out there. The true value, what we want to know about the population of interest, is fixed, but unknown. And confidence intervals are how we responsibly estimate that value and express how sure we are.

Watch our follow-up videos to learn more about:

• the elements that impact the width of a confidence interval, and
• how confidence intervals can be used to determine whether two estimates are significantly different.

(The Canada Wordmark appears.)