Confidence Intervals Part 2: Comparing Two Groups

Welcome back to our series on confidence intervals.

In Part 1, we focused on a single estimate and its confidence interval.

In this session, we will compare two groups using their confidence intervals and learn a quick visual check that helps us avoid making over-confident claims.

The guiding question we will be working with today is: "are men and women equally likely to meet the recommended guideline of 150 minutes of physical activity per week?".

We will begin with estimates, and then bring in their confidence intervals to interpret differences meaningfully.

Suppose we have data for men and women in 4 different age groups: 18 to 34, 35 to 49, 50 to 64, and 65 to 80.

These estimates show 52% of men aged 35 to 49 meet the 150 minute weekly guideline, compared to 36% for women in the same age group.

At first glance, it is tempting to conclude that relatively more men in this age group are active.

But before we do, we have to consider uncertainty.

We will do this using confidence intervals.

In part 1 of this series, we explained how confidence intervals surround an estimate with a range of plausible values, reflecting sampling uncertainty.

Narrower intervals suggest more precision; wider intervals reflect more uncertainty.

Today we'll be using confidence intervals to test whether relatively more men are active compared to women by doing a visual overlap check.

The visual overlap check is a quick, convenient and conservative way to screen for differences when you only have the published intervals and no access to the analytical file.

It is implemented as follows.

If the intervals do not overlap, then one may conclude to a statistically significant difference.

However, if the intervals do overlap, even slightly, then this visual check is inconclusive.

A formal test on the difference should then be conducted to confirm if the difference is significant, but this requires access to the analytical file and to software adapted to survey data.

We won't be getting into how that would be carried out in this video.

Using what we now know about the visual overlap test, let's add the confidence intervals to our chart.

For men aged 35 to 49, our estimate of 52% has a interval ranging from 47% to 57%. For women in the same age group, the 36% estimate has a interval ranging from 32% to 40%.

Because these confidence intervals do not overlap, we are allowed to conclude from the data that the true proportion differs by gender in this age group.

Now, consider a different age group: men and women aged 50 to 64.

Here, 40% of men are estimated to meet the recommended 150 minutes of physical activity every week, compared to 35% of women.

When we look more closely, we see that the confidence intervals overlap.

The visual check is therefore inconclusive, and we should avoid categorical statements based on this check alone.

A formal statistical test on the difference, using the underlying data, may still allow to conclude to a statistically significant difference between men and women in this age group, but we cannot do that from the visualization alone.

When we add confidence intervals to every age group and visually check for overlaps, we can summarize as follows: Relatively more men aged 18-34 and 35-49 meet the weekly physical activity recommendation compared to women in the same age groups.

However, the visual check for men and women aged 50-64 and 65-80 is inconclusive.

We should avoid drawing conclusions based on the chart alone.

Any time we try to communicate key findings, it's important to keep our wording aligned with what the chart or table can actually tell us.

In today's example, we learned that when confidence intervals do not overlap, you may safely conclude a statistically significant difference between men and women for the age groups concerned.

But when the intervals do overlap, one can only report that the visual check is inconclusive; a formal test would be needed to assess the difference.

Phrasing it this way keeps the message clear and measured without overreaching.

In the third and final part of our series, we'll see what affects the width of a confidence interval.

(The Canada Wordmark appears.)