When teaching confidence intervals, we talk about how bias is not accounted for by the margin of error. Seems easy enough to me, so when I quizzed my students over this for the first time I was surprised at how many of them got the questions on this concept wrong. After thinking about this for the evening, I realized they didn’t really understand how bias and error are different. Bias is due to issues with the data collection while error is due to expected sampling variation. Part of their misconception is due to how students use the word “error” in everyday language. We think of error as a problem or a mistake. In a survey, error is expected because we know that all samples vary from the population. Error is ok and expected, bias is not.
To hammer down the point that the margin of error does not account for bias, when I gave back the quizzes the next day I told the students I had something really serious to discuss.
“If you cheated on the quiz yesterday, I need you to raise your hand.”
Only one student raised their hand (I told the first person who came to class that day that I needed them to raise their hand when I told them to.) Students were sufficiently terrified by this point.
“Ok, one person out of 32. So let’s make that a 95% confidence interval for the proportion of students who cheated on the quiz.”
Then I go through and make the confidence interval, still very serious, and we find the interval is (-0.029, 0.092).
“So, I can expect that between 0% to 9.2% cheated on the quiz (can’t have a negative % who cheated). Do you think that’s accurate?”
Students hopefully at this point realize this is a ruse and relax a little and say it is not accurate because no one is going to admit that they cheated in front of the whole class to the teacher. We talk about how margin of error is only as good as the data it comes from. If the data is inaccurate, the confidence interval is inaccurate also.
Now this isn’t really a completely accurate example, I didn’t take a sample from the population but asked the whole population so an interval isn’t really needed, but that’s ok. The point is made and it really sticks with students.