Here are practice questions designed to test your understanding of the core concepts covered in this course. These questions mimic the style of university entrance exams. Take your time to answer them, and then review the explanations below to check your understanding.
Part 1: Multiple-Choice Questions
Question 1: What is the ”winner’s curse” in the context of education research? A) The tendency for schools with the highest test scores to receive the least funding. B) The phenomenon where the most successful, highly publicized interventions actually have overestimated effect sizes due to statistical noise. C) The rule that only studies with negative results get published in top journals. D) The fact that students who score highest on entrance exams often struggle in their first year of college.
Question 2: How does ”measurement noise” impact an education study? A) It reveals the exact, true impact of a teaching method. B) It completely prevents researchers from calculating any effect size. C) It introduces random errors that can make an intervention look much better or much worse than it actually is. D) It guarantees that the results of a study can be perfectly replicated in another school.
Question 3: Which of the following best defines a ”latent effect size”? A) The actual, underlying true impact of an educational program, separate from any measurement errors. B) The final number published in a news article about the study. C) The total number of students who participated in the research. D) The financial cost required to implement a new education policy.
Question 4: You are reading a study about a new math tutoring program. The study had a very small sample size (15 students) and reported a massive, record-breaking improvement in test scores. As a critical evaluator, what should you suspect? A) The program is guaranteed to work exactly as well for a group of 5,000 students. B) The latent effect size is definitely larger than the reported effect size. C) The results are likely influenced by high measurement noise, and the true effect size is probably much smaller. D) The researchers eliminated all measurement noise by using a small group.
Part 2: Short Answer / Scenario Question
Scenario: A local school district wants to improve graduation rates. They run a pilot test of a new mentorship program in one small high school. At the end of the year, graduation rates at that school jump by 25%. Thrilled by these results, the district decides to spend its entire budget to implement the program in all 50 high schools across the district, expecting a 25% increase everywhere.
Question 5: Using your knowledge of the winner’s curse and effect sizes, explain why the school district’s expectation is risky.
Answer Key and Explanations
Answer 1: B Explanation: The winner’s curse occurs when a study ”wins” by finding a large, exciting result that gets published or funded. However, because of random chance and measurement noise, this ”winning” result is usually an exaggeration of the true effect.
Answer 2: C Explanation: Measurement noise refers to the random variations and errors that happen in any study (e.g., students having a bad day during a test, poorly worded questions). This noise can artificially inflate or deflate the results.
Answer 3: A Explanation: ”Latent” means hidden. The latent effect size is the true, real-world impact of the program. We can never measure it perfectly; we can only estimate it, and our estimates always include some measurement noise.
Answer 4: C Explanation: Small sample sizes are highly vulnerable to measurement noise. A few lucky guesses by a couple of students can drastically skew the average. A massive result in a small study is a classic setup for the winner’s curse.
Answer 5 (Scenario Explanation): The district’s expectation is risky because they are likely falling victim to the winner’s curse. The 25% increase is the measured effect size, which includes both the latent (true) effect size and measurement noise. Because the pilot was conducted in only one small school, the measurement noise is likely high. The dramatic 25% jump might be due to a few specific, unrepeatable factors (like an exceptionally charismatic mentor or just a lucky year). When the program is scaled up to 50 schools, the noise will average out, and the true, latent effect size will be revealed. This true effect size is almost certainly lower than the 25% seen in the ”winning” pilot study.