Monthly Archives: May 2013

HSP Talk: On Correcting a Significance Test for Model Misspecification

Heinz Second Paper* presentation by Nathan VanHoudnos
Monday, June 10, 2013
Noon - 1:30 PM
Room 237, Hamburg Hall
Title: On Correcting a Significance Test for Model Misspecification**

* The Heinz Second Paper (HSP) is a PhD qualifier for public policy students. Since I am in the joint Statistics and Public Policy program, mine is mix of math and policy.
** Contact me for a copy of the paper or slides.

Learning about whether interventions improve student learning is sometimes more complicated than it needs to be because of errors in the specification of statistical models for the analysis of educational intervention data. Recently, a series of papers in the education research literature (Hedges, 2007a, 2009; Hedges and Rhoads, 2011) have derived post-hoc corrections to misspecified test statistics so that the corrected versions can be used in a meta-analysis. However, these corrections are currently limited to special cases of simple models.

The purpose of this paper is to extend these corrections to models that include covariates and more general random effect structures. We develop a sufficient condition such that the distribution of the corrected test statistic asymptotically converges to the distribution of the standard statistical test that accounts for random effects, and we examine the finite sample performance of these approximations using simulation and real data from the Tennessee STAR experiment (Word et al., 1990). The What Works Clearinghouse, a division of the US Department of Education that rates the quality of educational interventions, has a policy that applies a simplified version of the Hedges (2007a) correction to any study which randomized by group but does not account for the group membership in the original analysis. We discuss the implications of this policy in practice.