Beyond the traditional school value-added approach: Analysing complex multilevel models to inform external and internal school accountability in Chile

Access to files

•   FULL-TEXT.PDF (pdf)

Abstract

In the last few decades, educational research has largely demonstrated the effects of the socio-economic background on academic performance. Traditionally, researchers have used the so-called contextualised value-added (CVA) concept, implemented via multilevel statistical models, to assess variation in learning outcomes arising from schools and pupils. Depending on the stakeholders they intend to inform, two basic types of CVA models can be defined: models for school accountability and models for school choice. School accountability models can be further distinguished according to the ‘recipient’ of the information: internal models provide information for school authorities to improve their own practices, while external models provide information for government officials to assess school performance for policy-making purposes. Despite the evidence in favour of the use of more complex models for school accountability, government practice in Chile has been restricted to the use of raw school averages in standardised tests as indicators of effectiveness, which have been used indiscriminately for the purposes of school accountability and school choice.Using data from the Chilean National Pupil Database (SIMCE 2004-2006), this thesis demonstrates how the traditional CVA (2-level) models fall short in addressing the complex phenomenon of academic performance, especially in the context of a developing and highly unequal country, such as Chile. The novelty of the CVA modelling in this thesis is that it extends and improves the traditional models insofar as they explicitly assess the variation between pupils, classrooms, primary schools, secondary schools and local authorities, as well as the correlation between Mathematics and Spanish Language at all levels. This is done by implementing two univariate 4-level CVA models for progress in Mathematics and Spanish fitted separately via maximum likelihood estimation (MLE) and a bivariate 5-level cross-classified CVA model for progress in both subjects fitted via Markov Chain Monte Carlo (MCMC) estimation.External school accountability measures were derived from the extended univariate and multivariate models and compared to measures derived from a model akin to the traditional approach. A number of key differences were found, leading to the conclusion that further adjustments to the traditional CVA models are not negligible. The univariate 4-level CVA models provide more insight into school accountability than the traditional approach in a straightforward fashion, while the bivariate 5-level model encompasses a more reliable and ultimately comprehensive view on school performance.With regard to internal school accountability, further models were specified with the purpose of analysing pupils' heterogeneity to inform school improvement processes. The concept of "cultural capital" (Bourdieu, 1977) was chosen to shed light on the matter. Since cultural capital is essentially immeasurable, a latent variable was constructed from a group of manifest variables related to access and use of reading materials. From a substantive point of view, this thesis shows how access to all sorts of reading materials and reading habits can have not only a relevant impact on pupils' progress in Language, but also in Mathematics. Finally, this thesis concludes around three main ideas: firstly, school value-added models for school accountability, either external or internal, need to take into account the complexity of influences affecting pupils' academic progress as thoroughly as possible, in order to make a fair assessment of schools' performance and/or to inform school improvement policies. Secondly, school effectiveness is not a unidimensional process, which implies that school value-added models should ideally (when there are available data) reflect upon the multidimensionality of the phenomenon and take into consideration the relationship between different subjects, as well as non-academic outcomes. Thirdly, CVA models can also be used to inform internal school accountability by analysing the effects of meaningful modifiable factors and potentially serve as drivers of school improvement policies.

Keyword(s)

Chilean education system; Educational inequalities; Multilevel modelling; School value-added

Bibliographic metadata

Institutional metadata

Record metadata