How should we measure school performance and hold schools accountable? A study of competing statistical methods and how they compare to Progress 8

ESRC standard grant

Grant Code: ES/R010285/1
Duration: 01/10/2018 - 30/09/2021
Amount Awarded: £467,000

People

George Leckie, Principal Investigator
Harvey Goldstein, Co-Investigator (obituary)
Lucy Prior, Research Associate

Summary

In 2016, the Department for Education radically overhauled their secondary school accountability system and introduced 'Progress 8', arguing it to be the simplest and fairest school performance measure to date. Progress 8 aims to quantify and communicate the average academic value each school adds to their pupils' learning. Specifically, Progress 8 measures how much higher each school's pupils score in their age 16 GCSE examinations than expected given their age 11 KS2 test cores when they started secondary schooling. Progress 8 scores are used to hold schools to account, with the lowest scoring schools judged 'underperforming' and 'coasting'; classifications that trigger intense scrutiny and intervention from the school inspectorate, Ofsted. Given the high-stakes involved, research is urgently needed to first evaluate the strengths and weaknesses of the statistical method underlying Progress 8, and second, to explore the potential benefits of alternative methods for measuring school performance. Our proposed research will address these needs and in doing so will advance scientific understanding about school performance measurement.

See the ESRC page ES/R010285/1 for more details.

Videos

The following video is of the Northern Powerhouse Partnership giving oral evidence to MPs on the Education Select Committee about our recent Adjusted Progress 8 research on 05/06/2019. We previously gave evidence to this committee on this same topic at a private meeting with them in March 2019. Video.

The following video is a research presentation of the working paper below given to mixed audience of academics and non-academics at UCL Institute of Education 0n 14/11/2018:

Journal articles

French, R., Sariaslan, A., Larsson, H., Kneale, D. & Leckie, G. Estimating the importance of families in modeling educational achievement using linked Swedish administrative data. Journal of Research on Educational Effectiveness. Forthcoming. Article. Pre-print.
Gallo Cordoba, B., Leckie, G., Browne, W. J. Decomposing ethnic achievement gaps across multiple levels of analysis and for multiple ethnic groups: A mediation approach. Sociological Methodology? Forthcoming. Article. Pre-print.
Goldstein, H., Haynes, M., Leckie, G., Tran, P. (2020). Estimating reliability statistics and measurement error variances using instrumental variables with longitudinal data. Longitudinal and Life Course Studies, 11, 289-306. Article.
Leckie, G., Browne, W., Goldstein, H., Merlo, J., Austin, P. (2019). Variance partitioning in multilevel models for count data. Psychological Methods, 25(6), 787–801. Article. Pre-print. Slides.
Leckie, G. and Goldstein, H. (2019). The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal. Forthcoming. Article. Video. Slides.
Leckie, G. and Prior, L. A Comparison of Value-Added Models for School Accountability. School Effectiveness and School Improvement. School Effectiveness and School Improvement. Article. Pre-print.
Prior, L., Goldstein, H., & Leckie, G. (2021). School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability. School Effectiveness and Improvement. Article.
Prior, L. and Leckie, G. (2022). Student mobility: extent, impacts, and predictors of a range of movement types for secondary school students in England. British Educational Research Journal. Article
Prior, L., Jerrim, J., Thomson, D., & Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8' raised by COVID-19. Review of Education. Article.‌
Prior, L., Evans, C.R., Merlo, J., & Leckie, G. (2024). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA). Sociology of Race and Ethnicity. Article.

Working papers now published as journal articles (see above)

Leckie, G., Browne, W., Goldstein, H., Merlo, J., Austin, P. (2019). Variance partitioning in multilevel models for count data. arXiv:1911.06888. Article.
Leckie, G., Goldstein, H. (2018). Should we adjust for pupil background in school value-added models? A study of Progress 8 and school accountability in England. arXiv:1811.09240. Article.
Leckie, G., and Prior, L. (2021). A Comparison of Value-Added Models for School Accountability: arXiv: 2107.09410 [stat AP]. Article.
Leckie, G., and Prior, L. (2021). A Comparison of Value-Added Models for School Accountability. Bristol Working Paper in Education Series. Working Paper, 01/2022. Article.
Prior, L., Goldstein, H., & Leckie, G. (2020). School value-added models for multivariate academic and non-academic outcomes: A more rounded approach to using student data to inform school accountability. arXiv:2001.01996 [stat AP]. Article.
Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. arXiv:2104.06299 [stat.AP]. Article.
Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. Bristol Working Papers in Education Series. Working Paper, 01/21. Article.
Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. (CEPEO Working Paper No. 21-04). Centre for Education Policy and Equalising Opportunities, UCL. https://EconPapers.repec.org/RePEc:ucl:cepeow:21-04. Article.
Prior, L., Jerrim, J., Thomson, D., Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for 'Progress 8'. UCL IOE Quantitative Social Science Working Paper No. 21-12, April 2021. Article.
Prior, L., Evans, C., Merlo, J., Leckie, G. (2022). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) with application to students in London, England. Bristol Working Papers in Education Series. Working Paper, 06/2022. Article.
Prior, L., Evans, C., Merlo, J., Leckie, G. (2022). Sociodemographic inequalities in student achievement: An intersectional multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA) with application to students in London, England. arXiv: 2211.06321 [stat.AP]. Article.

Working papers

Leckie, G., Parker, R., Goldstein, H., Tilling, K. (2021). Multilevel models with random residual variances for joint modelling school value-added effects on the mean and variance of student achievement. arXiv: 2110.02079 [stat AP]. Article.
Goldstein, H., Leckie, G., Prior, L. (2020). Providing educational accountability for Local Authorities based upon sampling pupils within schools: moving away from simplistic school league tables. arXiv:2002.09897 [stat AP]. Article.
Leckie, G., Prior, L., Goldstein, H. (2019). The implications of Labour’s plan to scrap Key Stage 2 tests for Progress 8 and secondary school accountability in England. arXiv: 1911.06884 [stat AP]. Article.

Representative articles

The following articles are representative of the style of work we are aiming to do:

Leckie, G. (2018). Avoiding bias when estimating the consistency and stability of value-added school effects using multilevel models. Journal of Educational and Behavioral Statistics. Forthcoming. DOI: 10.3102/1076998618755351. Article. Slides.
Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’. British Educational Research Journal, 43, 193-212. DOI: 10.1002/berj.3264. Article. Video. Slides.
Leckie, G., & Goldstein, H. (2011). Understanding uncertainty in school league tables. Fiscal Studies, 32, 207-224. DOI: 10.1111/j.1475-5890.2011.00133.x. Article. Visualization.
Leckie, G., & Goldstein, H. (2009). The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172, 835-851. DOI: 10.1111/j.1467-985X.2009.00597.x. Article.

Keynote conference presentations:

April 2022. International Multilevel Modelling conference, Utrecht.
June 2020. Santiago. Cancelled due to COVID-19.
August 2018. '25 years of school leagues tables, accountability and choice: Lessons from England'. Educational Effectiveness EARLI SIG 18 & 23 conference. Groningen. Flyer. Slides.

Book chapters and Encyclopedia entries

Leckie, G. (2021). Cross-classified multilevel models, Encyclopedia of Quality of Life and Well-Being Research. Second edition.
Leckie, G. (2019). Multilevel models for continuous outcomes, SAGE Research Methods Foundations series.
Leckie, G. (2021). Multiple membership multilevel models, Encyclopedia of Quality of Life and Well-Being Research. Second edition.

Teaching presentations:

'The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England'. M-level: Statistics in Education. University of Bristol. December 2018.
'Introductory overview of quantitative methods and basic analysis approaches'. EdD: Conducting Educational Research. University of Bristol. June 2018.

Multilevel modelling short courses:

The following short courses use our school league table research as teaching examples:

December 2018, 2019, 2020, 2022, Introduction to Multilevel Modelling Using MLwiN, Royal Statistical Society, London
Jan and July 2019, 2020, Jan 2021, July 2021 Introduction to Multilevel Modelling Using MLwiN, Bristol.
October 2018, Introduction to Multilevel Modelling, Bamberg.
August 2018, Multilevel Modelling: Three-level, Cross-Classified and Multiple Membership Models, EARLI Special Interest Group 18 Educational Effectiveness, Groningen, Netherlands (Course Instructor). Flyer.

Education Select Commitee

June 2020. Submitted written evidence to Education Select Committee on “Left behind white pupils from disadvantaged backgrounds"
March 2019. Submitted written evidence to Education Select Committee on “Progress 8 and the need to adjust for pupil background”

Response to Ofqual consultation

April 2020. Submitted statement to Ofqual consultation on ‘GCSE and A level grading proposals for 2020’

Response to EPI consultation

February 2022. Submitted statement to Education Policy Institute (EPI) consultation on 'The Features of Effective School Groups: Measuring pupil inclusion at school-group level'

Policy briefings

Leckie, G., Goldstein, H. (2019). The Government’s Progress 8 school performance measure needs to account for pupil background. PolicyBristol. Policy Briefing 66: 01/2019.
Prior, L., Jerrim, J., Thomson, D., and Leckie, G. (2021). Is it worth calculating Progress 8 anymore? PolicyBristol. Policy Briefing 109:11/2021.

Impact case study

Secondary school accountability in England (2022). ADR UK. https://www.adruk.org/our-mission/our-impact/secondary-school-accountability-in-england/

Blogs

Prior, L., Jerrim, J, Thomson, D., Leckie, G. (2021). Is it still worth calculating Progress 8? BERA blog
Leckie, G., Goldstein, H., Prior, L. (2019). School accountability by Progress 8: A critique of Conservative anf Labour proposals. BERA blog.

Radio coverage

January 2019: Heart radio
January 2019: Love sport radio

Newspaper and other print coverage

March 2019: Times Educational Supplement.
January 2019: Bristol Post, Daily Express, The Guardian, The Independent, Leigh Journal, The Star, The Times, Times Educational Supplement, Warrington Guardian, Wigan Today, Yorkshire Post
June 2018. 'Testing children gives a poor measure of ability'. Letter in the Guarduan in response to Simon Jenkins’s piece on how ‘the cult of tests is ruining our schools’. Article.

Other online media coverage

March 2019: Schools Week
January 2019: Ekklesia, Greensheets, Humanists UK, Local Gov, My Science, Reclaiming Schools, Schools Improvement, Schools Week, World News Live 4 You

Knowledge exchange meetings:

July2021: Fischer Family Trust: Education Datalab. Discussed their related research work.
January 2021: Fischer Family Trust: Education Datalab. Discussed their related research work.
April 2020: Round table discussion on Progress 8 with 15 others. Via Zoom.
August 2019: Fischer Family Trust: Education Datalab. Discussed their related research work.
April 2019: Department for Education, London. Discussed Adjusted Progress 8 work.
February 2019: Ofsted, Bristol. Discussed Adjusted Progress 8 work.
December 2018: Fischer Family Trust. Discussed our Adjusted Progress 8 work.
September 2018: Ofsted, Bristol. Discussed our Adjusted Progress 8 work.
November 2017: Department for Education, London. Discussed grant proposal.
November 2017. HEFCE, Bristol. Discussed grant proposal.

Other meetings

January 2022. Round table discussion on educational equity hosted by Loic Menzies and Cambridge Assessment Network
August 2019. Discussion with Education Campaigns Manager at Humanists UK about Progress 8
May 2019. Discussion with Director and Chief Economist of Northern Powerhouse Partnership
April 2019. Presentation of Progress 8 research to Emma Hardy MP (Education Select Committee member) at Portcullis house
March 2019. Presentation of Progress 8 research to MPs at private meeting with UK Parliament Education Select Committee
February 2019. Discussion with Head of Co-Op Academies about Progress 8

Related advisory work:

June 2019. Discussion with Israel Academy of Sciences about school value added models.
2018-present: Discussions with Trinidad and Tobago Department for Education about school value added models.
2018-present. Discussions with Sweden Department for Education about school value added models.
2018-present. Office for Student (OfS) Expert Advisory Group on benchmarking methodologies.
March 2018. Discussions with Bulgarian Minister of Education and heads of directorates as expert member of World Bank team.
2017-present. Higher Education Funding Council England (HEFCE) Learning Gain Expert Group.

Related conference organisation:

Spring 2021. Co-organiser (with Dave Thomson) of NPD User Group meeting, Bristol.
January 2020. Co-organiser (with Dave Thomson) of NPD User Group meeting, London.
September 2018. Co-organiser (with Becky Allen) of NPD User Group meeting, Bristol.