Centre for Multilevel Modelling

menu

School effectiveness critiques

Update November 2000

Following the publication of the paper "School effectiveness research and Educational Policy" in the Oxford Review of Education (Goldstein and Woodhouse, 2000), Martin Thrupp has provided an important further perspective. This note is a response to a draft of his paper, a revised version of which is scheduled for the British Educational Research Journal (Thrupp, 2001).

School effectiveness researchers

Thrupp examines the counters to the recent critiques of SE, especially those of Teddlie and Reynolds (Teddlie and Reynolds, 2000). These authors are prominent among those singled out for criticism and it appears increasingly that they are among a very small number of SE researchers who  have, ipso facto, assumed the mantle of spokespeople for SE. Their views, therefore, take on added significance when making judgements about SE and its future. This is particularly important if one agrees with Goldstein and Woodhouse (2000), and many critics, that much recent work of some of prominent SE researchers is itself rather poor in quality.

Thrupp quotes Goldstein and Woodhouse extensively and generally with approval as coming from a 'relatively neutral' position which is in fact much closer to SE research than most of its critics.  He echoes Goldstein and Woodhouse in viewing the responses of people from  the SE community as superficial and inadequate. He sees Teddlie and colleagues as continuing to respond superficially, especially to 'political' criticisms as e.g. in Teddlie and Reynolds (2000).  Thrupp is particularly critical of their espousal of the 'pragmatic' over the acquisition of structural understanding of education, and the ways in which 'pragmatism' lines up conveniently with government policies by limiting its ability to  provide well founded critiques of such policies.

Technical matters

One problem with a part of Thrupp's critique is his relative lack of technical expertise. Thus he appears to accept the jibe of De Leeuw and Kreft (1995) reiterated by Teddlie et al. (2000) who deride the MLn multilevel modelling software for handling 'up to 15 levels'. Unfortunately, De Leeuw and Kreft as well as Teddlie et al. appear to be unaware of developments in multilevel modelling made in the early 1990s which found ways to incorporate highly complex (cross classified) structures which also required the ability to involve more levels in the model; 6 or 7  have been used in some applications and 15 was simply an arbitrary upper limit to ensure any problem could be handled. Thus, even in the case of a model with repeated measures on pupils, grouped within classrooms, within schools within school boards where several outcome measures are studied together, we require 6 levels; if the study spans transition from primary to secondary, involving a cross classified model (Goldstein and Sammons, 1997)  and also a cross classification with area of residence then a further 2 levels are added making 8 and one could go on. The point about increasing the complexity of these models, and the data collected for them, (see Goldstein, 1998) is that they then have a chance to match the complexity of the real world, and it is precisely the relative simplicity of many current SE analyses which, while they are an advance on what went before, should cause us to exercise caution.

Thrupp also points out that Teddlie et al. (2000) have reservations about the use of multilevel modelling because of its 'incomprehensibility' to many in the field and the way in which residual 'shrinkage' underestimates 'effectiveness'! In fact, neither of these concerns stands up to examination. While some of the procedures underlying these models is complex, the underlying rationale is not and there are perfectly straightforward ways of presenting results: of particular interest here is the Hampshire project which uses complex models and detailed graphical presentations for primary school heads who have little problem with the interpretation and are able to make constructive use of it (Yang et al, 1999). As for the 'shrinkage' issue this simply is not true. The results for a multilevel analysis provide unbiased estimates for the relative amounts of school level variation (within the context of the model being used). When estimating the actual underlying value for a school, however, shrinkage towards the mean does occur but this just reflects the relatively small amount of information available when only a few pupils are being measured. As long as individual school effects are based upon pupil data this will always be the case! As Goldstein et al (2000) argue, this places limitations upon their use for political accountability purposes and as Goldstein (1997) points out the main purpose of multilevel modelling is to understand the general factors associated with school differences and is not to produce individual school effect estimates. Such a misunderstanding of technical matters from prominent researchers is a real cause for concern.

Statistical models

Thrupp raises another issue in criticism of current quantitative models (though he appears to feel that this is a necessary feature of all quantitative models). He gives the example of 'social mix' influencing academic progress and suggests that without an understanding of exactly how peer social interactions occur and affect behaviour and performance and under what conditions, we will lack crucial understandings. This is perfectly reasonable; and there are several approaches to achieving such understanding. Qualitative studies of such interactions are vital since they will help to formulate both theoretical and empirical bases. Most importantly such studies will point to ways in which the essence of such interactions can be recorded and then systematically processed, for example as narrative descriptions or categorical summaries. Further down the line we can then begin to use such data, if collected in suitable ways using suitable scales, to incorporate into our statistical models so that they can begin to reflect more of the complexity of the processes operating. There is no real conflict between qualitative modelling and studying complex processes quantitatively, bit it is of course often difficult, time consuming and expensive. The development of more and more complex modelling techniques is currently being carried out in order that really complex structures are capable of being understood. In other areas, such as demography and epidemiology (see e.g. Goldstein et al, 2000) highly sophisticated uses of these models are beginning to be found. Because of the technical difficulties involved these models are often hard to come to terms with, but that is not a very good reason for rejecting them - by both defenders and critics of SE research, and sometimes, as above, it is the users of quantitative techniques themselves who fail to realise their true nature.

Government involvement

Thrupp is highly critical of attempts to defend SE researchers' involvement with government policies. Teddlie and Reynolds (2000) believe that the researcher can almost always provide 'objective'  evidence and should not be condemned for political misuse. Thrupp points to the difficulty in sustaining this position, e.g. that accepting government funding means that it is the government's agenda rather than anyone else's that gets researched. Thrupp also points out that Teddlie and colleagues omit any discussion of the criticism that some SE researchers are actively cooperating with government policies (see also Goldstein and Woodhouse, 2000). Nevertheless, there may be some situations where it is possible to take government funds and use them to produce critiques of policy which the government is unable to dismiss. A recent case in which I was involved was a study of 'value added' performance indicators funded by OFSTED which was critical of the use of test data by school inspections teams and league tables generally, pointing out the need to educate inspection teams into a proper understanding of such data. We were able to negotiate the right to freely publish the results (Goldstein et al, 2000) which were then 'accepted' by OFSTED.

The future for SE

Thrupp raises  the interesting possibility of the SE 'community' splitting into two groups - into those, such as Teddlie and Reynolds, who defend involvement with government and 'pragmatic' research into what schools can contribute to 'educational standards' and into those who are concerned about such an approach and share many of the views of SE critics. It would be a very great pity if such a divide were to happen along quantitative/qualitative lines and if such a division should occur it seems likely to effectively destroy SE as a serious field of study.

Goldstein and Woodhouse (2000) argue for a separation from government and for SE researchers to sort out the good from the bad. This will be painful; the critics from outside, and increasingly I believe from inside, will have to be listened to by those currently prominent in SE; editors of 'manuals' and editors of journals. Above all, the present tendency towards over-defensiveness needs to shift. Thrupp provides a useful summary of the key research issues at the end of his paper and does suggest that perhaps those being criticised may be coming round to accept some validity to those criticisms. I hope he is right, but perhaps we will have to wait for a new generation of researchers to appear and take over the field before substantial progress becomes possible.

References

Note: some of the documents on this page are in PDF format. In order to view a PDF you will need Adobe Acrobat Reader

Feedback

Edit this page