Progress 8 – being held to account for every grade, in every subject

Introduction and a note on 2015:

From 2016, secondary schools will be held to account using four accountability measures: Attainment 8, Progress 8, the percentage of students who achieve the English Baccalaureate and the percentage of pupils who achieve a grade C in both English and mathematics GCSE.

On this final measure, rumour has it that before he left office Michael Gove had the intention to impose a 50% floor on schools. So having fewer than 50% of pupils achieve both maths and English at grade C will cause a school to be below the floor. This was not going to be announced in advance, so could easily still be the intention. Update: Tim Leunig has clarified below in the comments that this is not the case.

Of these four accountability measures, Progress 8 is going to be by far the most important. Everything I have heard suggests that it will dictate whether a school is subject to an Ofsted inspection (or not), and is likely to dictate the outcome of the achievement grade during an inspection.

Progress 8 has been developed largely by Tim Leunig of the LSE seconded to the DFE, who I’d like to commend on answering questions via twitter (@timleunig) in order to assist schools with understanding Progress 8.

2015:

Before I write about Progress 8, a word on 2015. Unless schools have opted in to Progress 8 (and I can see little benefit to a state school in doing so at this stage) the published accountability measures will be  the same as 2014. So we will be judged on the percentage of pupils who get 5 GCSEs including maths and English, the percentage of pupils who make expected and above expected progress in maths and English, the percentage of pupils who receive the English Baccalaureate, and the Best 8 value added measure. Nothing has changed for current Year 11.

2016:

Progress 8 comes in for all schools. We will have received an indicative Progress 8 score for 2015 via RAISE (I believe). Progress 8 comes from Attainment 8, so we should understand how to calculate that first. Before we do that, we need to understand the scoring system.

GCSE scores:

At the moment, a G grade is worth 16, an F is worth 22 and so on. I’m not quite clear on the reason for these numbers, which go up in 6s.

From 2016, the scores will be 1 for a G, 2 for an F, 3 for an E and so on right up to 8 for an A*.

This seems to make more sense to me. So in calculating Attainment 8, we must have in mind these scores.

I presume this will mean that the numbering systems for the new GCSEs in English and maths in 2017, and then the rest of the EBACC subjects a year later, can just fit in with the scores above.

For guidance on what non-GCSEs are worth, please see the technical guide.

Attainment 8:

Baskets: Baskets or buckets or whatever they’re called, I know people working in schools are sick of hearing about them. I think they’re the easiest way to understand this measure though.

Qualifications are in three baskets as follows.

Basket One: Two qualifications – Mathematics and English OR English Literature. These qualifications count double as long as pupils have sat both English and English Literature. Assuming the student has sat both qualifications, the stronger of the English/ English Literature subjects goes in this basket. The other one can go in basket three.

NB: Update October 2014 from ASCL briefing – hat-tip Amanda (@MakeMathsmatter): It is reported that some maths linked subjects will both count in basket one rather than maths counting double (eg ‘applications of maths’ and ‘methods in maths’)
Free standing maths, pure maths and statistics will not count in basket three if the student has maths in basket one. I will update this blog when this is confirmed for sure. UPDATE: This decision has now been reversed.

Basket Two: Three qualifications – English Baccalaureate. In this basket goes three of the other English Baccalaureate qualifications. These include History, Geography, Sciences, Computer Science and a very large selection of Modern Foreign Language qualifications.

If pupils are doing Biology, Chemistry and Physics, all three of these can go in here. There is one exception. If students are doing Core Science and Additional Science, both of these qualifications can count in this basket. However, if they are doing Further Additional Science, this does not count in this basket.

Each qualification in this basket counts single.

Basket Three: Three qualifications – Other ‘High Value’ qualifications.

Pretty much every qualification that isn’t Mickey Mouse (and some which are) counts in this basket. These can be three vocational qualifications, and can also include any qualifications that don’t fit into Basket One or Two because they’re full.

These count single.

Taking Baskets One, Two and Three together makes 8 subjects. Basket One counts double (English or English Literature as long as both have been taken). These are added together to obtain an Attainment 8 score for the student.

Examples: This is perhaps better shown by examples, so here are some.

These are the results attained by an actual student from my current school last year (2014) H.

Please ignore the curriculum. I’ve restructured that so no child follows a curriculum like this any more but that only kicked in with our new Year 11s as we did it two years ago. Anyway, H’s scores in 2013:

English D, English Lit C, Maths C, Psychology C, Art and Design D, History E, Science D, RS Short Course C, Citizenship Short Course D

Sticking these into the structure I’ve described and multiplying by the weighting:

So H would have achieved an Attainment 8 score of 4.0 – or 40 if you prefer not to use the decimal point version of the weighting.

So what about fictional student Angela, who achieved these results: Art – B ,Maths – C ,English – B, Physics – B, Chemistry – D, Biology – B, Spanish – A, Music – C, Psychology – C:

Angela’s Attainment 8 score is 5.1 (or 51), significantly affected by not doing English Literature (hence English only counts single).

School attainment 8 score:

To work out a school’s Attainment 8 score, add up the total for all students and divide by the number of students. This is what will be published.

Progress 8:

To work out a pupil’s Progress 8 score, we take their Attainment 8 score and subtract the expected Attainment 8 score based on KS2 fine levelled scores in English and mathematics.

The fine levels (4.1, 4.2, etc) are available online from Fischer Family Trust.

A model of the expected scores is available for 2013 from the technical guidance issued by the DFE. See page 17 but I’ve published it here:

We should note that we will not know the expected attainment 8 scores for 2016 until 2016 results are out. They will not be the same as in 2013, because pupils have not followed a curriculum driven by this accountability measure.

So if Angela had an expected Attainment 8 score of 5.4, her Progress 8 score would be 5.1 -5.4 = -0.3

More examples: Let’s use another example. Fictional pupil Charlie achieved KS4 Level 4.0 average in English and maths. He is expected to achieve 34 Attainment 8 points. Here is his results set out in a different style table:

Charlie’s progress 8 score is therefore 4.7 – 3.4 = +1.3. A real positive for the school.

At this stage, I have to anticipate a form of gaming I believe may become commonplace. If there is a school in difficult circumstances, and Charlie is in Year 11 with this curriculum, come March or April, what is to stop the school deciding to force Charlie to take an English Baccalaureate subject via cramming so that he can at least achieve an F, or an E, or even a D with a headwind?

It’s not in Charlie’s interests, I’d suggest. It is in the schools. If the school does this and he achieves a D, his profile looks like this suddenly:

So now Charlie’s score is 5.1-3.4 = +1.7. I’m minded of Daisy Christodoulou’s talk at ResearchEd where she pointed out that when a measure becomes high stakes it ceases to be a measure that has as much validity and reliability, and I wonder if this is something that has been considered – the careful manipulation of the curriculum.

My final example is fictional student Barbara, who I designed to ensure that we learn the lessons of having an academic curriculum. Barbara got great KS2 results at average 5.5. Her expected attainment 8 score is hence 69 or 6.9.

Despite achieving an A* in everything, Barbara’s Progress 8 score is 5.6-6.9 = -1.3.

Progress 8:Add up all the scores from the pupils and divide by the number of pupils. Here is a table that might help understand that:

This school therefore has a progress 8 score of 36.5/142. This will be rounded to two decimal places and the published progress 8 score will be + 0.26 I believe this will be displayed publicly like this slide here so that parents, governors and the community can understand:

Progress 8 in future years: The model is likely to become ex ante. This means that while the expected attainment 8 scores for 2016 will be based on 2016 results, beyond that the expected attainment 8 scores for each level will be based on prior year’s attainment.

So the targets for 2018 will be set in 2016, 2019 set in 2017 and so on.

My understanding is that this is partly to allow for a self-improving school systems model.

Consequences: I can’t remember where I heard this, so it may be out of date or nonsense. Having said that, it makes sense to be something like this: A Progress 8 score of -0.5 or worse for a school will mean they are inspected that year. A Progress 8 score of 1.0 or higher means they won’t be.

This narrative doesn’t fit in with what Ofsted are saying, but I would hazard a guess that something like that will happen.

Implications:

All subjects count: We are moving on from the era of English and mathematics being everything. Nonetheless, not all subjects are equal. In my school, I’ve said that Subject Leaders will want to be on top of the predictions for every student at every grade boundary

All grades count: This is not about C grades. An improvement from A to A* or from U to G will count the same as an improvement from D to C.

Curriculum matters most: Schools with a ‘dumbed down’ or non-academic curriculum will really suffer from this measure. In my school I’ve said that 95% of students should study 8 qualifying subjects. Ideally I think it should be 100%. I think all pupils are entitled to an academic curriculum.

A part of this is that Progress 8 is the kind of accountability measure I would have designed myself if I wanted something to measure the curriculum I promote with our pupils.

Schools will need to decide how to track and ‘intervene’ without traditional ‘interventions’: So getting kids in to cram on Saturdays or after school like many schools do for maths and English now will have an effect, but nothing like the effect it’s had up until now. While 5A*-C including maths and English has been the benchmark measure, some schools have put incredible efforts into maths and English and assumed (usually correctly) that those students will achieve 3 other grade Cs. In addition, since progress in maths and English have been two of the other accountability measures, this has meant incredible focus on these two subjects. Should schools do this now, I’d imagine they’d suffer as all subjects and all grade boundaries are going to matter.

Our reaction has been to expand our Pupil Progress meetings, where Pupil Progress Leaders (Heads of Year) in Years 9,10 and 11 meet with representatives from faculties (in Year 11 these are usually Heads of Faculties) to talk about 6 pupils.

These meetings are twice per half term, in directed time, and the meetings are empowered to make decisions. Hence the colleagues in them have to have liaised with teachers and faculties about those 6 pupils.

This can drive ‘interventions’, but actually it’s resulted in problem solving or even problem anticipating and is starting to have effect beyond the 6 pupils identified. The agenda for this meeting is sent out at least a week in advance, and preferably two weeks, and is data driven.

There are no excuses tolerated, no blame attached, just solution focussed. Hence you have NQTs being a part of making decisions that impact on the Head of Faculty, and you have time, included in the time budget, for colleagues to actually liaise, chaired by the Pupil Progress Leaders.

I may write more about these in the future. We’ve done this for a year, but Progress 8 makes it even more important.

I’d really like to hear how other schools are reacting.

There is one note of caution (as well as the possible gaming I mention above). This measure seems right and fairly egalitarian to me. However, Headteacher Liam Collins (@kalinsky1970) reported that Suffolk LA had reported that in 2013, nationally only ONE grammar school would have had a negative Progress 8 score.

I hope that this will change as schools that are not grammar schools adapt their curriculum to Progress 8.

Please see the updated guidance here.

NB: See I highly recommend Tim Leunig’s comments below in order to assist with understanding, and for some clarification and corrections.

Disclaimer: I don’t think my blogpost is any easier to understand than just reading the DFE guidance below, but I have written it as several colleagues requested it. The documentation is below.

Links: Progress 8 school performance measure: information for school leaders

Progress 8 technical measure: a guide

Progress 8 early opt-in