The Policy Exchange Report on OFSTED

The report was released this morning and is available here

I wish I’d been involved in writing it – I agree with it despite some criticisms as it is metaphorically a few million steps forward. The first things to say is that it is worth reading the whole thing. One of the things I welcome most is that some of the most cutting edge events and conversations I’ve been lucky enough to be engaged in (including on twitter) are reflected in the report.

Please read the whole thing, not the papers’ versions or the executive summary. I think it’s a powerful manifesto for reforming (and actually, maybe rescuing) OFSTED. Policy Exchange are touted as a right wing think tank. I think that’s about right, but I’ve met Jonathan Simons a couple of times, conversed with him a few more, and despite my left wing leanings there is far more I agree with him about than disagree (apart from the relative merits of teams in Liverpool, where he is wrong).

Here are some of the parts I noted and my opinions (parts in bold are direct quotes):

It is right and proper to have an independent schools inspectorate. I couldn’t agree more. It’s worth saying this at the outset because there is a significant proportion of those who work in education who think we should be left alone to get on with it. In my shortish career, I’ve experienced a school where the pub opened early to house teachers in their early free periods through to Local Authorities with absolutely no interest in results. That doesn’t exist anymore, and I think the existence of a regular inspectorate is part of the reason.

The report finds that currently OFSTED grades Achievement as the same as the overall grade in almost all cases (even when the other subgrades are all different). This is what I said in a previous blogpost. It’s correct to do so in my view. The quality of students’ life chances is highly correlated to their grades and experiences, so that’s vital for schools and the primary thing that matters. This inevitably leads to questions about why OFSTED bother grading “quality of teaching” separately or visiting lessons at all. This is central to the report.

The report points to research that suggests that distributed leadership is more effective than concentrated leadership. This is something I wrote about in my blog post on the book Legacy. I’m pleased to read it. It also points out that the effect of governance is less important to educational outcomes. My experience of (sometimes) having to lead governors through how to challenge the school backs this up. The report says that governance is likely to be more important as schools become more independent, and I wondered, as I read it, how this gap is likely to be closed. That there is a gap should probably receive more press.

The report says that we should inspect behaviour, However, there is a less clear cut case in the literature that behaviour and learning are correlated than there is with teaching or leadership – most often, because behaviour itself can be a proxy for other factors that hamper learning.

I think it’s really important to inspect behaviour, because in my experience, challenging schools are challenging because behaviour is challenging. This is often (especially in those schools) the one that parents are most interested in.

Overall, therefore, the report finds that the current framework with an overall grade and four sub graded criteria above are the correct ones for Ofsted to assess when looking at the quality of school, and it is not recommended that these change. The more important issue comes when looking at how these factors can be assessed.

I don’t disagree, though given what the report later suggests, I don’t see why the Quality of Teaching and Achievement grade shouldn’t be merged.

There’s a quote on page 14 prefaced with:

strong correlation between Ofsted ratings and the ratings that pupils and parents give to their school / the school their child attends.

My current school has surveyed parents and students, has exceptional results and an awesome ParentView score (though the report reports that this can be used fraudulently, I’m not sure a school would dare) so this is good for us. We’re a Requires Improvement school (though we will be Good at our next inspection, easily). However, I just don’t think that parents’ and students’ views are as valid as data. I will have to read the research quoted and look for a counterargument. It’s not that I think their views are irrelevant – just that I don’t like the hint that this should play a larger role in accountability.

Chapter 2 is my favourite part. The reason for this is that it clearly reflects the view that many have seen on twitter, and that I agree with. Lesson observations are neither valid nor reliable.

I have said before that the cost of having highly paid inspectors spending time on invalid trawls of actual lessons is a waste of time. Robert Coe gets a mention in the report as he has led the destruction of the validity of the practice of lesson observation. It is worth repeating what Coe finds, and I’m sure the headlines in the morning will say this:

The stark conclusion is that when comparing a lesson judgement of teacher quality against its ‘actual’ quality (as defined by the Value Added progress made by pupils in that class), the lesson judgements often do not tally. In fact, overall the results are worse than flipping a coin – there is a 49% chance that the quality of the lesson will be empirically the same as judged, and a 51% chance that it will not be the same as that assessed by an observation. For lessons on either extreme (either Outstanding or Inadequate), the accuracy falls away even faster – with at best a 71% chance and an 83% chance that an observed judgement of a lesson into one of these two categories is wrong

This is so important for Leadership Groups up and down the country who judge their teachers by aping OFSTED. The OFSTED methodology is invalid. I hope the Policy Exchange report helps this message get through. Even if you think you’re awesome at judging lessons, you’re likely to be wrong, so stop!

The report also quotes the Sutton Trust:

‘Even when conducted by well-trained independent evaluators, classroom observations are the least predictive method of assessing teacher effectiveness’.

There’s also the obvious point, which the report mentions but I experience every inspection. Teachers change their lessons for OFSTED. In our last OFSTED inspection, our “best” teacher said she had a lesson plan that allowed her to “show progress in 20 minutes” at any point in the lesson, and she’d turn that part on as and when the inspector arrived. If they didn’t arrive, she would continue to do the things she knows work.

There is loads more on lesson observations and more importantly, how they’re conducted by OFSTED. It’s worth reading it in detail.
There is, in particular, good stuff on how the ‘Cladingbowl clarification’ (ask if you need to) does not actually make everything OK – even if inspectors are listening and follow the clarification. More importantly, the report points out that inspectors have allegedly not graded lessons since 2009, but this has not reflected practice. The report says:

an assumption that the latest clarification around lesson observations grading will end poor practice amongst either individual inspectors or schools is most kindly described as optimistic.

The report also shows a lot of conflicting advice that must be really hard for inspectors. Basically, if you read some of the documentation, you can grade lessons, but read other parts and you can’t.

I’m really pleased about this. It shows that my colleagues on twitter who have banged on about this have been listened to (in particular @oldandrewuk, who is mentioned). Judging lessons isn’t valid or reliable, and OFSTED tend to the progressive (which in my biased view, makes it worse).

In particular, it has been raised that OFSTED might not grade lessons by removing the “Quality of Teaching” grade from their Evidence Form and coming up with a Quality of Teaching judgement grade at the end of an inspection through a plenary (I know, I know!). However, the report says:

Yet – crucially – the two other problems would still remain. Most notably, any single judgement made by inspectors would still remain – must remain – susceptible to the flaws outlined above, both in terms of its lack of validity and reliability, and the danger of inspectors applying their own personal preferences to what they have sense. In fact, in a worst case scenario all that would happen is that a series of flawed judgements would be made throughout the visit during observations, which would be stored in inspectors’ heads, not shared with the school, and an overall judgement that compounded all those flaws would emerge from an opaque discussion between inspectors right at the end of the inspection and handed down to the school with little chance of discussion or debate.

That is no improvement and I agree that’s possible.

Chapter 3 is about other methods to inspect. It focuses on data. OFSTED correctly place a lot of stock in data.

I remember Michael Wilshaw saying a school can’t be outstanding if teaching is not outstanding. Given the above, that’s rubbish, as you can’t judge teaching reliably or validly (is that a word, I’m not sure). However, I do think that a school and school leaders should live or die by their results. This section of the report basically says they do! Good! There are some interesting nuances (ie if achievement is outstanding you might be less likely to get outstanding overall, but if achievement is inadequate, you are getting inadequate) that are worth reading. But achievement dominates. I think that’s right.

This is interesting, because my current school’s data is average at best, but we know we’ll do well in future (predictions look too good to be true), and we also know how amazing our environment is (ie orderly, no disruption, etc), but I’d rather be judged on data! Our data will come from our amazing improvements since 2012.

There is in the report, I think, cautious acceptance that data measures from 2016 will help. I agree.

There is not a lot said about book scrutiny – though I think OFSTED have made this something that happens within a lesson observation. I think it’s a really valid way of someone (trained) seeing progress. I’m not sure OFSTED can as it takes me an hour to properly scrutinise 6 books.

There was some praise of student voice. I love what our students say about our school (and it will benefit my school for this to be part of any inspection), but I will certainly be looking at the research base for this as detailed in the report, as I think students can be wrong regularly. This is one of the things I think the report might have got wrong throughout.

I agree with the report when it suggests that behaviour issues aren’t important enough. A school can turn on the pressure on kids, I think behaviour inspections for a day are the right thing (this is what happens, I think it should happen more!)

There is a load of stuff in Chapter 4 about recruitment of inspectors. I can’t be an inspector, as I’m in an RI school. I think that’s ridiculous (because I really do think I have held the school together, alongside colleagues, under poor leadership). Anyway, the training of inspectors gets a hammering, as does the fact that they’re employed by private companies. They are hence accountable to the companies.

Everyone rates HMIs so much higher than AIs. Um, yes! There are lots of points made about the lack of ability of inspectors with data. I think they’re right.

I’m quite impressed at how forthright the report is on saying that inspectors might be part of “the blob” – agrees with my experience. I don’t think it’s their fault, but we’ve been through a period of extreme progressivism. OFSTED, in my view, has to reform if we are to come out of it. Daisy Christodoulou gets a prominent mention for her work in identifying how OFSTED definitively promote certain styles (and I’d like to say here – read her book, it’s awesome.)

Many headteachers spoke of the variability of inspection teams, and how this made them feel the need to play to the lowest common denominator, particularly when it came to lesson observations.

I feel the need to write “Word”, a colloquial word we used when I was young for total agreement.

To be honest, the point about recruitment of inspectors is well made, and useful. It’s not my main interest. Just stop them working as consultants at the same time please!

Chapter 5 is about how OFSTED affects schools. To summarise:

• When the call comes, schools panic and do things differently
• Teachers aim to ‘fit’ OFSTED criteria
• Observations actually cause disruption to the pattern of learning established
• 7 of 305 respondents thought the pressure from OFSTED was positive
• schools box tick for OFSTED (shock)
• Encourages MOCKSTEDs (the report does not explicitly criticise, I think it should have)
• Schools still use Levels, even though abandoned, because OFSTED demand it in the framework ie: levels of progress

The report points out it is unsurprising, given it results in jobs lost or jobs gained, that schools orientate towards OFSTED. Crucially, it says, the problem is that what OFSTED want is unclear enough for “chinese whispers” to dictate direction.

There is a point about schools below a grade 2 – multiple inspections make improvement less likely. It also explicitly shows how precarious being a Headteacher is.

The report’s recommendations:

Every school – regardless of previous Grade – would be inspected at least every 2 years. This inspection would be renamed the ‘short inspection’ and would consist of two elements:
There would be an off site assessment of the attainment and progress data of pupils in a school. This data would play a major role in determining the overall grade of a school – making clear and transparent what in effect happens now.
Inspectors would be expected to be better trained to interrogate and analyse statistical data
Alongside that, one inspector would visit a school for one day (for primary, secondary and special schools). That inspector’s role would be strictly to validate the head and governing body’s own assessment of their school through its own process of self evaluation. This would include, for example, testing their understanding and presentation of the school’s data on achievement and progress to see if it was plausible, discussing (as now) the school’s actions leadership and management, and walking around the school at key intervals to observe behaviour and safety

Importantly, the inspector would judge quality of teaching only by scrutinising and challenging the head’s own assessment of their staff. In a short inspection, there would be no routine lesson observations at all of teaching staff.
Instead, the inspector would expect to see – and challenge – the head’s judgement made over the course of the year. This could include looking at copies of staff objectives and appraisals; decisions made on performance related pay and promotions; any internal notes of formative observations or programmes of lesson study done within the school; book scrutiny of pupils’ work over time; discussions with staff or pupils; or other ways in which the head and SLT had decided to assess teaching quality. Inspectors would still be able to drop into lessons and watch teachers to help validate any of their judgements, and to get a feel of the school. But as well as not giving grades for lessons – as clarified by Ofsted recently- under thismodel they would not complete any formal evidence forms as part of this observations. This would remove the nuance and confusion that still exists in the system, and make clear to inspectors that their role is as a validator, not an assessor themselves.

I hope Hunt/ Gove/ whoever are listening to this!

Then the outcome of the above:

For the majority of schools – as a rough rule of thumb, those currently graded Good or better – we would expect no risk to be identified.
For those schools, that would be the end of the Ofsted process. A report would be published, as now, with an emphasis on visual representation of progress and achievement made by pupils, and a brief accompanying commentary from the inspector
In terms of grades, an overall grade would be given as now. In addition, the four sub judgements currently made would be merged into one overall assessment of ‘school capability’, which would also be graded as Outstanding
through to Inadequate. Importantly, a school would only qualify as no risk if both its overall Grade and its Capability was judged as Good or Outstanding

In other words, most schools would receive a combined grade of either Outstanding / Outstanding, Outstanding / Good, Good / Outstanding, or Good / Good.

If a school fails this (I believe it should be pass/fail, but that will be a future blog) they have a longer inspection – that’s the recommendation.

Here is where I depart from the report (on first reading):

For one point, I think Policy Exchange might have forgotten how hard it is to recruit in those challenging schools. And now if you join those schools, suddenly you are subject to a longer inspection (the most stressful thing teachers go through). They call it a “tailored inspection”. I can’t see good staff putting themselves through that sort of thing. In addition, this will mean that RI and Inadequate schools are more likely to engage in all the ridiculous and nugatory* OFSTED games that are so prevalent in schools across the country at the moment. The fact that this tailored inspection is also, if the report was implemented in full, going to include lesson observations is a really contentious point. I would also say this will mitigate against some of the positive steps the report recommends.

The inspection team would be trained to a high level, to allow them to carry out limited lesson observations, where necessary, to probe Quality of Teaching.

Even the MET study referenced (lots of training) is not convincing, and references to being more rigorous contradict earlier assertions in the report. I can’t agree that it is possible to probe Quality of Teaching in “limited lesson observations”. To me this is a real weakness of the report, and I think it needs to be identified. This is the main point of contention in the report and as I’ve said I think it weakens the assertions made about lesson observation earlier in the report.

The final recommendation is that an Outstanding school must be having influence on other schools. That’s great, though I think most are already.

Final thoughts: I’d prefer this model, and I like that it costs so much less, but the references to student voice and the sop to lesson observations are, in my opinion, wrong.

I agree with most of the recommendations. However, as always, we’re interested in the areas of disagreement… so to summarise…

I wish the lesson observation recommendations went further than “more training and don’t do them in the routine inspections” and explicitly said “don’t do them in any circumstances” – I think that’s an error in the report and I actually wonder whether the authors debated keeping it in. I think that  keeping them in any form of inspection will mean that nugatory* behaviour inspecting lessons will continue in many schools. 

I think student voice is important on behaviour and safety, not on quality of teaching.

Apart from that, I wish I’d written the report. I hope people in power are listening.

A point I haven’t made that the report expresses best is when the report says:

Ofsted should be exercise more caution in publications which seem to endorse certain teaching methods:

(space left for tomorrow when I insert the video from Andrew Old on how OFSTED endorse something ridiculous)

*As an aside, from the report I also learnt (by googling) what nugatory means. Good word.