Revamping Moderation

by senorcordero

Completing standardisation and moderation is an important part of securing an accurate picture of where a cohort is and what next steps need to be made in teaching and learning in order to secure solid outcomes. Working as two secondary schools that are geographically close often meant going to each other’s school to conduct joint standardisation and moderation; however, East London traffic often meant that we lost valuable time in travelling between schools (up to an hour in the worst case!) and so I decided to explore a virtual moderation model. This is what we have changed to so far, and what the next steps are.

The thinking

As an examiner myself, we go through strict standardisation. This is done on an online system where you gain instant feedback on how you have scored compared to the lead examiner. It’s incredibly powerful, albeit high stakes, as you cannot copy anyone else’s answers – it’s all about how you apply the mark scheme. I wanted to create a similar system where people couldn’t be swayed by other’s responses to gauge a read of if the team were aligned or if there was divergence. The challenge was trying to make sure it didn’t come across high stakes.

The process 

To replicate the system, I created folders on Sharepoint that contained the mark scheme, several students from the present cohort’s work, along with some ‘seed’ scripts. When you examine, you receive your allocation of scripts from a centre, but along with these and mixed in amongst your allocation are seed scripts. These are further questions or papers that have been marked by the lead examiner that help to quality assure the consistency of marking as a benchmark and a way to calibrate. I wanted to make sure my team was singing off the same hymn sheet as our exam board, so to replicate this I included some examples we had returned from the exam board that had been marked to see if we were in line with the marks that have been historically allocated by the exam board on those questions, along with some examples uploaded to the exam board’s website with commentaries. This helps to diagnose if anyone is being over generous or overly harsh, and can help me see who I might need to offer more training to on how to apply the mark scheme. 

With the scripts in place and the mark scheme, I wanted a way to capture responses that eliminated the bias that seeing what others have awarded and being swayed by that mark. To do this, I used Microsoft Forms (Google Forms works in a very similar same way). For each section of marks on the exam paper, I created a compulsory question that staff had to answer through multiple choice selection, followed by a space for comments. This meant I could see a numerical breakdown of what had been awarded and if staff wanted to add any comments, such as justification or if they had any doubt, they could leave this here. For GCSE Spanish Foundation paper, the set up looked like this:

Question 1 – Photocard
– Multiple choice question – 8 marks – communication 
– Comments text-based question 

Question 2 – 40-word Paragraph 
– Multiple choice question – 10 marks – communication 
– Comments text-based question 
-Multiple choice question – 6 marks – Quality of Language
– Comments text-based question 

Question 4 – 90-word paragraph
– Multiple choice question – 10 marks – Content
– Comments text-based question 
– Multiple choice question – 6 marks – Quality of Language
– Comments text-based question 

As we wanted to focus on the essay tasks, we left the translation out this time, but I would do exactly the same for the translation section.

For each candidate, I created a separate section in the quiz. This enables each candidate’s marks to appear as one page on the form when sent out.

After this, I then sent the form out to staff to complete for moderation. 

On completion of the form, you can collate all of the responses in visual form in the forms platform, or you can see all of the responses from your team in one spreadsheet, allowing you to compare marks question-by-question and to see where there is consistency and where variation lies within marking.

The follow up

Following the collation of the results, as a leader I was able to see where the team were aligned and where the team varied in their marks more, allowing me to target in future training sessions areas that we need to revisit for marking or that we need to look at more in depth as a group. I wasn’t going to share how others marked; but the team actually requested this so they could see how they compared and could ask others how they marked for tips and advice, therefore I shared the sheet that collected their answers 

The feedback 

The feedback from the team was very interesting. All were in favour of the new way of moderation as they realised it saved the time and effort of going to our sister school. It also meant that they were able to get through more scripts, and should they wish, could complete it earlier in the week if they needed to rather than having to all do it at the same time, helping them balance the demands of the week better. The team also commented it avoided wasting time arguing over tiny points that at times used to be perceived as a waste of time, and also liked the fact that they could do it in their own time – not being rushed if they were a newer member of staff and needed more time to process the mark scheme.

These are positive points, but as we move forward, there are a few things I need to consider. The first being although we save time not going to our sister school, it is good to bring the two teams together for cohesion and to get to know each other, as at the end of the day, we teach the same curriculum. The second is the arguments. As much as they can be draining at times, they can provide good food for thought. As is such, I plan on having a 10-minute debrief to go through any problem areas where divergence is and talk through the marks for the learning of all – especially for less experienced staff (I have 2 ECTs on my team). Finally, I would like to find a way where on forms you can save your progress and come back as presently you cannot, meaning you have to leave the form open or do it in one chunk.

Either way, it was a new way of doing things, has provided me with a better understanding of the team’s understanding and training needs. But with refinements, this could be even more powerful.