The thought occurred to me this morning that the A level algorithm may be illegal under GDPR law, which I recalled having provisions on automated, data driven, decision making. This is what the Information Commissioner's Office say under the headline:
And they add this:
They then elaborate, saying this:
All of this is open to interpretation, of course. Option 3 in the last box seems the most likely fall back for OfQual. But so far there is no personal appeal allowed for resulting errors. The process is then in breach of these rules. I would suggest for that reason alone that what is happening is likely to be illegal.
But there is also a complete failure to provide to all A level students information on how their decision was reached. That too is a failure.
And it can fairly be said that consent was never sought for this process.
In that case the chance that this algorithmic process was legal looks to be low to me.
If anyone is planning a legal challenge this seems like the way to go to me.
Thanks for reading this post.
You can share this post on social media of your choice by clicking these icons:
You can subscribe to this blog's daily email here.
And if you would like to support this blog you can, here:
Brilliant!!
Ah but GDPR is EU legislation is it not and therefore with an 80 seat majority Johnson feels free to “demonise” anything he wants does he not especially coming from the EU? I’m sure he’s taking his lead from Donald Trump who uses “demonisation” as his primary modus operandi to get what he wants, for example, tear-gassing a peaceful BLM protest in Lafayette Square in Washington DC and currently “knee capping” the US Post Office to avoid a postal vote in the coming presidential election. “Demonisation” works well with human beings with limited ability to engage in analytical thinking! Monty Python had fun with this aspect of human nature:-
https://www.youtube.com/watch?v=zrzMhU_4m-g
GDPR is incorporated into the UK data protection act (2018).
We are still in the EU until December the 31st. Removing the GDPR from UK legislation means that no EU data could be sent into the UK, meaning UK businesses operating in the EU would have serious problems (to add to the serious problems that will arrive at the start of 2021). Fortunately, many formerly UK-based companies are migrating their business to an EU member state so as to continue to operate there. Including airlines. Don’t forget, after we leave the EU the govt can wail and scream as much as it likes. It’ll have no effect.
GDPR is an interesting angle but I think Ofqual will say the grade allocation process is not completely automated. For example https://www.computerweekly.com/news/252487692/ICO-acknowledges-GDPR-concerns-over-A-Level-results-scandal
“Ofqual has stated that automated decision-making does not take place when the standardisation model is applied, and that teachers and exam board officers are involved in decisions on calculated grades. Anyone with any concerns about how their data has been handled should raise those concerns with the exam boards first, then report to us if they are not satisfied.”
and
“Under the GDPR, students have a right to request information about their performance which may include their teachers’ assessments, written comments about their provisional grade and/or rank order, and past performance records.”
and
“However, they do not have a right to access any information they have recorded themselves, which means they cannot get copies of answers from mock exams, assignments or other assessments.
and on and on:-
https://www.computerweekly.com/news/252487692/ICO-acknowledges-GDPR-concerns-over-A-Level-results-scandal
What is very clear is this sociopathic Tory government is refusing to take responsibility for their failure and pushing the onus of responsibilty onto students to challenge their grading through a slow and unfair process. Voters were warned about Johnson’s running away from responsibility, his lying and demonisation of anybody who stood in his way, all hallmarks of sociopathy, but 43% went ahead and voted him into office. Those who have down-graded children will now find they’ve very likely lost their children’s respect!
My issue was with consent as well though
Plus students cannot appeal
Only schools can
That seems to deny required rights
This is probably what is being changed overnight & this weekend. The appeals process is being reviewed. Now!
The usual overlooking of details, lack of planning ahead on has got used to seeing from this Mafia.
I suspect you are right
But if I could see, and evidence, it why couldn’t they?
I am no lawyer and cannot offer an opinion on whether that is right regulation to challenge.
However I imagine the exam boards would respond by pointing out that automated algorithms are normally part of the process. In usual years raw results come out of individual examiner assessments, and then an automated moderation minimises discrepancy between examiners after double marking sampled students. Finally a more complicated algorithmic process assigns grades based partly on more sampling by senior examiners but also normative criteria (i,e, retaining broadly the same ratio of grades awarded year to year). This year the “raw marks” come from the schools, but the moderation is based on school based characteristics rather than script sampling.
I’m not defending them, merely reinforcing what you said in a previous post about the big weakness being the absence of any sort of moderation that looked at the individual student evidence. It would have been a challenge but surely they could have tried.
The biggest failure may be the exam boards not having the sense to check their process for unintended consequences – and applying corrections if necessary. It is clear a fair process shouldn’t have favoured independent schools over comprehensives, and that would have been a trivial analysis as part of checks. More complex but perfectly achievable would have been checks there was no bias by level of deprivation or ethnic mix (by using geographical areas with different social mix as proxies).
My pint us that students consented to that when applying for the exam and to UCAS
They have not consented to the current process
The moderation system developed by the SQA in Scotland had a clear inbuilt process for appeals, which may have insulated them from potential GDPR issues. Paradoxically (perhaps), reversion to the results estimated by teachers (except where awards were higher) has obviously caused problems with the appeal system. The system is currently being reviewed, and it’s quite hard to see how it can be modified to fit the outcome of the changes to awards. I hope they get it right.
So do I
I have been using an intern to build a model for renewables and the “levelised cost of electricity”. Covers different regions (of Europe) and different tech. Intern was very proud of his model. I then told him to get real world data (on results from renewable auctions) and test these against the model – the two should be close.
Off-Dumb could have taken last year’s results – & compared them to a run using the algo’. Pretty trivial bit of number curnching – could even just run a smaple – 10,000?. Given the current fiasco sounds like Off-dumb were too dumb to do it. Or probably too prtective of what was a lash-up.
Legal angle is interesting. Class action?
It would be an interesting idea
But would waste a year of lives, nonetheless
I think you misunderstand. Develop the algo – take a sample from last year (teachers estimates etc for last years results exist) – feed that data & data relating to schools as was in 2019 and run the algo – then compare algo output with actual result (2019). Would not affect anybody but would provide a metric – how good or bad the algo was. Well we know how bad it is – point is Off-unqual were too lazy (or too supid) to be bothered to do this – or maybe they did – with a small sample. Who knows.
One thing for sure – the result fits with tory “education” policy: private good, state very very bad.
Agreed
According to https://www.youtube.com/watch?v=wZODW080gsc (Report Shows Teachers Much More Accurate Than Algorithm) Ofqual did run a test against last year – page 204 of their report – with very poor results. See also https://www.youtube.com/watch?v=cCGqiuFy8ZM (Exam Grades Awarded Breaks Multiple Laws) for his take on the legality issue.
Thanks
Thank you, Richard. It is helpful for us that as well as accounting, tax, and economics, you have expertise in GDPR, and put forward clear views so unhesitantly. How do I subscribe to your blog, please?
I am GDPR registered, several times
I have to know something of it: it’s a legal requirement
But all you know about is trolling
Martin
You don’t have to be here you know? If you don’t like to see turned on, concerned citizens interacting with the world, why don’t you just go somewhere else?
If you like being passive, go and be passive elsewhere – you’ve no need to feel uncomfortable here.
Blimey!
On Channel 4 news there was an interview with someone from the statistical society and it was mentioned that the description of the algorithm ran to 300 pages. This length leads me to believe that many additions have been added overtime making it difficult to work out the ramifications. The admitted use of class size will certainly give advantage to private schools.
Remember that you can make the algorithm do anything and it would not surprise me to find some more dubious additions.
The universities are already facing a loss of income due to a reduction in foreign students and all they can do to increase their income is to lower the marks for their courses to get more UK students. This is just the result of the market I note some universities have already said they will accept mock results.
Finally what this points to is that the examination system is not to measure the students competence in their studies or even skill in passing exams but to limit the number of students going to further education.
“The best judges of the relative ability of students in a school or college were the teachers who had been preparing these students… we know from research evidence that people are better at making relative judgements than absolute judgements and that teachers’ judgements tend to be more accurate when they are ranking students rather than estimating their future attainment”
https://stumblingandmumbling.typepad.com/stumbling_and_mumbling/2020/08/limits-of-radicalism.html
I enjoyed that
Have they actually published this algorithm for transparency and scrutiny?
Yes
It is in the Ofqual site
Where? I couldn’t find it, and I am not the only one.
Neither can I now…
Is this it – https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909368/6656-1_Awarding_GCSE__AS__A_level__advanced_extension_awards_and_extended_project_qualifications_in_summer_2020_-_interim_report.pdf
Also this – https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/909042/Requirements_for_the_calculation_of_results_in_summer_2020_inc._Annex_E.pdf
I should have mentioned that I got those links from https://ffteducationdatalab.org.uk/2020/08/a-level-results-2020-how-have-grades-been-calculated/, which has other interesting information.
The Royal Statistical Society, having observed Ofqual’s own analysis of their algorithm, offered to lend a hand..
” Our response does not put forward a particular model, but notes that there are important unresolved statistical issues, regarding both the estimation of individuals’ exam results, and the moderation of those by statistical modelling, with which we think the RSS could help. ”
https://rss.org.uk/news-publication/news-publications/2020/general-news/rss-alerts-ofqual-to-stats-issues-relating-to-2020/
Time they were asked for help
I think the cartoon in today’s Observer is spot on:
http://www.theguardian.com/commentisfree/picture/2020/aug/15/the-a-level-algorithm-meltdown-cartoon
I can’t help thinking that what is emerging is a sales drive for private schools.
This from Norway (yes, way!) appears relevant, suggest those cheery folk at Mishcon de Reya https://www.mishcon.com/news/potential-data-protection-challenges-to-a-level-results
They have a very good point, that I might happen to agree with on this occassion