Modern test theory holds that in order to determine the true measure of a candidate’s ability based on results of tests, an infinite amount of retests should be allowed — and the final grade should be computed by averaging every attempt. How accurately each individual test measures that ability can then be determined by figuring out how far that particular score falls from the average. But what kind of an impact would a change in policy that would allow students to retake A-levels an infinite number of times – or merely two or three times – have on the final outcome of the exam?
To figure this out, researchers at the Centre for Education Research and Policy used an analogy. They tried to compute how the ability to go more than once effected the results of the 2008 Olympic high jump event in Beijing. Their findings are laid out in a newly published paper titled “Should the best mark count when resitting at A-level?”
The paper begins by asking a question: Did Stefan Holm, a Swedish high jumper, lose out on the silver medal that should have been his?
The identity of the person who was going to top the podium was not in doubt by the time the athletes took their last attempt to clear the bar. Andrey Silnov, who was heading into the final round a clear winner, and was looking to improve his showing by beating the current world record, was as good as crowned. But the fight for the second place and the silver medal was close.
Germaine Mason had failed his first jump at 2.29 metres, a jump which three athletes who were placed after him, had flown over at their first attempt. These athletes included the Olympic gold medal winner from 2004, Stefan Holm, whose average best jump in major competitions that year had been six centimetres higher than Mason. According to Olympic rules, the rank order of the athletes is determined by their highest jump, and they are allowed up to three consecutive failures in the course of the competition. Mason, who went on to clear 2.34 metres, was therefore placed second despite failing in his first attempt at 2.29 metres
Is that fair? Well, that depends on what you’re trying to measure. If the goal is to identify the three best high jumpers in the world — rather than the three best performers on the day of the competition — then the Olympics should remove re-jumps. When the athletes’ results on the day of competition are compared to their true ranking according to the modern test theory, allowing a single attempt at any one height, and no further attempts once the bar falls, will show results that mirror the true skill of athletes more closely than if multiple re-jump attempts are allowed.
If the modern test theory can be applied to A-level test results, and if the purpose of A-levels is determining the students’ true skill levels, the best way to achieve this goal would be to allow students to take the exam once — and only once. But there’s another way to achieve the same result while still allowing students to retake the exam if they wish. Graduates can sit any A-level as many times as they want, but only the last, rather than the best, score will count.
A theoretical simulation of A-level results shows that if candidates are allowed to keep the best mark when they retake, the accuracy with which all candidates are classified into grades falls. This is because the retake candidates may outperform their true grade. If the retake candidates are given their final mark rather than being allowed to keep the better of the two, then retakes have no impact on the accuracy of grading. However, the impact of such a change to resitting policy on students’ and teachers’ decision marking needs careful consideration.