Overcode by MIT Scales Up Automatic Grading

overcode

Researchers at MIT have developed a new system to better manage computer science assignments handed in by thousands of students. The system, named OverCode, will help instructors manage their classes in a faster and more effective manner.

OverCode reviews online homework at scale by analyzing and comparing student’s code assignments to identify where these align and where they depart in their solution approach. The program then creates several templates of techniques that arrive to the right solution and groups students’ assignments according to these templates so that instructors can further process their solutions.

This clustering of student assignments — which could range from a few hundred to a few thousand — enables instructors to easily identify broader trends as to the approaches student favor when trying to solve their computer science assignments. Larry Hardesty of the MIT News Office explains,

“[Overcode] creates a program template, using variable names that a preponderance of students happen to have converged on. It then displays templates side-by-side, graying out the code they share, so the differences stand out in relief.”

Hardesty focuses on two benefits of the program: better course design and efficient management and feedback given to students:

“The system could allow instructors of online courses to provide generalized feedback that addresses a broader swath of their students. But it could also provide information on how computer-science courses — both online and on campus — could be better designed.”

Overcode has been evaluated in two usability studies. Twenty-four programmers reviewed three programming assignments with different complexity levels. The programmers used Overcode and another data mining and processing tool that displays solutions one at a time.

The results show that in regard to simple assignments Overcode performs as well as the one solution at a time tool in identifying solution strategies. However, when it comes to more complex assignments, Overcode performs significantly better. With Overcode, programmers managed to analyze about 45 per cent of the more demanding assignments while with the other mining tool only 9 percent.

Elena Glassman, one of the usability study authors, contends that the two experiments confirm the program’s suitability for analyzing and offering valuable feedback on more complicated assignments.

Overcode tackles a persistent obstacle that professors are facing. The growing popularity of MOOCs mean that hundreds if not thousands of students are enrolled in one course at a time and Overcode brings in an efficient approach to assignment evaluation. As Leila Mayer writes:

“it’s challenging for instructors to conduct a detailed analysis of the programming methods the students used when a course has hundreds of students, or in the case of some online courses, thousands of students.”

Without a truly efficient system for analyzing and assessing coursework, students cannot get the level of support and learning they signed up for – something MIT researchers hope to have solved with OverCode.