By Junaid Mubeen, Whizz Education
How do you explain machine learning prediction models to a group of 8 and 9 year-olds?
That is precisely the challenge I was faced with back in May, ahead of the Whizz/UHi sequencer trial at Rosemead Preparatory School. Thankfully, my misspent childhood came to the rescue as I framed the trial in terms of the Terminator movies. Once the students were assured that the stakes were not so apocalyptic, the month-long trial got underway.
Ninety-six Year 4 and Year 5 students were randomly divided into two groups; the first continuing along their Maths-Whizz tutoring journey as usual and the second group receiving lessons recommended by iTalk2Learn’s so-called Vygotsky Policy Sequencer (VPS), developed by UHi.
Thankfully, the VPS is not half as scary as the Terminator! Image via www.transpanish.biz
Both sequencers serve the same purpose: to recommend the right lesson to the student based on their individual learning needs. The current Whizz sequencer has done this effectively for ten years, while the VPS utilises cutting edge machine learning techniques that take into account more data than ever before, promising more accuracy and, ultimately, a better tutoring experience for students.
The trial has been celebrated as a success by the consortium, with four weeks of data gathered to compare the two sequencers. While the data is still being mined and the results yet to be analysed, the trial has revealed some fascinating insights into the merits and pitfalls of both sequencers.
Here are three of our lessons learnt so far…
1. Performance prediction must account for student engagement
During the trial, and in the post-trial surveys, students on the VPS often complained that the tutoring experience was too repetitive. From a pure prediction model standpoint, it is reasonable to deliver the same topic, and even the same lesson, repeatedly in succession if that lesson is deemed to be the best suited to the child’s current understanding.
However, engagement is just as key to learning and, indeed, supports deeper understanding. With that recognition, the Whizz sequencer has hard-coded policies to ensure that students see enough of a variety. Very rarely, for example, do students visit the same topic twice in a row.
Such policies are required to ensure the prediction model stays in touch with the human side of learning.
Whizz staff thanking Rosemead students for their contribution to research
2. Curriculum structure matters… a great deal
The best prediction model in the world may deliver lessons in a sequence that defies local curriculum structure. There is no absolute ‘right’ or ‘wrong’ ordering of lessons and, with that subjectivity, a pure prediction model may lose relevance in the classroom.
For example, a teacher may be left frustrated that the VPS insists on teaching adding fractions before multiplying fractions when, in fact, either approach is plausible. The Whizz sequencer fixes the ordering of lessons within a topic to align with local curriculum – the ‘sequencing’ occurs between topics rather than lessons.
Since the VPS personalises lesson delivery to an even greater extent, there is the risk that some students will experience the curriculum in vastly disparate ways. Even the most ardent supporters of personalisation would have to yield to the need for some consistency between what children learn, especially when their educational outcomes are often defined in terms of fixed curriculum structures.
3. Better prediction may allow for acceleration through the curriculum
With the improved predictive power of the VPS, students are no longer forced to progress through a topic one lesson at a time.
Consider, for example, a student who has been inactive on Maths-Whizz for a long period of time. Their Maths Age (the measure of their ability, measured per topic) will typically be lower than their ‘true’ ability. Such students may be left frustrated as they toil in lessons that are far below their level.
Students at Rosemead received certificates and trophies for participating in the trial
The VPS is able to adapt more quickly, allowing students to jump through several lessons at a time to get them up to speed. There is a caveat: jumping through lessons may leave gaps in students’ understanding and, once again, hard-coded policies may be required to ensure students do not jump too far ahead (e.g. no more than half a year at a time).
What will we do with these findings?
The VPS is showing great promise, as the trial analysis will hopefully confirm. But the early indications are that its data-driven approach must be balanced with hard-coded policies that take into account the human side of learning, local curriculum structure and the risks of making wrong predictions.
With that balance Maths-Whizz students can expect to enjoy an even more rewarding tutoring experience that, more than ever before, delivers the right lesson at the right time for every child.
To find out more about the sequencer trials, get in touch with the iTalk2Learn consortium today.
