Students must show their work when completing math problems to identify their thinking mistakes and help them understand math correctly. New Artificial Intelligence projects aim to automate this process by training AIs to identify and predict students’ errors while studying math so that teachers can correct misconceptions in real time.
Developers can now create exciting algorithms within products that will assist teachers, without needing to learn about machine learning. According to Sarah Johnson, CEO of Teaching Lab, who provides professional development for educators, this is the first time this is possible.
Many of these initiatives emerged from Eedi Labs, a U.K.-based education technology company, which began hosting coding competitions in 2020 to challenge developers to use A.I. to improve students’ Math performance. The most recent competition focused on using A.I. to identify student misunderstanding of multiple-choice questions and understand the reasons for the wrong answers.
The winning project in the Eedi Labs coding competitions was one that captured student misconceptions using AI-generated analysis of multiple choice questions and their explanation by students.
data but was run by The Learning Agency, an education consultancy firm in the U.S. A joint project with Vanderbilt University — and using Kaggle, a data science platform — the competition received support from the Gates Foundation and the Walton Family Foundation, and coding teams competed for $55,000 in awards.
The latest competition achieved “impressive” accuracy in predicting student misconceptions in math, according to Eedi Labs.
Researchers and edtech developers hope this kind of breakthrough can help bring useful AI applications into math classrooms — which have lagged behind in AI adoption, even as English instructors have had to rethink their writing assignments to account for student AI use. Some people have argued that, so far, there has been a conceptual problem with “mathbots.”
Perhaps training algorithms to identify common student math misconceptions could lead to the development of sophisticated tools to help teachers target instruction.
But is that enough to improve students’ declining math scores?
Solving the (Math) Problem
The deluge of money being poured into AI is increasing every day. There is concern about the health of the economy and whether we are currently experiencing an “AI bubble.” However, the leaders within the edtech sector remain optimistic that there is significant potential to be realised through the smart use of this technology combined with solid research on the best practices for utilising it to improve learning outcomes for students.
In the beginning of generative AI, many people believed that simply integrating an education platform with a large language model would yield quality results, according to Johnson from Teaching Lab. Numerous chatbot wrappers began appearing on the market, all of which claimed that teachers would be able to create high-quality lesson plans for their students through the use of Chat GPT inside their LMS.
That is not the case, according to Johnson. In order for applications of AI technology to be truly beneficial to teachers within a classroom setting, they must have been trained on education-specific datasets. Therefore, when you look at how Eedi Labs is trying to differentiate itself, you see this clearly.
Today, Eedi Labs provides AI tutoring for mathematics. Eedi Labs calls its AI model “human in the loop.” Under this model, Eedi Labs uses automated responses created by its platform, which are examined by human tutors prior to being sent to their respective students, for errors and/or other improvements as necessary.
Plus, through efforts like its recent competition, leaders of the platform think they can train machines to catch and predict the errors students make when studying math, further expediting learning.
But training machine learning algorithms to identify common math misconceptions a student holds isn’t all that easy.
Cutting Edge?
The effectiveness of leveraging AI to detect areas of confusion in students will ultimately hinge upon what computer scientists refer to as “ground truth,” namely, how good the data was that was utilized to train these algorithms from the beginning. Therefore, the level of quality associated with both the types of multiple choice math problem questions and also the specific areas of confusion illustrated by the analysis of that data must first be affirmed (Malamut, Postdoctoral Researcher, Stanford Graduate School of Education). Malamut is not associated with Eedi Labs or any of The Learning Agency’s competitors.
In Malamut’s opinion, this new entry is not the biggest innovation yet.
The datasets used in this year’s competition were derived from students’ multiple choice answers, accompanied by very brief rationales created by those who submitted their answers. Thus, while it may represent a step forward for the company, as opposed to earlier iterations of the technology that were based solely on multiple choice questions.
Still, Malamut describes the use of multiple choice questions as “curious” because he believes the competition chose to work with a “simplistic format” when the tools they are testing are better-suited to discern patterns in more complex and open-ended answers from students. That is, after all, an advantage of large language models, Malamut says. In education, psychometricians and other researchers relied on multiple choice questions for a long time because they are easier to scale, but with AI that shouldn’t be as much of a barrier, Malamut argues.
Pushed by declining U.S. scores on international assessments, in the last decade-plus the country has shifted toward “Next-Generation Assessments” which aim to test conceptual skills. It’s part of a larger shift by researchers to the idea of “assessment for learning,” which holds that assessment tools place emphasis on getting information that’s useful for teaching rather than what’s convenient for researchers to measure, according to Malamut.
Yet the competition relies on questions that clearly predate that trend, Malamut says, in a way that might not meet the moment
Still, some think the questions used in the competition were not fine-tuned enough.
Woodhead states that while competitors had a broader understanding of what constitutes a ‘misconception’ compared to Eedi Labs, the company did find the AI predictions to be quite accurate during the competition.
Others remained uncertain whether the AI captured all student misunderstandings.
Malamut has seen great improvement in the types of questions used to better understand students’ thought processes and misconceptions than what was previously offered in the contest dataset. However, according to Malamut, several of the questions in the dataset are not particularly well-suited to do this. Although they contain both multiple choice and short answer options, Malamut would like to see an increase in the development of better-quality questions. There are many different types of questions that can expose students’ ideas. For instance, instead of having students respond to a fraction question, you can have them critique their peers’ rationale. Example: “Jim added these fractions in the same manner, showing how he arrived at the answer; do you agree with Jim? Why/why not? Where did Jim go wrong?”
Whether it’s found its final form, there is growing interest in these attempts to use AI, and that comes with money for exploring new tools.
