TL;DR
Turn the school run and nap time into listening time
- Thousands of audiobooks, podcasts and originals
- Listen on your phone, tablet or Echo — also offline
- Cancel anytime
A randomized trial involving 1,179 students in 10 U.K. public schools found that students receiving AI or human tutoring answered follow-up math questions correctly 56% to 58% of the time, compared with 43% after standard app remediation. After 10 weeks, however, tutoring did not improve scores on a broad standardized math test; tentative subgroup findings suggested girls may have benefited while boys did not.
Eedi and Google DeepMind say a randomized trial of 1,179 students in 10 U.K. public schools found their AI math tutor performed about as well as expert human tutors on immediate follow-up questions. Yet after 10 weeks, neither tutoring approach improved scores on a broad standardized math test compared with standard app practice, leaving the longer-term learning effect uncertain.
The trial ran from April through July, with secondary school students using Eedi’s platform for one class period a week, about 30 to 35 minutes, while a teacher was present. After a student answered a question incorrectly, the platform assigned them to one of four conditions for the study: standard in-app remediation, live text chat with an expert human tutor, AI tutoring with a human approving each exchange, or AI tutoring supplied with a summary of the student’s past misconceptions and work habits.
On similar follow-up questions, students in the tutoring groups answered correctly 56% to 58% of the time, compared with 43% among students receiving standard remediation. The company’s chief impact officer, Bibi Groot, described the AI and human results as “basically on par.” The finding concerns performance on related questions during practice; it does not show that students’ overall math achievement improved.
Of the students, 916 took both the pre- and post-trial Star Maths standardized tests. The report says scores were flat for AI tutoring compared with static practice, while the human-tutored group scored slightly lower than both. Researchers suggested that tutored students may have covered less material because they spent more time discussing problems. Adding personalized information about students did not appear to improve results beyond the AI’s existing information about the misconception behind a wrong answer.
The study also examined message quality and participation. Human reviewers approved 94.7% of AI draft messages without edits, but researchers found 11 AI messages containing false math statements, seven of which reached students. The report says human tutors’ messages also included seven math errors. The source excerpt reports that one in five students never replied to the tutor, but does not provide further breakdowns of participation or outcomes for that group.
Immediate Help, Uncertain Learning Gains
The results distinguish between helping a student recover from a particular mistake and improving performance across a wider math assessment. The AI and human groups did better than standard remediation on similar follow-up questions, but the trial found no added gain for AI tutoring on the standardized test. Schools considering these tools therefore have evidence of a possible in-session benefit, not proof of broad or lasting achievement gains.
Cost and staffing are part of the question. Groot said human tutoring is expensive and that having experts approve AI messages is also costly and “not a scalable solution.” The trial’s human oversight may make the system safer to test, but it does not establish how well the AI performs when it interacts with students without routine human review. That distinction matters to schools weighing whether tutoring technology can extend support without requiring a tutor for every exchange.
Subgroup results merit attention but should not be treated as settled. The report says girls who received AI or human tutoring tended to do slightly better on Star Maths than girls assigned to static practice, while tutored boys did worse, particularly those working with human tutors. Researchers characterized these findings as tentative; they do not establish why the patterns appeared or whether they would recur in other settings.
Products Worth Considering
LEGO The Legend of Zelda Great Deku Tree 2-in-1 Set
LEGO Technic Ducati Panigale V4 R Motorcycle 42107 Building Set
LEGO Icons Tranquil Garden Building Set – Japanese Zen Garden DIY Kit for Adults, 10315
As an affiliate, we earn on qualifying purchases.
How This Trial Expanded the Test
The new study follows a smaller trial reported the previous December, in which the company said a supervised AI tutor slightly outperformed human tutors on new problems in later topics: 66.2% compared with 60.7%. The larger trial reported a different result: AI and human tutoring were broadly similar on immediate follow-up questions. Differences between the studies mean the earlier result should not be read as a guaranteed advantage for AI.
Researchers designed the larger trial partly in response to concerns that tutoring studies can rely on small samples or measure learning too soon. This trial included a standardized test before and after the intervention, but only 916 of the 1,179 students sat for both tests, according to the report. The study was conducted with Eedi and Google DeepMind; the company’s chief impact officer discussed the findings with The 74.
““The AI and human tutor are basically on par.””
— Bibi Groot, Eedi’s chief impact officer, speaking to The 74
Limits in Scores and Safety Data
The reported test results do not establish whether tutoring affects learning beyond the 10-week trial, or whether any gains might emerge with a different schedule, curriculum or student group. The report also does not provide enough detail in the supplied material to assess the precision of the subgroup estimates or how consistently the gender patterns would appear elsewhere.
Researchers proposed that being asked to talk after every mistake might feel like a “signal of failure” to some boys, and that text chat could favor students more confident in reading and writing. Those are possible explanations, not confirmed causes. The excerpt says there were no clear differences by family income or age, while lower-achieving students assigned to human tutors scored lower than similar students receiving standard practice in one comparison; researchers suggested that covering less material may have played a role.
Safety findings also have limits. Reviewers approved most AI drafts, but seven false math statements reached students. The report says no harmful content reached students and that reviewers blocked two drafts containing a winking emoji they considered potentially too familiar. The supplied account does not give the full text of the released data, the error rate per total message, or longer-term monitoring results.
A Larger Trial Without Routine Review
Eedi began another trial in September with more than 15,000 students, according to Groot. In that study, the AI interacts directly with students, with human involvement intended for conversations that go off track. Its results could help address whether a tutoring system can work at a larger scale without a human approving each exchange, though they will not by themselves resolve whether the approach improves standardized achievement.
Eedi said it was releasing the current study’s data, including subgroup results and conversation-level analysis, along with findings the company did not expect. Further scrutiny of those data may clarify how students engaged, when the AI made errors, and whether outcomes varied across groups. Until those details and the larger trial’s results are available, the present evidence supports a narrower conclusion: AI matched expert tutors on immediate follow-up practice under human supervision, while broader test-score effects remain unshown.
Key Questions
How many students took part in the AI tutor study?
The randomized trial included 1,179 students from 10 U.K. public schools. A total of 916 students took both the pre- and post-trial Star Maths tests.
Did AI tutoring improve students’ math test scores?
The study found no improvement on Star Maths for students assigned to AI tutoring compared with standard app practice after 10 weeks. Students receiving AI or human tutoring did answer similar follow-up questions correctly more often than students receiving standard remediation.
Did the AI tutor perform as well as a human tutor?
On immediate follow-up questions, the tutoring groups had similar correct-answer rates: 56% to 58% for tutoring, compared with 43% for standard remediation. The AI was used with a human approving each interaction in this trial.
Were there differences in results for boys and girls?
The report describes tentative findings in which tutored girls tended to score slightly better than girls receiving static practice, while tutored boys scored worse, especially in the human-tutor group. Researchers said the findings need caution and offered possible explanations, not confirmed causes.
Was the AI tutor error-free?
No. Researchers identified 11 AI messages with false math statements, seven of which reached students. Human reviewers approved 94.7% of drafts without changes, and the report also found seven math errors in human tutors’ messages.
Source: the74million.org
