Socratic and dialogic tutoring AI tutors and learning outcomes Metacognition: knowing what you know
Teens who argued a science question with a Socratic AI scored higher on reasoning tests straight after
In a two-day study at a high school in Taiwan, 90 tenth graders were randomly split into three groups. One group built a science argument with ChatGPT's Study Mode asking them guiding questions, one did the same task with peer review instead of AI, and one had no lesson. Right afterwards, the AI group scored highest on tests of scientific argumentation and critical thinking, and reported more confidence and engagement. Their planning and self-monitoring habits did not change.
| Paper | Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Randomized Controlled Trial |
| Authors | Sean Kao, Patricia Grant, Steven Woltering |
| Published | 15 December 2025 · Research Square preprint rs-8118546 v1 |
| Study type | Randomized controlled trial, three arms · Preprint (not yet peer-reviewed) |
| What we read | Full text read. We read the full preprint on Research Square (version 1). |
| Summarized |
What they did
Students took baseline tests, then a 30-minute module on AI literacy and how arguments are built. On day two, the AI group worked individually on a socio-scientific issue with ChatGPT Study Mode (GPT-5, August 2025) prompting clarification, justification and reflection; the comparison group used Argument-Driven Inquiry with double-blind peer review; the control group had regular lessons. The whole session was 200 minutes, followed by the same tests again. Groups were compared adjusting for pretest scores.
Who took part
90 tenth graders aged 16 to 17 at one large high school in northern Taiwan, 30 per group, all with prior experience of argument-based science lessons and a TOEFL Junior score of at least 745.
What they found
- Scientific argumentation: the AI group outperformed both the control group and the peer-review group (overall effect F(2, 86) = 10.63, partial η² = .20).
- Critical thinking: the AI group scored higher than both other groups (partial η² = .32); the peer-review group also beat the control group.
- Self-rated confidence (self-efficacy) and engagement were higher in the AI group than in both other groups.
- Metacognitive self-regulation (planning and monitoring one's own learning) showed no significant difference between groups (p = .12).
- Raw posttest means for critical thinking were 25.4 (AI), 22.2 (peer review) and 20.5 (control), from similar starting points.
Effect size: Partial η²: argumentation .20, critical thinking .32, self-efficacy .13, engagement .32; metacognitive self-regulation .05 (not significant).
Limitations
- A preprint, not yet peer-reviewed.
- One well-resourced school with highly motivated students who had to pass an English test, so results may not carry over.
- Immediate post-test after one day; no evidence about whether the gains last.
- The control group had no lesson at all, and confidence and engagement were self-reported.
- Students in the AI group used the same commercial tool the paper is about; the authors report no competing interests.
What this means for learners and parents
- A tutor that asks "why do you think that?" and "what is your evidence?" can sharpen reasoning in the moment, at least for motivated teenagers.
- It did not make students better at planning and checking their own learning. The authors suggest the AI may have done some of that reflecting for them.
- For younger children the same idea needs the adult to stay involved: ask them to explain their reasoning back, without the AI.
Put it into practice (free tools)
Try it: Ask Poppy is voice-first homework help for kids, built first for ages 7 to 10 (teens and adults use it too), on iPhone, iPad and Mac. It asks guiding questions instead of handing over the answer, so children learn how to find it. Free with 30 messages a day, then $3.99/week or $9.99/month. (We make it.)
Citation
Sean Kao, Patricia Grant, Steven Woltering (2025). Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Randomized Controlled Trial. Research Square preprint rs-8118546 v1. https://doi.org/10.21203/rs.3.rs-8118546/v1
Related summaries
- Unrestricted ChatGPT help raised practice scores but lowered exam scores · Hamsa Bastani et al., 2025 · Preregistered randomized controlled trial
- A carefully designed AI tutor beat an active-learning physics class on a short-term test · Greg Kestin et al., 2025 · Randomized controlled trial with a crossover design
- Supervised AI tutoring matched human tutors in UK maths classrooms · LearnLM Team (Google) et al., 2025 · Exploratory randomized controlled trial
- An AI coach for human tutors raised topic mastery, most for weaker tutors · Rose Wang et al., 2024 · Preregistered randomized controlled trial