--- title: "Socratic AI in a High School Science RCT (Kao 2025)" description: "A 2025 preprint RCT with 90 Taiwanese 10th graders: ChatGPT Study Mode raised short-term argumentation and critical thinking, not self-regulation." canonical: https://research.askpoppy.ai/papers/kao-2025-socratic-ai-chatgpt-study-mode-science-rct/ last_updated: 2026-11-05 --- [Socratic and dialogic tutoring](/topics/socratic-tutoring/) [AI tutors and learning outcomes](/topics/ai-tutors/) [Metacognition: knowing what you know](/topics/metacognition/) # Teens who argued a science question with a Socratic AI scored higher on reasoning tests straight after In a two-day study at a high school in Taiwan, 90 tenth graders were randomly split into three groups. One group built a science argument with ChatGPT's Study Mode asking them guiding questions, one did the same task with peer review instead of AI, and one had no lesson. Right afterwards, the AI group scored highest on tests of scientific argumentation and critical thinking, and reported more confidence and engagement. Their planning and self-monitoring habits did not change. | | | | --- | --- | | Paper | [Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Randomized Controlled Trial](https://doi.org/10.21203/rs.3.rs-8118546/v1) | | Authors | Sean Kao, Patricia Grant, Steven Woltering | | Published | 15 December 2025 · Research Square preprint rs-8118546 v1 | | Study type | Randomized controlled trial, three arms · Preprint (not yet peer-reviewed) | | What we read | Full text read. We read the full preprint on Research Square (version 1). | | Summarized | 5 November 2026 | ## What they did Students took baseline tests, then a 30-minute module on AI literacy and how arguments are built. On day two, the AI group worked individually on a socio-scientific issue with ChatGPT Study Mode (GPT-5, August 2025) prompting clarification, justification and reflection; the comparison group used Argument-Driven Inquiry with double-blind peer review; the control group had regular lessons. The whole session was 200 minutes, followed by the same tests again. Groups were compared adjusting for pretest scores. ## Who took part 90 tenth graders aged 16 to 17 at one large high school in northern Taiwan, 30 per group, all with prior experience of argument-based science lessons and a TOEFL Junior score of at least 745. ## What they found - Scientific argumentation: the AI group outperformed both the control group and the peer-review group (overall effect F(2, 86) = 10.63, partial η² = .20). - Critical thinking: the AI group scored higher than both other groups (partial η² = .32); the peer-review group also beat the control group. - Self-rated confidence (self-efficacy) and engagement were higher in the AI group than in both other groups. - Metacognitive self-regulation (planning and monitoring one's own learning) showed no significant difference between groups (p = .12). - Raw posttest means for critical thinking were 25.4 (AI), 22.2 (peer review) and 20.5 (control), from similar starting points. **Effect size:** Partial η²: argumentation .20, critical thinking .32, self-efficacy .13, engagement .32; metacognitive self-regulation .05 (not significant). ## Limitations - A preprint, not yet peer-reviewed. - One well-resourced school with highly motivated students who had to pass an English test, so results may not carry over. - Immediate post-test after one day; no evidence about whether the gains last. - The control group had no lesson at all, and confidence and engagement were self-reported. - Students in the AI group used the same commercial tool the paper is about; the authors report no competing interests. ## What this means for learners and parents - A tutor that asks "why do you think that?" and "what is your evidence?" can sharpen reasoning in the moment, at least for motivated teenagers. - It did not make students better at planning and checking their own learning. The authors suggest the AI may have done some of that reflecting for them. - For younger children the same idea needs the adult to stay involved: ask them to explain their reasoning back, without the AI. ## Put it into practice (free tools) - [How to tell whether an AI tutor makes you think](https://learn.askpoppy.ai/guides/ai-tutor-that-doesnt-give-answers/) - [Explain it back (Feynman self-check)](https://learn.askpoppy.ai/explain-it-back/) **Try it:** [Ask Poppy](https://apps.apple.com/app/apple-store/id6757080904?pt=124290112&ct=research-kao-2025-socratic-ai&mt=8) is voice-first homework help for kids, built first for ages 7 to 10 (teens and adults use it too), on iPhone, iPad and Mac. It asks guiding questions instead of handing over the answer, so children learn how to find it. Free with 30 messages a day, then $3.99/week or $9.99/month. (We make it.) ## Citation Sean Kao, Patricia Grant, Steven Woltering (2025). Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Randomized Controlled Trial. *Research Square preprint rs-8118546 v1*. ## Related summaries - [Unrestricted ChatGPT help raised practice scores but lowered exam scores](/papers/bastani-2025-generative-ai-without-guardrails-harms-learning/) · Hamsa Bastani et al., 2025 · Preregistered randomized controlled trial - [A carefully designed AI tutor beat an active-learning physics class on a short-term test](/papers/kestin-2025-ai-tutoring-outperforms-active-learning/) · Greg Kestin et al., 2025 · Randomized controlled trial with a crossover design - [Supervised AI tutoring matched human tutors in UK maths classrooms](/papers/learnlm-eedi-2025-ai-tutoring-uk-classrooms-rct/) · LearnLM Team (Google) et al., 2025 · Exploratory randomized controlled trial - [An AI coach for human tutors raised topic mastery, most for weaker tutors](/papers/wang-2024-tutor-copilot-human-ai-tutoring/) · Rose Wang et al., 2024 · Preregistered randomized controlled trial Summary by the Ask Poppy team, AI-assisted and checked against the paper. Summarized 21 October 2026. Spotted an error? [simon@askpoppy.ai](mailto:simon@askpoppy.ai) · [Editorial policy](/about/) ```json {"@context": "https://schema.org", "@type": "Article", "headline": "Socratic AI in a High School Science RCT (Kao 2025)", "alternativeHeadline": "Teens who argued a science question with a Socratic AI scored higher on reasoning tests straight after", "description": "A 2025 preprint RCT with 90 Taiwanese 10th graders: ChatGPT Study Mode raised short-term argumentation and critical thinking, not self-regulation.", "url": "https://research.askpoppy.ai/papers/kao-2025-socratic-ai-chatgpt-study-mode-science-rct/", "mainEntityOfPage": "https://research.askpoppy.ai/papers/kao-2025-socratic-ai-chatgpt-study-mode-science-rct/", "author": {"@type": "Organization", "name": "Ask Poppy team", "url": "https://research.askpoppy.ai/about/", "parentOrganization": {"@id": "https://askpoppy.ai/#org"}}, "publisher": {"@id": "https://askpoppy.ai/#org"}, "datePublished": "2026-10-21", "dateModified": "2026-11-05", "inLanguage": "en", "about": [{"@id": "https://doi.org/10.21203/rs.3.rs-8118546/v1"}, {"@type": "Thing", "name": "Socratic and dialogic tutoring"}, {"@type": "Thing", "name": "AI tutors and learning outcomes"}, {"@type": "Thing", "name": "Metacognition: knowing what you know"}], "citation": {"@id": "https://doi.org/10.21203/rs.3.rs-8118546/v1"}, "isBasedOn": "https://doi.org/10.21203/rs.3.rs-8118546/v1", "image": "https://research.askpoppy.ai/og/kao-2025-socratic-ai-chatgpt-study-mode-science-rct.png", "isAccessibleForFree": true} {"@context": "https://schema.org", "@type": "ScholarlyArticle", "@id": "https://doi.org/10.21203/rs.3.rs-8118546/v1", "headline": "Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Ran", "name": "Socratic AI in K–12 Science Classrooms: Effects on Critical Thinking, Motivation, and Self-Regulation in a Randomized Controlled Trial", "author": [{"@type": "Person", "name": "Sean Kao"}, {"@type": "Person", "name": "Patricia Grant"}, {"@type": "Person", "name": "Steven Woltering"}], "datePublished": "2025-12-15", "isPartOf": {"@type": "Periodical", "name": "Research Square preprint rs-8118546 v1"}, "identifier": {"@type": "PropertyValue", "propertyID": "DOI", "value": "10.21203/rs.3.rs-8118546/v1"}, "sameAs": "https://doi.org/10.21203/rs.3.rs-8118546/v1", "url": "https://doi.org/10.21203/rs.3.rs-8118546/v1"} {"@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [{"@type": "ListItem", "position": 1, "name": "Home", "item": "https://research.askpoppy.ai/"}, {"@type": "ListItem", "position": 2, "name": "Summaries", "item": "https://research.askpoppy.ai/papers/"}, {"@type": "ListItem", "position": 3, "name": "Socratic AI in a High School Science RCT (Kao 20", "item": "https://research.askpoppy.ai/papers/kao-2025-socratic-ai-chatgpt-study-mode-science-rct/"}]} {"@context": "https://schema.org", "@type": "Organization", "@id": "https://askpoppy.ai/#org", "name": "Ask Poppy", "legalName": "Poppy Labs AS", "url": "https://askpoppy.ai/", "logo": "https://askpoppy.ai/poppy-mark.png", "email": "hello@askpoppy.ai", "foundingLocation": "Oslo, Norway", "address": {"@type": "PostalAddress", "addressLocality": "Oslo", "addressCountry": "NO"}, "founder": {"@type": "Person", "name": "Simon Dreyer"}} ```