Socratic and dialogic tutoring AI tutors and learning outcomes
A strictly Socratic AI helper was rated least helpful, and some students went to another chatbot
Students in an intro programming course solved three coding tasks with one of four AI helpers. Two answered only with guiding questions (Socratic), two gave brief direct help without full solutions, and each style either could or could not see the problem and the student's code. The Socratic helper that could see everything was rated least helpful for getting the task done, and descriptively had the most stress, the most switching to an outside chatbot and the fewest full explanations afterwards, though those last differences were not statistically significant.
| Paper | Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming |
| Authors | Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor |
| Published | 24 September 2026 · arXiv preprint 2609.29995 (cs.HC) |
| Study type | Randomized controlled trial, four arms (2×2) · Preprint (not yet peer-reviewed) |
| What we read | Full text read. We read the full preprint on arXiv (version 1, 24 September 2026). |
| Summarized |
What they did
Students were randomly assigned to one of four GPT-4o helpers that differed only in their instructions and context: Socratic or direct, with or without the problem statement and current code. After the tasks they rated the helper, reported whether they used another AI, and explained the bug and fix for one task in their own words. Logs captured how many questions they asked and how long their messages were.
Who took part
132 students in an introductory Python course (CS1) at a large US public university, Fall 2025, who took part for 0.5% extra credit.
What they found
- Perceived support for finishing the task differed significantly by condition (p = .007); the Socratic helper with full context scored lowest (mean 3.53 on a 5-point scale, vs 4.12 to 4.27 for the others).
- Overall 15% of students (20 of 132) used another chatbot during the study; in the Socratic full-context group it was 23% (7 of 30), vs 9–15% elsewhere. This difference was not statistically significant.
- Students' main reasons for switching were inadequate guidance, going in circles and the helper forgetting context. One called the Socratic helper "super rigid".
- The Socratic helper without context needed the most back-and-forth: about 11.1 questions per problem, vs 4.1 to 5.1 in the other groups.
- Across 128 answers, 57% showed full understanding of the bug afterwards. Direct full-context had the highest share (67%) and Socratic full-context the lowest (48%), descriptively.
Effect size: No standardized effect sizes; Kruskal–Wallis tests. Only perceived support for task completion, number of questions and message length differed significantly.
Limitations
- A preprint, not yet peer-reviewed.
- University beginners in programming, on three short tasks, not children or homework.
- Understanding was measured once, immediately, for one task; nothing on longer-term retention.
- Many of the differences, including understanding and use of another chatbot, were not statistically significant.
- Only 132 of 598 enrolled students ended up in the analysis.
What this means for learners and parents
- "Only ask questions" is not automatically good teaching. A helper that never gives a straight hint can frustrate learners into finding a chatbot that will just answer.
- The authors suggest helpers that shift between questions and more direct guidance depending on how stuck the learner is.
- For a child, that looks like the hint ladder: a question first, then a smaller step, then a worked example, rather than endless questions.
Put it into practice (free tools)
- Hint ladder: hints instead of answers
- How to tell whether an AI tutor makes you think
- Best AI tutors that don't just give answers
Try it: Ask Poppy is voice-first homework help for kids, built first for ages 7 to 10 (teens and adults use it too), on iPhone, iPad and Mac. It asks guiding questions instead of handing over the answer, so children learn how to find it. Free with 30 messages a day, then $3.99/week or $9.99/month. (We make it.)
Citation
Madeleine Eastwood, Harshith Narne, Joseph Hilby, Paul Denny, Ashish Aggarwal, Amanpreet Kapoor (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming. arXiv preprint 2609.29995 (cs.HC). https://doi.org/10.48550/arXiv.2609.29995
Related summaries
- Unrestricted ChatGPT help raised practice scores but lowered exam scores · Hamsa Bastani et al., 2025 · Preregistered randomized controlled trial
- A carefully designed AI tutor beat an active-learning physics class on a short-term test · Greg Kestin et al., 2025 · Randomized controlled trial with a crossover design
- Supervised AI tutoring matched human tutors in UK maths classrooms · LearnLM Team (Google) et al., 2025 · Exploratory randomized controlled trial
- An AI coach for human tutors raised topic mastery, most for weaker tutors · Rose Wang et al., 2024 · Preregistered randomized controlled trial