New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

Balancing Efficacy and Engagement in Interactive Texts Jonah Bard1 1Dartmouth College, Hanover, NH 03755, USA Abstract We introduce Phosphor, a digital learning platform that integrates LLM-graded formative assessment directly into instructional content, and report results from a deployment with 151 students across three sections of Introductory Statistics at Dartmouth College. Full dosage of the Phosphor material is associated with an increase in final exam performance of between 0.71 SD (adjusting for prior exam scores) and 1.30 SD (unadjusted). Presented as an entirely optional, ungraded alternative to traditional readings, the platform was adopted by 90.2% of enrolled students, far exceeding typical reading-compliance rates. Additionally, score results across a natural variation in quiz formats suggest that embedded constructed-response questions are a valuable ingredient in driving outcomes. Together these results indicate that high engagement and measurable efficacy are simultaneously achievable; we discuss implications for the design of AI-augmented instructional tools. Keywords intelligent textbooks, large language models, formative assessment, intelligent tutoring, retrieval-augmented generation, reading engagement, constructed-response assessment 1. Introduction Now is the best time in history to be working on personalized instructional software. With the advent of LLMs, every student has a 24/7 knowledgeable personal tutor in their pocket. But if a basic LLM is all students need, why isn’t academic achievement suddenly skyrocketing? It appears that unrestricted, external AI use is largely a hindrance to students despite its convenience. Bastani et al. [1] demonstrated in a randomized controlled trial with nearly 1,000 students that unfettered access to GPT-4 actually harmed subsequent performance by 17% when the tool was removed — students used it as a crutch rather than a learning aid. Only a version with carefully designed pedagogical guardrails mitigated these negative effects. Meanwhile, student use of generative AI for academic work has surged: a 2026 survey by the Higher Education Policy Institute found that 94% of university students reported using generative AI on assessed work, up from 53% just two years earlier [2].

A recent meta-analysis suggests an overall positive effect of LLMs for targeted use cases [3], though broadly effective interventions proven in rigorous experimental designs remain scarce. Against this backdrop, it is well documented that university students almost never read the textbook. Reading compliance among undergraduates has declined dramatically since the 1980s [4], with students actively resisting reading assignments, and objective measures consistently showing substantially lower compliance than students self-report [5]. In our deployment, student-reported reading completion baselines for MATH 010 were approximately 15%, with instructors estimating 10%. Individual student reports of reading compliance ranged from "literally no one does that" to "is this being recorded?" These observations motivated the design of Phosphor (f.k.a. Spongium), a digital learning platform integrating LLM-powered formative assessment into instructional content. Phosphor embeds AI-graded quizzes into the reading workflow, making active recall a structural feature of the learning experience. The core premise is that AI is most effective when integrated directly into the content delivery system — a design philosophy aligned with the emerging vision of intelligent textbooks [6] and supported by the well-documented “doer effect,” in which completing practice questions integrated into readings yields several times the learning impact of reading alone [7, 8]. iTextbooks’26: Seventh Workshop on Intelligent Textbooks, June 28, 2026, Seoul, Republic of Korea $ jonah.z.bard.27@dartmouth.edu (J. Bard) https://jonahbard.com (J. Bard) © 2026 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2. Platform Design and Deployment 2.1. Platform Overview Phosphor is deployed as a web application. Instructional content is organized into lessons, each rendered as navigable web pages with a sidebar showing the full curriculum and per-lesson completion indicators. The statistics curriculum was custom-authored, grounded in open educational resources. The platform includes the following: Lesson Quizzes. Each lesson is associated with a bank of 15–20 exercises. Students take Lesson Quizzes consisting of four randomly selected questions from the lesson’s bank.

Multiple-choice questions (MCQ) are auto-graded; constructed-response questions (CRQ) are graded by Claude Sonnet 4.6 against instructor-defined, question-specific rubric criteria. The grading prompt receives the student’s response alongside the question text, a model answer, and explicit grading criteria, and returns a correctness judgment with an explanation. Students who achieve 75% or higher are marked as having "passed" and have "completed" the quiz. The test bank consists of 40% CRQ and 60% MCQ. Content is not gated: students may freely read and take quizzes regardless of past results. Quizzes permit unlimited retries. The use of LLMs for constructed-response grading has shown promising alignment with human raters [9], though reliability varies with question complexity. We did not conduct a formal inter-rater reliability study; however, the grading prompt followed best practices in LLM-based assessment [10]. Module Reviews. Phosphor offers cumulative Module Review quizzes covering content across all lessons in a module—each containing 10 questions, with a 90% pass threshold. The Module Review is MCQ-only by default but features an "all question types" mode in which students can opt to include a combination of CRQ and MCQ in the quiz. Students have unlimited retries. RAG-based chat assistant. A retrieval-augmented generation (RAG) chat sidebar allows students to ask questions while reading. The student’s query is embedded and matched against a vector index of curriculum content via cosine similarity, with top-matching chunks placed into the LLM’s context alongside guardrails restricting responses to the boundaries of the course curriculum. 2.2. Deployment Context Phosphor was deployed in an early pilot across three sections of MATH 010 (Introductory Statistics) at Dartmouth College in Spring 2026. Enrollment followed typical withdrawal rates, starting at 151 students and finishing at 143. Designed for non-math majors, MATH 010 is typically taken by underclassmen. The platform was presented as an entirely optional, ungraded alternative to traditional course readings. The course curriculum was organized into three modules, aligned to two Midterm exams and one cumulative Final Exam. We evaluate results from each exam, which were administered on-paper and in-person with heavy proctoring.

As Phosphor was deployed within regular course instruction, the results reported here draw on a secondary analysis of de-identified, aggregate student records. Critically, the quiz format differed between modules due to iterative design changes made in response to student feedback: Module 1 (Descriptive Statistics, 9 lessons). Each lesson included a mixed-format quiz containing both MCQ and CRQ, all graded by the LLM against rubric criteria. Module 2 (Probability & Sampling, 8 lessons). In response to widespread student feedback that the CRQ auto-grader was rigid and discouraging, Lesson Quizzes were reduced to MCQ only. The Module Review was also introduced in response to strong student demand. Module 3 (Statistical Inference, 7 lessons). After analyzing exam results, which suggested that MCQ-only Lesson Quizzes provide negligible learning benefits, CRQ were re-introduced to them. The Module Review was available in Module 3 as well. Note: the Module 1 Review was introduced after Midterm 2, upon request from students who desired more formative assessment resources when studying for the cumulative Final Exam. Module 1: Descriptive Statistics (9 lessons + review) Module 2: Probability & Sampling (8 lessons + review) Module 3: Statistical Inference (7 lessons + review) Lesson content Quiz (MCQ + CRQ) Quiz (MCQ only) Review (MCQ or MCQ + CRQ) 3. Results 3.1. Engagement Table 1 reports cumulative student engagement at various thresholds. 90.2% of enrolled students engaged with the platform, via Module Reviews or Lesson Quizzes, at least once. The engagement threshold table below uses the final denominator of 143 enrolled students, among which the median lesson reached was 22 of 24 (91.7%). Specifically among the 137 students who created a Phosphor account, the median lessons reached was 23 of 24 (96%). We assemble multiple measures of total reading compliance. An upper bound is given by exposure, with 75.6% of student–lesson pairs reached (taking the later of each student’s last-viewed and furthest completed lesson). The aggregate quiz completion rate was 48.1%. Because completing a lesson’s quiz requires having read the lesson, this is interpreted as a lower bound.

Total reading compliance thus falls within [48%, 76%], against student- and instructor-reported baselines of 10–15% for this course. Table 1 also reports survey results administered live, in-class to each of two course sections via Mentimeter, once immediately after Midterm 1 (S1, 𝑛 = 33) and once immediately after Midterm 2 (S2, 𝑛 = 31); we report these as descriptive indicators given potential social desirability bias. Table 1 Platform engagement and student reception. Compl. = lesson quiz completed (a lower bound on reading, since passing assumes having read the lesson); Reached = furthest lesson reached, via last-viewed page or furthest completion (an upper bound, assuming sequential progression). The threshold panel is over the final roster (𝑁 = 143); the per-module panel uses per-module exam-taker denominators. "Passing" a Module Review consists of achieving a minimum 90% score. Overall Completion Metrics Threshold Compl. Reached ≥5 lessons 69.9% 87.4% ≥10 lessons 57.3% 81.8% ≥15 lessons 39.9% 76.2% ≥20 lessons 25.2% 59.4% All 24 lessons 11.2% 44.8% Median 45.8% 91.7% Mean 48.0% 75.6% ≥1 Review 76.9% — ≥1 LQ or MR 90.2% — Lesson Quiz Engagement Module Compl. Rate M1 670/1350 49.6% M2 691/1176 58.8% M3 334/1001 33.4% All 1695/3527 48.1% Module Review Engagement Mod.