WorksComputer Science
Instructor-designed AI improves learning in a randomized trial across 19 universities
Abstract
Universities are alarmed by recent evidence that artificial intelligence (AI) can undermine learning, with students prescribed generic chatbots or centrally designed AI tools performing worse. In a pre-registered randomized controlled trial across 7,200 students at 19 universities over three semesters, we tested whether AI designed by frontline instructors can improve learning. We encouraged instructors to design AI solutions they believed would best help their students. Drawing on their unique knowledge of their classes and students, the instructors created a wide array of innovations. We show that these instructor-designed AI innovations improved student learning, measured by AI-proof assessments (proctored, in-person examinations). We found no change in student engagement, measured by course activity records, or satisfaction, measured by course evaluations. Our findings demonstrate that AI can improve student learning when its adoption is frontline-driven, with instructors empowered to design and refine AI solutions for their own classrooms.
When left to their own devices, can educators find ways to improve instruction and increase learning with artificial intelligence (AI)? The answer to this question will largely determine the course of higher education in the next few years. Recent studies and university reports paint a bleak picture of “tectonic disruption,” [4, 5, 6] as students prescribed with off-the-shelf chatbots or centrally-designed AI tools offload critical cognitive processes associated with learning[4, 7] and perform worse on corresponding tasks and assignments. [1, 8]
However, this picture is likely incomplete, for two reasons. First, methodologically, many existing studies have limited disciplinary coverage (often a single course or subject) [1, 9, 10], limited exposure time (often hours or weeks rather than full academic terms) [1, 2, 11], lack random assignment[12], or they assess performance with narrowly defined tasks [2] or while AI assistance remains available [12, 13]. Recent reports thus highlight the scarcity of rigorous analyses conducted over longer periods and across diverse disciplines [14, 15]. More importantly, most existing studies focus on one particular type of AI treatment: one-size-fits-all AI tools—either generic AI chatbots or centrally-planned AI agents—which are not amenable to instructor customization, and typically deployed in a top-down fashion[3].
This is not how teaching innovations typically take place in university classrooms, as higher education is intrinsically decentralized: institutions generally avoid mandating uniform rules across classes. The heterogeneous nature of the classroom necessitates giving instructors discretion to choose their teaching style, content, and tools based on their unique understanding of their students. In this regard, AI is likely no different: successful integration of AI will probably require significant input, design effort, and creativity from individual frontline instructors[16].
This instructor-led integration makes our study unique. Rather than prescribe a single AI tool and mandate a single method for using it, we encourage the instructors in our study to use their knowledge of their students and subjects to design an AI-powered solution that best fits their class; we then provide technological support and a knowledge sharing platform to help them realize and deploy their designs. This approach gives instructors ownership and autonomy over the AI in their classrooms [17]—they can create something they are proud of, and dynamically refine it in response to student feedback. The instructors in our study developed a wide array of innovative solutions.
Another distinguishing feature of our study is its scope. We conduct pre-registered randomized evaluations across 7,200 students enrolled in 55 applied science, technology, engineering, and mathematics (STEM) classes (i.e., course-instructor pairs) at 19 institutions, over three academic semesters from spring 2025 to spring 2026 (Fig. 1a,b). We measure learning, engagement, and satisfaction over full academic terms. We compare student performance with AI-free assessments (for instance, proctored paper-based exams) in the years leading up to and following our experimental intervention, with a difference-in-differences evaluation design. Our study thus responds to the emerging call for a comprehensive evaluation of AI in higher education[14, 15, 18].
The participating courses covered a broad range of applied STEM subjects, including statistics, probability, machine learning, optimization, simulation, data analytics and visualization, regression, forecasting, decision analysis, and modeling, with applications in economics, management, health care, supply chain and logistics, information technology, and finance. We grouped classes by subject and academic level (undergraduate or graduate) and randomly assigned classes within groups to implement their instructor-designed AI innovations either immediately (treatment) or with a one-teaching-term delay (control). Following our pre-registered protocol, we excluded classes for which the instructor, curriculum, or assessments meaningfully changed during the study period. For analyses of learning outcomes, we additionally excluded classes whose assessments permitted AI assistance during examinations. The resulting class-level sample comprises 44, 54, and 47 experimental class-terms for learning, engagement, and satisfaction, respectively, whereas the student-level sample comprises 44, 50, and 47. Both samples incorporate an additional 67 pre-experimental class-terms, for historical comparison.
Results
Instructors found diverse uses for AI
With technical support from the research team, the instructors in our study designed and implemented many different AI use cases, tailored to the particulars of their respective classes. These AI interventions—several of which were the first meaningful implementation of their kind in higher education—spanned 33 types, and seven pedagogical functions (Fig. 2). For example, instructors developed interactive simulations (8 classes) and stakeholder role-play sessions (8 classes), in which students interact with—and actively soliciting information from—AI-simulated personas or organizations. Instead of generic teaching assistant bots, instructors designed individualized coaching agents (19 classes) that used various forms of Socratic questioning, as well as tutors (18 classes) that supported targeted practice, examination preparation, and formative feedback. Many instructors (25 classes) also developed AI-powered course navigation tools that complement traditional learning management systems with personalized material delivery and logistics management. Instructors also designed AI agents to help students outside of the core course content areas, by, for example, having students reflect on the personal relevance of the course (6 classes), or help students develop AI literacy (5 classes). Instructors typically combined a mixture of these use cases, with the average class’s AI innovation serving 3.0 distinct pedagogical functions.
Instructor-designed AI solutions improved learning
We measured learning with uncurved scores from AI-proof assessments, typically from proctored, in-person, pen-and-paper examinations. We standardized learning scores to account for heterogeneity at the class level and to measure the treatment effects in standard deviations. We used a pre-registered difference-in-differences regression to compare changes from historical outcomes between treatment and control classes, accounting for class differences and common time effects. The estimates suggest that the instructor-designed AI innovations significantly improved student learning (Figure 3). The student-level treatment effect estimate is 0.32 standard deviations, SD (SE=0.13; P=0.019; 95% CI=[0.06, 0.59]). The class-level estimate is 0.26 SD (SE=0.12; P=0.038; 95% CI=[0.02, 0.51]).1 The results are similar across three estimators (the ordinary least squares (OLS) estimator and two instrumental variable (IV) estimators) and two inference methods (Student’s intervals and CR2-adjusted intervals), at both the student and class levels (Extended Data Fig. ED2).
Figure 3. Effects of instructor-designed AI innovations on learning, engagement, and satisfaction. a, Student-level analysis; b, class-level analysis. Points are OLS estimates of the contemporaneous AI-innovation effect () from the primary specification (Methods), measured in within-class standard deviation units, with CR2 standard errors clustered by class-term in parentheses. Thick bars show 90% and thin lines 95% confidence intervals. is the number of student observations in a and of class-terms, including historical terms, in b.
No detectable effect on engagement and satisfaction measures
For student satisfaction measures, we use the numerical approval scores of either the course or instructor provided in the institution’s end-of-the-term course evaluations. For student engagement measures, we used the best available data for each class: some instructors tracked attendance or in-class participation, while others tracked homework completion, time spent, or activity duration on their respective learning management systems (such as Canvas or Blackboard). We standardized these heterogeneous measures at the class level.
We found no statistically significant effects for student engagement or satisfaction (Figure 3). The engagement effect estimates are SD at the student level (SE=0.14; P=0.60; 95% CI=[, 0.20]) and SD at the class level (SE=0.10; P=0.60; 95% CI=[, 0.16]). The satisfaction effect estimates are SD at the student level (SE=0.10; P=0.29; 95% CI=[, 0.10]) and SD at the class level (SE=0.09; P=0.30; 95% CI=[, 0.10]).
These results are consistent across alternative specifications (Extended Data Fig. ED2).
Student and instructor perspectives
Student perspectives: We invited students in all treatment classes to complete an anonymous end-of-course survey, which included both quantitative and open-ended questions. 2,019 out of 4,898 students responded. The students responded to the statements “I put a lot of effort into this course,” “I learned a lot in this course,” and “I am likely to recommend this class to a peer” on a five-point Likert scale, ranging from “strongly disagree” to “strongly agree”. The mean ratings were 4.00, 3.97, and 3.72, respectively.
The students also rated their AI experience. Overall, the students overwhelmingly rated their experience as positive, with 16.1 times as many students reporting that “The AI learning aids improved the educational experience” as reporting that “The AI learning aids worsened the educational experience,” and 5.1 times as many reporting that “Additional AI learning aids would have improved the educational experience” as reporting that “Additional AI learning aids would have worsened the educational experience.”
Further, the students’ appreciation of the AI innovations correlated strongly with their appreciation of the class (Figure 4). For example, the respective mean scores for prompts “I learned a lot in this course” and “I am likely to recommend this class to a peer” are 4.18 and 3.98 for students who reported that “The AI learning aids improved the educational experience,” but are only 3.35 and 2.96 for students who reported that “The AI learning aids worsened the educational experience.” Both of these differences are statistically significant at the level.
Figure 4. “I put a lot of effort into this course.” 1 = “strongly disagree”, 5 = “strongly agree”
“I learned a lot in this course.” 1 = “strongly disagree”, 5 = “strongly agree”
“I am likely to recommend this class to a peer.” 1 = “strongly disagree”, 5 = “strongly agree”
“Did the professor’s use of generative AI help you learn?”
“The AI learning aids worsened the educational experience.”
“The AI learning aids neither worsened nor improved the educational experience.”
“The AI learning aids improved the educational experience.”
29 107 420 717 724 60 124 363 718 730 138 218 443 470 727
100% 50% 0%
“Should the professor have leveraged generative AI even more to help you learn?”
“Additional AI learning aids would have worsened the educational experience.”
“Additional AI learning aids would have neither worsened nor improved the educational experience.”
“Additional AI learning aids would have improved the educational experience.”
29 107 418 716 726 60 123 364 717 730 138 217 444 467 729
100% 50% 0%
How to interpret the figure
Out of the 729 students who selected 5 (“strongly agree”) in response to the prompt “I am likely to recommend this class to a peer,” 43% selected “Additional AI learning aids would have improved the educational experience” in response to the prompt “Should the professor have leveraged generative AI even more to help you learn?”
Student Survey Responses - Structured Questions Results are based on an anonymous end-of-course survey: 2,019 out of 4,898 students enrolled in treatment classes responded.
In responses to the open-ended question asking what aspects of the AI innovations they most valued (), students most frequently highlighted practice and examination preparation (49.2%)—they liked having access to “endless problems,” and they appreciated the AI’s support for recall, comprehension and problem solving (33.0%), describing interactions that helped “break down complex topics into simpler explanations.” Students also mentioned the value of content tailored to their courses (23.2%): one described tools “built around the class content—it increased my trust.” They valued time savings (14.5%), with one noting, “I didn’t attend office hours for every single minuscule question,” and interactivity and engagement (6.4%), describing the experience as “engaging a different part of my brain.” Students have also noted that these interactions increased their own AI literacy (2.8%), including learning “how different prompts impact outcomes.” Finally, students specifically mentioned support for critical thinking (2.0%): “I like how this AI learning tool doesn’t give answers to my hw questions right away. It gives hints along the way and helped me get to the answer.” Figure 5a provides the summary and Extended Data Table ED1 provides the detailed quotes.
Figure 5. Panel a summarizes themes in students’ responses to the open-ended question “What did you like about the AI learning aids used in this class?” Panel b summarizes themes in responses to “What did you dislike about the AI learning aids used in this class?” and “How can we improve the AI learning aids?” Responses to these two questions overlapped substantially and were therefore analyzed together. Themes were identified independently by a human researcher and a large language model (LLM), then manually reviewed and refined. See Methods and Extended Data Tables ED1 and ED2 for details.
In response to the open-ended question about what they did not like and how to improve (N=1,157), students most frequently requested more regular AI use throughout the course (23.5%), including for regular checkpoints “after every unit,” and more challenging and varied practice questions (22.7%). They also sought closer alignment with course content (20.1%), and clearer, deeper explanations (19.9%). They suggested ways to improve AI response accuracy (10.5%), requested more guidance on AI use (11.1%), such as “examples of effective prompts,” and asked for more engaging activities (6.3%), such as “more AI simulations.” Some students (8.6%) preferred less AI use in class. Students also expressed concerns about overreliance (4.0%), and requested continued human support (3.0%).
Instructor perspectives: We interviewed all 25 instructors who implemented AI in our study. We asked each interviewee (1) what motivated their AI designs, (2) why they considered these designs suitable for their instructional goals, and (3) what they experienced and learned during implementation.
Qualitative analysis of the instructors’ answers to questions (1) and (2) identified seven themes concerning their objectives for using AI and its perceived benefits. Instructors highlighted accessible, on-demand learning support (84%), including help “in the evening or over the weekend when a TA is not available” (Professor 18). They also emphasized scalable practice and personalized support (76%): “Here, you have unbounded capacity, and everybody can do as much as they feel ready to do” (Professor 16). For guided reasoning, critical thinking, and AI literacy (48%), one instructor “prompted [a chatbot] to use the Socratic method, asking them questions instead of blurting out three paragraphs” (Professor 2). Instructors also emphasized using AI to deliver more interactive simulation and applied learning (40%) in class, including gamified activities: “We had a leaderboard and a very gamified version of the DuPont model” (Professor 10). Instructors also sought compatibility with existing courses and teaching approaches (24%), aiming to “bring AI into the classroom while keeping the structure of the course the same” (Professor 1). They also reported using the AI chat logs to identify student learning difficulties (16%), to “see where people are getting tripped up” (Professor 5), and they used AI activities to prepare students for classroom learning (8%), for example by previewing “what is going to happen in the next lecture” (Professor 21). Figure 6a summarizes these overlapping themes, and Extended Data Table ED3 provides detailed accounts.
Figure 6. Instructor interviews: themes. Share of the 25 instructor interviews featuring each theme, with representative keywords and illustrative quotations. a, What instructors were trying to do with AI, and why they thought AI would help. b, What happened when they implemented it. Each interview counts once per theme; themes may overlap.
Qualitative analysis of the instructors’ answers to question (3) identified five themes concerning actual student use, surprises, and adjustments during implementation. The majority of instructors reported positive experiences with AI support (76%): “It succeeded on that front: it let students who wanted to prepare more do so” (Professor 1). They also described uneven and context-dependent AI use (52%), with one instructor observing that “some people used it a lot, while the rest used it once” (Professor 20). Instructors described technical friction and ongoing refinement (36%), including difficulties controlling AI responses and the need to fine-tune prompts: “I changed the prompt several times, but it kept giving out details and answering discussion questions” (Professor 25). They also noted changes in students’ use of human support (24%), with one reporting “fewer students email me about course logistics or come to my office hours” (Professor 6). Finally, instructors reported task-oriented AI use (12%), with one observing increased use typically around deadlines: “As assignment deadlines approach, students use them more and more” (Professor 22; Figure 6b; Extended Data Table ED4).
Discussion
Universities have recently sounded the alarm that AI might undermine student learning and cognitive development [4, 5, 6]. However, after adoption of the instructor-led AI in our multi-site randomized trial, student learning outcomes increased by 0.26 SD at the class level and by 0.32 SD at the student level. These effects are sizable: for comparison, a study of pre-K–12 randomized interventions reported a median improvement of 0.10 SD[19], and an analysis of active learning in undergraduate STEM reported a mean improvement of 0.47 SD[20].
Our positive learning estimates contradict the negative estimates found in the other major studies on AI in education[1, 2, 3]. Our study’s key differentiator is its use of frontline-driven AI. Whereas other studies use a top-down approach, imposing a uniform protocol devised by the researchers, we use a bottom-up approach, spurring instructors to develop AI solutions to problems faced in their particular classes. Empowering instructors to create and continuously improve their own AI innovations leverages their unique knowledge of their classes and their students[21].
The AI innovations addressed longstanding challenges in higher education. Large enrollments limit opportunities for individualized dialogue and feedback[22], and scheduling conflicts and social pressure further discourage students from seeking academic help[23, 24]. The instructors in our study sought to overcome these constraints by using AI to provide personalized explanations, guided practice, and feedback beyond scheduled class hours. Timely, actionable feedback may help students identify gaps, adjust their approach, and continue practicing while they still have the opportunity and inclination to improve[25, 26]. Furthermore, requiring students to attempt problems and explain their reasoning before receiving answers, as most of the instructors in our study chose to do, may help preserve the constructive engagement central to learning[27]. Instructors also used AI to support active and experiential learning through activities such as real-time stakeholder roleplay, allowing students to apply concepts, make decisions, and reflect on their consequences[20, 28, 29]. Finally, as surfaced from our surveys and interviews, AI implementation that focuses on frontline instructors can also support continuous instructional improvement: students’ questions and difficulties can reveal where explanations, prompts, or activities need revision[30].
The results of our study should be interpreted with its limitations in mind. First, the participating classes were primarily applied STEM, and effects in other disciplines and educational settings remain to be tested. Second, few classes were observed during a second or third implementation, limiting conclusions about the persistence of learning gains across repeated course offerings. Third, our engagement measures varied across sites and courses, due to heterogeneity in data availability and learning management systems. This reflects a longstanding measurement challenge in education research[31]: attendance and recorded participation capture only the in-class dimensions of engagement[32], while learning activity outside the classroom are typically unobserved. Finally, our RCT does not have enough statistical power to distinguish effects across different AI innovations and instructional settings. Future studies should determine which innovations are more effective and in which settings, and identify design features that contribute to targeted learning gains.
References
Methods
Ethics
We received ethical approval for all 19 participating institutions, either through a single institutional review board (IRB) reliance agreement, or through independent review by the institution’s IRB or equivalent ethics body. The University of Michigan IRB served as the board of record for the single-IRB arrangement. Reliance agreements were established with Cornell University, Georgetown University, Indiana University, The Ohio State University, Purdue University, Southern Methodist University, Stevens Institute of Technology, The University of Texas at Dallas, the University of Virginia, and the University of Washington. The study was not federally funded, and use of a single IRB was not required. Independent approval was obtained from Carnegie Mellon University, The University of Hong Kong, the Indian Institute of Management Ahmedabad, Northwestern University, the University of Illinois Urbana-Champaign, the University of Pennsylvania, the University of South Carolina, and Yale University.
Pre-registration
The study was pre-registered on the Open Science Framework (osf.io) on 28 February 2025, and amended on 6 March 2025, 15 August 2025, and 24 August 2026.2
RCT design and implementation
Study population The study included applied STEM courses delivered in residential, hybrid, and online formats at participating institutions. Subject topics included statistics, probability, machine learning, optimization, simulation, data analytics and visualization, regression, forecasting, decision analysis, and modeling, with applications in economics, management, health care, supply chain and logistics, information technology, and finance. Overall, there were 30 undergraduate classes, and 25 graduate classes.
Recruitment and eligibility Recruitment proceeded in two stages. The research team first recruited site principal investigators (site PIs), who then recruited instructors teaching eligible courses at their institutions. Enrollment proceeded on a rolling basis, with courses grouped into cohorts by entry term. This structure allowed the study to account for term-specific factors, including changes in students’ baseline AI usage. Eligible instructors were required to (i) have taught the course at least once before joining the experiment, (ii) have not yet systematically integrated AI into the curriculum, and (iii) have no planned major curricular changes during the experimental term other than the proposed AI innovations.3
Participants The experiment involved 19 universities, 42 instructors, and 7,200 students. It comprised 55 classes and 67 experimental class-terms, where a class is a course–instructor pair, and a class-term is a course–instructor–term triple. The three entry cohorts, in Spring 2025, Fall 2025, and Spring 2026, comprised 15, 20, and 15 classes from newly randomized courses, respectively. Participation in previously assigned courses contributed an additional 3 class-terms in Fall 2025 and 14 in Spring 2026 (Fig. 1b).
Randomization Before each semester, instructors submitted course titles, start and end dates, academic levels (undergraduate or graduate), modalities (residential, hybrid, or online), expected enrollments, and prior-year syllabi. We clustered courses by subject and academic level: most clusters comprised two or three classes, but some comprised only one, if the class is like no other. We block randomized assignment within cluster, ensuring that the fraction of each cluster assigned to the treatment group is as close as possible to one-half: e.g., all three-class clusters either have one or two treatment classes. Most professors contributed only one class to the experiment in a term, but some contributed two. When a professor contributed two classes in a term, we randomly assigned one to the treatment group, and the other to the control group. Further, if multiple instructors taught the same course at the same institution in the same term then we allocated their classes to the same experimental arm, with one ad hoc exception: for complicated reasons, two instructors taught a similar subject at the same institution but each to a different audience level (undergraduate vs. graduate) had to be in the same experimental cluster, and had to switch the treatment and control assignment in the second term they participated in the study.
Equal access and student consent For ethical and instructional reasons, we randomized treatment at the course level, rather than the student level. We made sure all students enrolled in a course had equal access to any AI support or activity introduced in that course, regardless of whether they consented to sharing data for research use. Instructors de-identified the student-level data, dropping the learning and engagement scores for the students in the treatment class-terms who did not consent to sharing data. Across class terms, the consent rate had mean 79.4%, and quartiles 60.0%, 84.6%, and 100%.
Outcome metrics Before receiving their randomized assignment, instructors specified empirical measures for learning, engagement, and satisfaction. These measures had to be consistent across teaching terms, both before and during the experimental period. Learning measures were raw, uncurved assessment scores, such as points earned on quizzes or examinations (curved grades were ineligible because they obscure changes in class-level performance). Engagement measures included attendance, class participation, assignment completion, and logins and page views on the learning management system (LMS), such as Canvas or Blackboard. Satisfaction measures stemmed from the institution’s anonymous course evaluations. Classes assigned to the same block randomization cluster were encouraged to choose similar measures, but consistency within a class over time took precedence over consistency across classes within a cluster.
AI innovation proposals Before randomization, instructors submitted a written AI innovation proposal for each participating course. Each proposal described (i) the planned use of AI and (ii) the rationale for that design, including how it may affect learning, engagement, and satisfaction. Proposals had to exceed students’ ambient AI usage, incorporating the instructor’s own design or customization: e.g., allowing ChatGPT on a homework assignment would not suffice, but creating a dedicated homework agent trained on the assignment and supporting course materials would.
Instructor support All instructors in both treatment and control arms received technical and AI-design support before and during the teaching term. Video conferences were held every two months throughout the academic year, covering model prompting, system instructions, principles of agentic AI, and the creation and deployment of agents on different platforms. A typical meeting also featured two to three instructors presenting and discussing their innovations. Two in-person consortium workshops, held in June 2025 and July 2026, provided more intensive training, demonstrations, and opportunities for instructors to exchange ideas.
Asynchronous resources included a teaching-with-AI tutorial [33], demonstration videos, and libraries of sample agent instructions. We also co-developed Faculty Twin with Google’s Rapid Innovation Team. This platform allowed instructors to configure and deploy AI agents through graphical user interfaces in a secure cloud environment compliant with the Family Educational Rights and Privacy Act (FERPA) and approved by IRB.
Data collection Each instructor provided data from their classes after the start of the experiment, and from at least one term before the start of the experiment. As described in our pre-registered protocol, a pre-experimental benchmark was valid only if (i) it corresponded to the same course, instructor, and teaching modality, (ii) it was less than 10 years old, (iii) it was either from the most recent four years of the class or generated more than a year after the class’s creation, (iv) it fell outside the institution’s COVID-19 lockdown period, and (v) its data are sufficient to derive the measure being analyzed.
Instructors provided class-level summary statistics (mean, standard deviation, median, and quartiles) across all students, as well as de-identified individual scores for consenting or non-treatment students, per IRB requirement. All datasets were uploaded to a FERPA-compliant secure server managed by the University of Michigan’s Information and Technology Services.
AI implementation Treatment classes were required to implement their proposed innovations in the assigned term and thereafter, and control classes were required to defer implementation for at least one term.4 The next time a control instructor taught, they could (i) implement their AI proposal, (ii) not implement their AI proposal, or (iii) subject this decision to another round of randomization. In practice, every control instructor chose to implement their AI innovation in the following teaching term.
Attrition and exclusions Figure ED1 describes the filters we imposed upon our sample. We dropped two treatment class-terms from our ordinary least squares (OLS) regressions (but not the auxiliary instrumental-variables (IV) regressions presented in Extended Data Fig. ED2), because they did not meaningfully implement their proposals: one instructor could not implement the AI agents due to a lack of development time5, and the other did not adequately inform students about the agents. And we excluded another class-term from the learning and engagement analyses because of a classification error: we didn’t realize that there were two versions of the class (one for undergraduates and the other for graduates), so we failed to randomly assign the graduate version.
Figure ED1. Inclusion and exclusion criteria for the analysis sample
We excluded four additional class-terms following instructor changes and six more following major changes to the curriculum or assessment formats. We excluded eight additional class-terms from the satisfaction analysis, because their universities changed the course-evaluation survey questions. And, finally, we excluded ten additional class-terms from the learning analysis, because their assessments were not AI-proof. Our definition of AI-proof assessments goes beyond prohibiting AI use through honor codes or written policies in the syllabus. We require verifiable procedures for ensuring that students cannot use AI when they take the assessments. Hence, all the AI-proof assessments in our analysis were proctored, in-person examinations, most of which were paper-based, and the rest strictly monitored, computer-based examinations that used surveillance software, such as LockDown Browser, or restricted-access computers at university-managed testing centers. We did not pre-register this last exclusion criterion, as we did not anticipate the problem of AI use on the assessments. But since real-time AI assistance can artificially enhance performance, we believe imposing this filter is necessary to properly capture true learning outcomes[12, 13].
Analysis sample Restricting to experimental class-terms, our class-level sample comprises 44 class-terms for learning (4,768 enrolled students), 54 for engagement (5,637 students), and 47 for satisfaction (5,020 students) (Figure ED1), and our student-level sample comprises 44 class-terms for learning (4,768 students), 50 for engagement (5,225 students), and 47 for satisfaction (5,020 students) (Figure ED1).6 Including historical terms, our class-level sample comprises 101 class-terms for learning (10,471 students), 117 for engagement (11,394 students), and 102 for satisfaction (7,118 students), and our student-level sample comprises 101 class-terms for learning (9,202 students), 112 for engagement (9,658 students), and 102 for satisfaction (7,118 students).
Empirical specification
Regression model Our regression equations have a difference-in-differences structure, comparing the student performance of the treatment group, relative to their historical average, with the student performance of the control group, relative to their historical average. A given equation characterizes one outcome (learning, engagement, or satisfaction) at one level of aggregation (student or class). For the student-level analyses, we use the following specification:
The dependent variable reports the -score associated with student , teaching term , and class : , where denotes the student’s scores winsorized at 3% and 97% levels, and and denote the mean and standard deviation of these winsorized class- scores, pooled across all of the class’s students and teaching terms.7 The regression explains these scores in terms of fixed effects and , which account for class-level and term-level fluctuations, independent variables and , which account for the effects of AI, and statistical error , which accounts for the residual. Recall a class is defined as a course-instructor pair, so class fixed effects also capture teacher fixed effects.
Both and are binary variables: the former indicates that AI was introduced in period , and the latter indicates that AI was introduced before period . Specifically, if and only if for some , and if and only if the class was assigned to the treatment group in term , or it was assigned to the control group in the last term prior to that the class was taught (recall that all control-group instructors implemented their AI innovations in the next teaching term).8 Note that the lagged effect, , could differ from the contemporaneous effect, , if there’s a learning curve to AI adoption.
For the class-level analysis, we first derive class-level -scores:
where
and
In the expressions above, , , and correspond to the means, standard deviations, and student counts we observe in our class-level data (i.e., aggregate statistics for all students enrolled in a class, regardless of consent for individual-level data). Calculating the aggregate -score in this manner makes the following class-level regression equivalent to our student-level regression (except for the 3% winsorization and differing sample composition):
Note, the scaling assigns equal weight to every student, in alignment with our student-level analysis.
Estimators For our primary analysis, reported in Figure 3, we estimate AI-effect coefficients and with ordinary least squares (OLS), and compute confidence intervals with the CR2 bias-reduced cluster-robust variance estimator, clustering by class-term and using Student- critical values with Satterthwaite degrees of freedom [34, 35]. For our auxiliary analyses, reported in Extended Data Fig. ED2, we consider two other estimators, and another set of confidence intervals. We construct the alternative confidence intervals by combining conventional standard errors, clustered by class-term, with critical values from a Student- distribution with degrees of freedom, where is the number of class-term clusters. And we derive the alternative estimates with instrumental variables (IV) regression.
The Simple IV specification instruments for with
It instruments for with
Here , , , and are binary var-treatment terms and in every term.
ables that respectively indicate a treatment assignment in the current teaching term, a control assignment in the previous teaching term, a treatment assignment prior to the current teaching term, and a control assignment prior to the previous teaching term (note that the subscripts are a slight abuse of notation, because rather than the last term before , they refer to the last term before that class was taught). Note, since all control-group professors implemented their AI innovations in the subsequent teaching term, equals and equals , for all but the two non-compliant treatment-group class-terms. Indeed, this IV regression would be equivalent to our OLS regression if it weren’t for the two treated instructors who did not implement their AI proposals (we exclude their classes for the OLS regression, which cannot accommodate non-compliance, but not for the IV regressions, which can accommodate non-compliance).
The General IV specification instruments for and with the following four instruments:
The numerator variables are defined analogously to those in the Simple IV specification, but with and subscripted with , rather than . The denominator, , is the total number of experimental batches that class enrolled in.9 We discuss these IV specifications in more depth in Supplementary Materials S1.
Extended Data Fig. ED2 reports all 12 combinations of the three estimators, two aggregation levels, and two inference methods. All tests were two-sided, with 90% and 95% confidence intervals.
Student survey textual analysis
We classified the responses to “What did you like about the AI learning aids used in this class?” with the following procedure: First, we exclude responses that are empty, non-informative (e.g., “Good.” without further elaboration), or strictly concerning software functionality (e.g., “the interface is simple”). This filter limited the sample to 1,414 responses. Second, a researcher identified six prominent themes after reading a random sample of 200 responses. Independently, a large language model (gpt-6-astra) identified six themes after reading all responses. The human and AI themes overlapped significantly: combining the lists yielded seven unique themes. We then instructed the LLM to label all responses according to these seven themes. A given response could receive multiple labels. Last, we confirmed the accuracy of the AI’s theme designations by comparing them to the themes specified by a human researcher for a random 200 responses: the human and AI agreed for 95.9% of labeling decisions.
The responses to the other two open-ended questions “What did you dislike about the AI learning aids used in this class?” and “How can we improve the AI learning aids used in this class?” overlapped significantly.10 Thus, we combined the responses to these questions, and analyzed the pooled collection with the procedure described above. Again, the human labeler and the LLM identified overlapping themes (ten in total), and agreed for 95.8% of labeling decisions.
Instructor interview
Following the study, we performed a semi-structured interview with each of the 25 instructors who designed AI innovations for their classes and whose data were included in the analysis sample. Each interview lasted about 25 minutes. 24 instructors were interviewed over zoom online meetings and one interviewed via email due to scheduling conflict. Our semi-structured interviews explored the following questions:
Part 1 - What were you trying to do with AI? Sample prompt: Could you briefly describe your course and the AI intervention(s) you implemented? What teaching problem, pain point, or opportunity were you trying to address, and why did you choose this particular use of AI?
Part 2 - Why did you think AI would help with that problem? Sample prompt: What did AI make possible or easier in this setting? Could you have addressed the same problem without AI, or was there something about AI that made it particularly well suited to what you were trying to do?
Part 3 - What actually happened when you implemented it? Sample prompt: How did students actually use or interact with the AI? Did they use it in the way you expected? Was there anything that worked differently than expected, surprised you, or caused you to change the intervention during the semester?
Interview analysis We conducted a thematic analysis of interview notes employing an inductive approach to identify patterns (themes) within the data. Because instructors’ objectives and how AI benefited were closely connected, we analyzed responses to Parts 1 and 2 jointly. Responses to Part 3 were analyzed separately for implementation experiences, including actual student use, comparisons with expectations, surprises and subsequent adjustments. The research team first manually reviewed the interview notes line by line to identify recurring low-level codes, which were then grouped into higher-level themes based on conceptual similarity, and were revised iteratively through peer debriefing within the team of five primary researchers until they reached consensus. The resulting framework comprised seven themes for objectives and how AI benefited, and five for implementation experiences. Each interview was counted once within each theme. An interview could contribute to multiple themes.
Reporting summary
Further information is available in the Nature Portfolio Reporting Summary linked to this article.
Data availability
All data used in this research are stored on a secure server managed by the University of Michigan Information and Technology Services (ITS), in accordance with Institutional Review Board requirements. Restricted-use research data will be made available to qualified researchers with appropriate IRB approval and upon execution of a data use or memorandum of understanding (MOU) agreement, as applicable.
Code availability
The quantitative analyses were conducted in R. The analysis code is stored on the secure server managed by the University of Michigan Information and Technology Services (ITS) and is available to editors and referees upon request.
Funding
Research reported in this manuscript was supported by discretionary research funds from Northwestern University and the University of Michigan.
Acknowledgments
The authors acknowledge the University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board (IRB-HSBS) for serving as the Board-of-Record for the relying institutions in this multisite study. We particularly thank Corey Zolondek and the IRB-HSBS staff for their leadership and administrative coordination of the single-IRB review, oversight, and reliance arrangements.
The authors thank Gina Fratarcangeli and Google’s Rapid Innovation Team for its collaboration in co-developing the Faculty Twin AI platform to support the research. The authors also thank the University of Michigan Information and Technology Services (ITS) for providing staff and computing support for the deployment and management of the Faculty Twin platform in a secure, FERPA-compliant environment, and Kenneth Moore in particular for his technical assistance.
Author contributions
J. Li, R. Bray, S. Martin, N. Van den Steen and A. Wu are the primary research team and share first authorship.
J. Li oversaw the study: she led instructor recruitment, led ethics protocol development, and contributed to study design, data collection, statistical analyses, Faculty Twin platform development and manuscript preparation. R. Bray conceived the study and led the study design and statistical analyses and contributed to data collection and manuscript preparation. S. Martin led AI innovation, AI training, and in-person workshops and contributed to study design, instructor recruitment and manuscript preparation. N. Van den Steen led RCT protocol implementation, data collection, instructor interviews and qualitative data analysis, and contributed to study design and manuscript preparation. A. Wu led the development of the Faculty Twin platform, its deployment in a FERPA-compliant environment, subsequent upgrades and instructor training, and contributed to study design, instructor recruitment and manuscript preparation. X. Ni led data audits and sample creation and contributed to data collection and statistical analyses.
The site principal investigators were J. Li, V. O. Babich, H. Bastani, C. Blanco, S. Chen, Y. Cui, P. Feldman, D. Hurtado-Lange, Z. Jiang, Z. Lian, M. Lu, M. Park, A. Qi, D. Roy, T. Tan, N. Tereyagoglu, Y. Wang, O. Wu and Z. Yang. They recruited instructors, oversaw ethical and research protocols at corresponding sites.
N. Van den Steen, A. Wu, A. Anderer, E. Astashkina, C. Blanco, C. Chen, S. Chen, B. Collier, Y. Cui, P. Feldman, B. Flicker, Y. Hu, D. Hurtado-Lange, J. Jiang, Z. Jiang, M. Köksalan, Z. Lian, M. Lu, L. Meng, K. Moon, M. K. Newsom, M. Park, C. Parker, A. Qi, M. Qi, D. Roy, N. Sharma, A. Sungu, T. Tan, N. Tereyagoglu, S. Venkataraman, G. Wang, Y. Wang, Z. Yang, Q. Yu, J. Zhang, W. Zhang and R. Zhu recruited participants, designed AI interventions and de-identified student level data. J.-P. Baldwin, O. Celebi, B. Han, I. Kirci, H. Liang and H. Tyagi recruited participants. S. Yu designed AI interventions.
H. Bastani, C. Blanco, P. Feldman, B. Flicker, K. Moon, C. Parker, D. Roy, A. Sungu, T. Tan, N. Tereyagoglu, S. Venkataranman, O. Wu, and W. Zhang provided feedback and editorial input on the manuscript.
Competing interests
The authors declare no competing interests.
Figures and tables
| University | S25 | F25 | S26 | Class-terms | Enrollment |
| Carnegie Mellon University | ● | · | ● | 4 | 412 |
| Cornell University | ● | ● | ● | 7 | 897 |
| Georgetown University | · | · | ● | 1 | 100 |
| Indian Institute of Management Ahmedabad | ● | · | · | 1 | 157 |
| Indiana University | ● | · | ● | 2 | 138 |
| Northwestern University | · | · | ● | 1 | 131 |
| The Ohio State University | · | ● | ● | 7 | 852 |
| Purdue University | · | ● | ● | 3 | 170 |
| Southern Methodist University | ● | ● | ● | 5 | 441 |
| Stevens Institute of Technology | · | ● | ● | 2 | 50 |
| The University of Hong Kong | · | ● | ● | 2 | 155 |
| University of Illinois Urbana-Champaign | ● | ● | ● | 5 | 594 |
| University of Michigan | ● | ● | ● | 7 | 835 |
| University of Pennsylvania | ● | · | ● | 5 | 611 |
| University of South Carolina | ● | ● | ● | 4 | 424 |
| The University of Texas at Dallas | ● | ● | ● | 4 | 318 |
| University of Virginia | · | · | ● | 3 | 342 |
| University of Washington | · | ● | · | 1 | 156 |
| Yale University | · | ● | ● | 3 | 417 |
| Total, 19 universities | 67 | 7,200 |
Table 1. a Participating universities
| Spring 2025 | Fall 2025 | Spring 2026 | Total | |
| Universities | 10 | 12 | 16 | 19 |
| Instructors | 14 | 19 | 26 | 42 |
| Courses | 15 | 20 | 25 | 47 |
| New courses | 15 | 18 | 14 | – |
| Assigned to treatment | 9 | 10 | 7 | – |
| Assigned to control | 611 | 812 | 7 | – |
| Continued courses | 0 | 2 | 11 | – |
| Classes | 15 | 23 | 29 | 55 |
| New classes | 15 | 20 | 15 | – |
| Assigned to treatment | 9 | 12 | 8 | – |
| Assigned to control | 6 | 8 | 7 | – |
| Continued classes§ | 0 | 3 | 14 | – |
| Enrollment | 1,692 | 2,301 | 3,207 | 7,200 |
Table 2. b Participation by semester
fig:1: RCT participation. A class is a course–instructor pair; a class-term is one term of a class. In a, dots mark the semesters in which a university took part (S25, spring 2025; F25, fall 2025; S26, spring 2026) and bars show enrollment summed over class-terms. In b, totals count by university, instructor, course and class across semesters; enrollment sums semesters. Treatment and control counts refer to new courses or classes. 11 12 13
Figure 2. Instructor-designed AI innovations by pedagogical function. Legend on the next page.
Figure 2 (continued). Figure 2 (continued). We characterized AI interventions using the agent prompts used by the instructors, post-implementation instructor surveys, and pre-registered innovation proposals. We restricted this characterization to classes that received treatment and were included in at least one class-level regression analysis, retaining only interventions involving direct student interaction. These restrictions yielded a sample of 30 classes. We combined interventions that created substantively the same student experience, yielding 33 types of distinct student-facing GenAI interventions. Subsequently, we identified seven recurring pedagogical functions and manually assigned each intervention one primary and, where clearly supported, one secondary function. Interventions are listed under their primary pedagogical function. Squares count classes with at least one intervention serving the corresponding function, whether primary or secondary, including interventions listed under another primary function. Each class contributes at most once to a given function’s count, even when several interventions in that class serve that function. This yielded 89 unique class–function pairs across 30 classes, averaging 3.0 distinct pedagogical functions per class.
| Theme | Share of answers | Representative keywords and students’ answers | |
| a | “What did you like about the AI learning aids used in this class?” 1,414 students gave an informative answer. | ||
| Practice more | 49.2% | practice questions · exam preparation · study · review · repetition<br>“...give me practice problems and conceptual questions close to the ones I could expect on the exam.”<br>“It allowed me to have endless problems and ensure that I wasn’t just memorizing the practice problems...” | |
| Better recall, comprehension, and problem solving | 33.0% | recall · understand · clarify · explain · guided steps · problem solving<br>“I liked how the AI tools helped break down complex topics into simpler explanations”<br>“Help me to figure out some formula that i don’t remember.” | |
| Tailored content and increased trust | 23.2% | course content · instructor methods · personalized · adaptive · trust<br>“They were tailored to the class and built around the class content—it increased my trust.”<br>“...trained on class content, often more accurate than ChatGPT” | |
| Save time | 14.5% | quick · instant · convenient · anytime · accessible · less waiting<br>“I didn’t have to go around and spend too much time finding information.”<br>“...made it where I didn’t attend office hours for every single minuscule question” | |
| Interactive and engaging | 6.4% | conversation · follow-up · engaging · fun · interesting · novel experience<br>“I enjoyed the conversational testing nature. It felt like it was engaging a different part of my brain while learning.”<br>“I have never taken a class that utilized a specialized AI... It was a pretty cool experience” | |
| AI literacy | 2.8% | prompting · AI models · building agents · responsible use · future skills<br>“...taught me how to use AI in smarter ways and adapt with it.”<br>“Learning how to properly program an AI model and how different prompts impact outcomes.” | |
| Critical thinking | 2.0% | independent reasoning · hints · think first · evaluate · avoid shortcuts<br>“Told you how it got the answer and allowed for personal thinking before it just gave you the answer”<br>“...It gives hints along the way and helped me get to the answer...” | |
| b | “What did you dislike about the AI learning aids used in this class?”<br>“How can we improve the AI learning aids used in this class?”<br>1,157 students gave an informative answer to either question (pooled). | ||
| Use more | 23.5% | earlier · frequent use · lectures · assignments · reminders<br>“Incorporate checkpoints after every unit where we can engage with the AI.”<br>“...implementing it into the course a little more frequently instead of just midterm/final” | |
| Increase practice difficulty and variety | 22.7% | harder questions · variety · more practice · repetition · conceptual practice<br>“Wider variety of practice problems, with harder questions available.”<br>“Maybe try to make it closer to the actual exam or harder then it.” | |
| More tailored to the course | 20.1% | course content · exams · lecture materials · instructor methods · individual needs<br>“...more aligned with class content and adding more examples that match our assignments and lectures.”<br>“...standardized explanations that follow the same approach taught in lectures...” | |
| Clearer and deeper explanations | 19.9% | clarity · depth · concise answers · examples · guided steps<br>“...more in-depth responses that explain what I may be doing wrong and provide step-by-step instructions.”<br>“...not giving the answers but guiding us to understand why certain functions in excel were used” | |
| AI literacy | 11.1% | instructions · demonstrations · prompting · appropriate use<br>“Just leverage it more and teach students how to prompt to take advantage of the model.”<br>“...best practices for using AI responsibly in coding and data analysis tasks...” | |
| Increase accuracy | 10.5% | wrong answers · hallucinations · calculations · reliability<br>“Give them some way to fact-check themselves, or make it so that they don’t give false information.”<br>“...AI could always be lying, hallucinating, or just a tiny bit off on a small piece of information.” | |
| Use less | 8.6% | optional · non-mandatory · alternatives · reduced use<br>“Don’t use it. and less flipped classroom, more in person...” | |
| More engaging | 6.3% | interactive · conversation · visuals · games · simulations<br>“...more interesting if we had more AI simulations...”<br>“...different exercises where we use AI along with graphs and modules to visualize data in a different way.” | |
| Overreliance concerns | 4.0% | dependence · independent thinking · cheating · shortcuts<br>“Balance pushing students to do their own work while AI is an aid.” | |
| Keep human support | 3.0% | professor · TAs · in person · peers · human connection<br>“It felt very weird staying with AI instead of forming an actual study group and maybe using AI as a tool.”<br>“...I would have liked to have more support from the Professor in terms of lecturing and explaining concepts.” |
Table 5. Figure 5: Student Survey Responses - Open-Ended Questions. Legend on the next page.
Supplementary materials
Pre-registration and Estimators
We pre-registered our study on February 28, 2025, and amended it on March 6, 2025, August 15, 2025, and August 24, 2026. The March 2025 amendment relaxed an exclusion restriction. We initially planned on dropping data from the first two years a course was taught, to avoid classes’ initial growing periods with potentially volatile student enrollments. We decided to be less restrictive, after learning it is practically difficult to enroll classes in the study under this requirement.
The August 2025 amendment provided the first detailed characterization of our estimator. The original version provided a high-level description of the regression model and the difference-in-differences specifications, but it did not explain how we would estimate its coefficients. The August 2025 amendment rectified this omission, describing the General IV specification in detail. In conjunction, we also made a few minor adjustments to the statistical specification, including making the winsorization thresholds symmetric (from 3% and 99% to 3% and 97%), and replacing academic-year fixed effects with teaching-term fixed effects.
The August 2025 amendment also made one modification to our experimental design, in response to our in person workshop in June 2025, where the consensus was that, due to the anticipated proliferation of AI usage across universities, it would soon be difficult to enroll new classes that had not yet incorporated any AI. In response, we updated the protocol to provide instructors options to (i) participate in in multiple experimental batches by subjecting follow-up AI proposals to randomization, and (ii) incorporate other non-randomized AI innovations (i.e., new AI features added outside of the experimental context). All modifications were made before analyzing the data.
The General IV specification instruments for and using the following four instruments: , , , and . This specification divides the independent variables by to correct for instructors self-selecting into multiple rounds of experimentation.
The August 2026 amendment reverted the August 2025 design modification. No instructor exercised either option, thereby making the modification moot. Correspondingly, the August 2026 amendment introduced the Simple IV specification, which is essentially the General IV specification without the August 2025 modification. Specifically, switching to Simple IV drops the denominators; because a class participates in only one round of experimentation, this variable is now mechanically one. The Simple IV specification also replaces the four instruments , , , and with the two instruments and . The rationale for combining and in the Simple IV is that these variables have essentially the same effect without the two modifications, namely making the course imple ment AI for the first time. Likewise for combining and : these variables are almost perfectly co-linear with .
Further, we did not pre-register our OLS regression, but that specification is almost equivalent to the Simple IV specification, differing only in its treatment of the two non-compliant treatment-group class-terms, which, in turn, is similar to the General IV specification, differing only in its construction of the instruments. All three specification yield similar results. We did not pre-register our class-level regressions. Our pre-registration specifies that we use individual-level data, but the release of these student records are sometimes affected by low consent rates. Importantly, we can conduct almost the exact analysis we had pre-registered with our class-level data, which includes all students, consenting and not. Indeed, the only step in our pre-registered analysis that we cannot perform with our class-level aggregates is the 3% winsorization of the distribution’s tails. Other than that, our class-level regression is mechanically equivalent to the student-level regression that we had pre-registered.
Supplementary figures and tables
a Selection of primary analysis samples
Randomized controlled trial participation 67 class-terms · 7,200 student enrollments · Spring 2025–Spring 2026
Learning Excluded: 23 class-terms (2,432 student enrollments)
Class-level primary sample 44 class-terms 4,768 student enrollments
No further exclusions
Student-level primary sample 44 class-terms 4,768 student enrollments
Engagement Excluded: 13 class-terms (1,563 student enrollments)
Class-level primary sample 54 class-terms 5,637 student enrollments
Excluded: 4 class-terms (412 student enrollments)
Student-level primary sample 50 class-terms 5,225 student enrollments
Satisfaction Excluded: 20 class-terms (2,180 student enrollments)
Class-level primary sample 47 class-terms 5,020 student enrollments
No further exclusions
Student-level primary sample 47 class-terms 5,020 student enrollments
b Exclusions from primary analyses
Entries are class-terms (student enrollments).
| Primary documented reason | Learning | Engagement | Satisfaction |
| Before class-level analysis | |||
| Major curriculum or assessment change | 6 (972) | 6 (972) | 6 (972) |
| Assessments not AI proof | 10 (869) | 0 (0) | 0 (0) |
| Instructor change | 4 (383) | 4 (383) | 4 (383) |
| No or minimal AI implementation | 2 (186) | 2 (186) | 2 (186) |
| Missed randomization due to course misclassification | 1 (22) | 1 (22) | 0 (0) |
| University course evaluation change | 0 (0) | 0 (0) | 8 (639) |
| Total excluded before class-level analysis | 23 (2,432) | 13 (1,563) | 20 (2,180) |
| Additional exclusions before student-level analysis | |||
| Unavailable engagement data | 0 (0) | 4 (412) | 0 (0) |
| Total additional exclusions | 0 (0) | 4 (412) | 0 (0) |
Table ED1b.
Class-term = one term of a course–instructor pair. Each excluded class-term is listed under one reason; the same class-term can appear in several outcome columns.
Figure ED2. Estimates across alternative empirical specifications. Points show effect estimates for student learning, engagement, and satisfaction, expressed in within-class standard deviation units; thick bars show 90% and thin lines 95% confidence intervals. Panel a reports student-level estimates; panel b reports class-level estimates. Both panels report contemporaneous effects in the first term of AI implementation and lagged effects associated with having implemented AI previously. Only three class-terms in the eligible analysis sample contributed observations from second or third AI implementations, limiting the precision of lagged estimates. For each outcome, twelve specifications combine three estimators (ordinary least squares [OLS], the general instrumental-variable [IV] estimator, and the simple IV estimator), two aggregation levels (student and class), and two confidence-interval methods (conventional and CR2). OLS analyses exclude the two treatment-assigned class-terms that did not implement their innovations; both IV analyses include them. The IV specifications are described in Methods, Empirical Specifications. Student-level aggregation was pre-registered, but restricting learning and engagement data to consenting students may introduce consent bias (average class-level consent rate 79.4%, with a median of 84.6%). Class-level learning and engagement summaries include all enrolled students, including those who did not consent to individual data use; satisfaction summaries reflect anonymous course-evaluation respondents that do not require consent. Both interval methods use Student- distributions and standard errors clustered by class-term. Conventional intervals use degrees of freedom, where is the number of class-term clusters; CR2 intervals use bias-reduced cluster-robust variance estimates and Satterthwaite degrees of freedom.
| Theme | N (%) | Representative keywords | Sample quotes |
| Practice more | 696 (49.2%) | Practice questions; exam preparation; study; review; repetition | [1] ...give me practice problems and conceptual questions close to the ones I could expect on the exam. [2] It allowed me to have endless problems and ensure that I wasn’t just memorizing the practice problems but actually knew the steps in a new problem |
| Better recall, comprehension, and problem solving | 466 (33.0%) | Recall; understand; clarify; explain; guided steps; problem solving | [3] Help me to figure out some formula that i don’t remember. [4] I liked how the AI tools helped break down complex topics into simpler explanations [5] I liked that the AI tools gave clear, step-by-step explanations for Excel functions and quantitative concepts, which made the material much easier to understand. They also helped me check my work quickly and reinforced what we learned in class . . . [6] I liked that the AI learning aids made it easier to understand complex coding and modeling concepts, especially when working with large datasets and machine learning models. |
| Tailored content and increased trust | 328 (23.2%) | Course content; instructor methods; personalized; adaptive; trust | [7] They were tailored to the class and built around the class content—it increased my trust. [8] .. referred to the correct vocabulary that was in the course... [9] It helped me understand in the same learning approach that the professor used. [10]. . . trained on class content, often more accurate than ChatGPT |
| Save time | 205 (14.5%) | Quick; instant; convenient; anytime; accessible; less waiting | [11]... the AI helped me understand the conceptual things well and made it where I didn’t attend office hours for every single minuscule question [12] I didn’t have to go around and spend too much time finding information. [13] I also didn’t have to wait a long time like for the professor to respond to an email to get a basic explanation. |
| Theme | N (%) | Representative keywords | Sample quotes |
| Interactive and engaging | 90 (6.4%) | Conversation; follow-up; engaging; fun; interesting; novel experience | [14] I enjoyed the conversational testing nature. It felt like it was engaging a different part of my brain while learning.<br>[15] I have never taken a class that utilized a specialized AI in the way Professor Nils did. It was a pretty cool experience |
| AI literacy | 40 (2.8%) | Prompting; AI models; building agents; responsible use; future skills | [16] Learning how to properly program an AI model and how different prompts impact outcomes.<br>[17] ... taught me how to use AI in smarter ways and adapt with it.<br>[18] It made it relatable to what we are likely to see in the future especially with programming our own agent |
| Critical thinking | 28 (2.0%) | Independent reasoning; hints; think first; evaluate; avoid shortcuts | [19] Gave me a chance to think about what facts I needed to apply, as opposed to being given all the facts...<br>[20] Told you how it got the answer and allowed for personal thinking before it just gave you the answer<br>[21] I like how this AI learning tool doesn’t give answers to my hw questions right away. It gives hints along the way and helped me get to the answer – why I think is a much more effective learning process than asking Chatgpt to give the final answer. |
Table ED1. Students’ positive perceptions of AI learning components
Note. In total, 2,019 out of 4,898 students enrolled in treatment classes responded to the survey and 1,689 students responded to the free-text question “What did you like about the AI learning aids used in this class?” Among them, 1,414 were informative responses. Non-informative responses included non-use, no liked aspect, or no substantive information (such as “I liked the AI aid”). Themes were identified through human and LLM reading of the responses with the following process. First, human reader selected a random sample of 200 responses and identified the prominent themes. LLM independently read all responses and also identified prominent themes. Human and LLM both initially identified the first 5 of the 7 themes listed above. The human reader independently identified critical thinking, whereas LLM independently identified AI literacy. We thus arrived at 7 themes in total. LLM then labeled all responses according to the 7 themes. To verify accuracy, the human reader labeled the 200 randomly selected responses and calculated the human-LLM classification agreement rate of 95.9%. Theme labels were not mutually exclusive; a response could contain more than one theme. Therefore, frequencies and percentages do not sum to the sample total or 100%. Percentages in the table are calculated as percentages of the 1,414 informative responses.
| Theme | N (%) | Representative keywords | Sample quotes |
| Use more | 272 (23.5%) | Earlier; frequent use; lectures; assignments; reminders | [1] ... implementing it into the course a little more frequently instead of just midterm/final [2] Incorporate checkpoints after every unit where we can engage with the AI. |
| Increase practice difficulty and variety | 263 (22.7%) | Harder questions; variety; more practice; repetition; conceptual practice | [3] Wider variety of practice problems, with harder questions available. [4] Maybe try to make it closer to the actual exam or harder then it. [5] Help AI create more challenging ideas. |
| More tailored to the course | 232 (20.1%) | Course content; exams; lecture materials; instructor methods; individual needs | [6] ...more aligned with class content and adding more examples that match our assignments and lectures. [7] ...standardized explanations that follow the same approach taught in lectures... tailored to our specific assignments. |
| Clearer and deeper explanations | 230 (19.9%) | Clarity; depth; concise answers; examples; guided steps | [8] ...more in-depth responses that explain what I may be doing wrong and provide step-by-step instructions. [9] Using it to aid in the excel assignments, not giving the answers but guiding us to understand why certain functions in excel were used |
| AI literacy | 129 (11.1%) | Instructions; demonstrations; prompting; appropriate use | [10] One possible improvement would be to provide more structured guidance or recommended use cases for AI tools within the course, such as examples of effective prompts or best practices for using AI responsibly in coding and data analysis tasks. [11] Just leverage it more and teach students how to prompt to take advantage of the model. |
| Increase accuracy | 122 (10.5%) | Wrong answers; hallucinations; calculations; reliability | [12] Give them some way to fact-check themselves, or make it so that they don’t give false information. [13] ...AI could always be lying, hallucinating, or just a tiny bit off on a small piece of information. |
| Use less | 99 (8.6%) | Optional; nonmandatory; alternatives; reduced use | [14] Don’t use it. and less flipped classroom, more in person . . . Better learning experience when following along in person instead of hearing video |
| Theme | N (%) | Representative keywords | Sample quotes |
| More engaging | 73 (6.3%) | Interactive; conversation; visuals; games; simulations | [15] ...more interesting if we had more AI simulations ... [16] ...different exercises where we use AI along with graphs and modules to visualize data in a different way. |
| Overreliance concerns | 46 (4.0%) | Dependence; independent thinking; cheating; shortcuts | [17] Balance pushing students to do their own work while AI is an aid. |
| Keep human support | 35 (3.0%) | Professor; TAs; in person; peers; human connection | [18] I think the AI learning aids were helpful, but I would have liked to have more support from the Professor in terms of lecturing and explaining concepts. [19] It felt very weird studying with AI instead of forming an actual study group and maybe using AI as a tool. |
Table ED2. Students’ criticisms and suggestions for improving AI learning aids
Note. In total, 2,019 out of 4,898 students enrolled in treatment classes responded to the survey. Among them, 1,647 responded to at least one of the open-ended questions “What did you dislike about the AI learning aids used in this class?” and “How can we improve the AI learning aids?” Responses to these two questions were combined for analyses because they overlap significantly. Among them, 1,157 were informative responses about instructors’ AI design. Among the rest, 302 were noninformative responses such as non-use or positive-only responses. There were also 184 software-functionality-only responses such as “make the log in process more smooth” and “allow users to save chat histories”, and 4 off-topic responses. Themes were identified through human and LLM reading of the responses with the following process. First, human reader selected a random sample of 200 responses and identified the prominent themes. LLM independently read all responses and also identified prominent themes. Human reader identified all themes listed above, whereas LLM identified the same themes but separated “Keep Human Support” from “Use Less”. LLM also identified three additional detailed categories (i.e., more scaffolding, more real world problems, and answer verification and disclosure), each accounted no more than 1% of the responses. We therefore combined them with “clearer and deeper explanations”. LLM then labeled all responses according to the 10 themes. To verify accuracy, the human reader labeled the 200 randomly selected responses and calculated the human-LLM classification agreement rate of 95.8%. Theme labels were not mutually exclusive; a response could contain more than one theme. Therefore, frequencies and percentages do not sum to the sample total or 100%. Percentages in the table are calculated as percentages of the 1,157 informative.
Table ED3. Faculty intentions and reasons for using AI
Panel A. What were you trying to do with AI, and why did you think AI would help?
| Theme | N (%) | Representative keywords |
| Accessible, on-demand learning support | 21 (84%) | Concept and syllabus questions; availability; accessible, low-pressure support |
| Professor [2] “I was trying to fill the gap of not being able to talk to each of them or even answer their questions about every single class. Sometimes they are embarrassed to ask things because they feel they are silly, so they ask the chatbot, because it knows all the content from the class.” | ||
| Professor [18] “We are trying to provide an additional resource for the students. Sometimes students may have difficulty finding a TA. Other times, they may be too shy to find a TA, or they face problems in the evening or over the weekend when a TA is not available. So this is quite a help for them.” | ||
| Professor [13] “I think maybe previously, if they needed to come to my office or write me emails to ask questions, and the question was not too significant, maybe they didn’t want to do it because of the hassle. But with these tools available to them, the cost of asking for feedback is lower. I’m thinking that could be one way this helps them.” | ||
| Scalable practice and personalized support | 19 (76%) | More practice problems; tailored explanations; personalized learning at scale |
| Professor [1] “The thing that was most exciting for me about AI was to give feedback and infinite practice problems in a way that I could endorse, that I felt would facilitate understanding and meet students where they’re at. This largely took two different forms. The first was the Faculty Twin, and I set up a custom Faculty Twin instance for each of my exams. I tried to train it to follow the Socratic method and be supportive of whatever the students were able to answer, but slowly break things down and adapt to them. I think this is pretty typical for folks in our consortium, but this seemed really exciting. I could give students all the practice they wanted.” | ||
| Professor [16] “I think scaling for individualization is never possible without AI. There are 160 students, and how do you really personalize for everybody? Here, you have unbounded capacity, and everybody can do as much as they feel ready to do. Depending on the individual, we provide a platform, and the questions keep changing the context. I think that is the beauty of AI. I think generative AI was particularly about preparing everyone in their own way so that they touch the baseline. For me, that was a big game changer. I think the students were very happy with that.” | ||
| Professor [22] “As a student, what are the areas you are most interested in? What are the areas you work on the most? The AI basically goes in there and links these two. It explains blockchain, for example, if a student is a marketing student. It will link blockchain with the marketing concept at that point. […] So again, instead of answering the question in a generic AI model, you have to first ask: what is your background? What have you been working on?” | ||
| Theme | N (%) | Representative keywords |
| Guided reasoning, critical thinking and AI literacy | 12 (48%) | Socratic questioning; challenging assumptions; critical thinking; AI literacy |
Table ED3.
Professor [2] “Those are also student-facing chatbots. We do quizzes with very simple questions. Instead of giving them a PDF with the solutions, I give them a chatbot that can discuss the solution with them. It’s prompted to use the Socratic method, asking them questions instead of blurting out three paragraphs. It’s the same with the midterm and final prep. For the midterm and final, we give them a set of problems that only includes the quantitative material, and then a chatbot that can guide them through it.”
Professor [3] “If you are talking about all the agents I build, that’s the end goal. I want them to practice how they communicate and be critical. For this particular agent and its adversarial approach, the reason is that I think people think harder when they are challenged, and AI provides a safe environment for them to defend themselves.”
Professor [24] “Say why this course is necessary, even though AI can do everything, as one would imagine. And second, what is an instinct you should have as a manager? What things do you want to check? Where can AI go wrong? This was a pen-and-paper-based exam. It was an in-class quiz at the end of every module, a 10-minute quiz where I would give them five scenarios and the AI output for each scenario, and then ask: do you think this is perfect, or would you change something? Or what is the next question you’re going to ask your AI to do?”
| Theme | N (%) | Representative keywords |
| Simulation and applied learning | 10 (40%) | Realistic interactions; simulations; case work; feedback on applied tasks |
Table ED3 (PDF p. 39).
Professor [23] “The agent would simulate being a business owner with a problem. It would say, ‘Hey, I’ve got this problem. What do I do?’ They had to interact with it to tease out the information: what is your objective, really? And what would the coefficients be? It wouldn’t inform them. It would just say, ‘I’ve got this problem. I don’t know what to do to make my business run better.’ So the students had to tease out the information: it turns out he wants to make a product mix choice, but it’s got different revenues and different cost structures. We first have to figure out how much profit is being made. They had to figure out: this is the profit we’re going to be maximizing, but he doesn’t give it to me. He doesn’t even necessarily know what the profit is. I have to do that calculation myself.”
Professor [10] “I ran a simulation on day one, and that was fun. We had a leaderboard and a very gamified version of the DuPont model. I created the financial metrics of two companies, company A and company B, and at the bottom, the names of the companies, but they don’t know which is which. Then they were to mix and match. I gamified that, and we had a leaderboard, so that was fun too. And in real time.”
Professor [2] When explaining the move from a conventional written case to an AI-based case:
“They would load the PDF and ask a few questions, and that’s all they did. So we thought of it as a fun way to fight fire with fire: ‘You’re cheating this way. This is what we’re going to do now.’ It ended up being a lot of fun. The students liked the realistic aspect of it. We get a lot of MBA students who want to do consulting, so they said, ‘This really feels like consulting.’ Even when they said they spent more time on the case, and it was busier than the regular cases, they liked it better.”
| Theme | N (%) | Representative keywords |
| Compatibility with existing courses and teaching approaches | 6 (24%) | Shared course structures; consistency across sections; grounding in course material |
| Professor [1] | “I think this is unlike that of some others: I teach three of our 30 sections annually in core undergraduate operations management. It is a relatively standardized course. I know some other universities have even more standardization, like weekly meetings about what will be taught each week. That’s not what we have, but we do have a common homework system with common homework problems. So I felt relatively constrained in rethinking everything. I didn’t want to change everything about three sections out of 30. I didn’t want to completely change 10% of the sections of a class. That just doesn’t seem very useful. What did I try to do? I wanted to bring AI into the classroom while keeping the structure of the course the same: the same homework and exams.” | |
| Professor [12] | “The big concern we had was that it would start using terms that we didn’t use, or symbols that we didn’t use. It would give people answers, and then they would not really learn things. Those were our biggest concerns. So we put guardrails on it—a pretty verbose, two- to three-page prompt.” | |
| Professor [23] | “I record all my classes, so I had all the transcripts from my prior semesters. I was able to take those transcriptions, give them to ChatGPT, have it create a summary, and upload those summaries as some of the material. I think one of the key things [. . . ] I saw as a strength was that it could explain the material to students in my voice, because I’ve got a very particular way of explaining some of this material. For example, when we’re dealing with reliability calculations: how long is a satellite going to be working? You can do that formula for an exponential distribution using the time between failures or the frequency of failures. It’s basically the same thing, but the formula looks slightly different. Students were sometimes using Google or AI, like ChatGPT, to look up how this works, but they would get the other formula. It’s correct; it’s still the same formula, but it’s the inverse of what you’re doing. You either put in the time between failures or the frequency of failures.” | |
| AI-informed diagnosis of student learning needs | 4 (16%) | Student conversations and attempts; conceptual gaps; instructor diagnostic insight |
| Professor [5] | “Just submit that to Canvas. What was also nice is that I can actually look through them. That allows them to have some privacy: if they want to cut out some portion of the conversation, they can. So I see what they want; they can curate if needed. I can at least read them and see where people are getting tripped up. Or, even if the AI helped them, I can see that the AI was helpful in covering something, and I can generalize that. So it is really helpful to figure out what students are actually experiencing.” | |
| Professor [24] | “And I thought, hey, this is cool, and I would want to know what they don’t understand. It was a smaller class, so I basically had the same idea implemented. Let’s say there is a question with options A, B, C, D. The right answer is B, and you say A. Then I’m like, oh, that’s a great way of thinking about it, but have you considered this? Try to nudge them in the direction of B rather than giving them the exact answer, and then see whether after A they went to C and then came to B, or whether after A that was enough for them to go to B, so at least I would understand where the gaps are.” | |
| Theme | N (%) | Representative keywords |
| Professor [9] “For instance, this manufacturer is ending up with $0 because that team set up the objective as making sure to sell the item. Exactly, same thing here: $12,000 is so low in that simulation. They also set the objective as making sure you sell it. They don’t specify any lower bound.” | ||
| AI-supported preparation for classroom learning | 2 (8%) | Pre-class exercises; next-lecture previews; preparation for classroom learning |
| Professor [5] “What I ended up doing was adapting a little bit from there and bringing in some other things, but it was inspired from there. I had them do one pre-class assignment towards the end of the forecasting unit, where they are interacting virtually with Hannah Banana, who is a designer or head of something like that at a fashion goods firm. There is a whole story about that, and they are doing some more forecasting. She is finding the process really painful, so she describes what she is going through and what she set up. She describes why it is working or not working. And you are her friend: you are just meeting her for coffee, and you have to give her advice. So you are reacting and listening to what she did. For example, every year they have new clothing come out.” | ||
| Professor [21] “That is one of the key functions of OpsCAD, and it also previews what is going to happen in the next lecture. Again, I think in a way I could probably implement all these functions with Canvas as well. But it is awkward to navigate through Canvas. Using generative AI is more conversational, and I feel like it also has more character. So hopefully students find it useful that way.” |
Table ED3 (PDF p. 40). Table ED3 (continued)
Note. These themes summarize objectives and perceived AI affordances elicited by Parts 1 and 2. N = 25 interviews. Each interview counts once per theme. Themes may overlap. Professor IDs refer to the revised participant register. Quotations have been lightly edited for clarity and readability, including removal of fillers, repetitions and false starts, without changing the intended meaning. [. . . ] indicates omitted passages. Adjacent turns by the same speaker are joined.
| Theme | N (%) | Representative keywords |
| Useful AI support in practice | 19 (76%) | Useful explanations and task help; additional and varied practice; positive reception |
| Professor [1] “In the expected way, I knew what students wanted: they wanted more practice. Providing tools that offered more practice, I think, made them better prepared, or at the very least made them not complain to me about it. [. . .] It succeeded on that front: it let students who wanted to prepare more do so, and those who didn’t just had to live with that decision.” | ||
| Professor [11] “Not all students engaged with the AI I trained. However, many of those who engaged with it found it useful. They especially used the apps to get numerous examples from different contexts. Providing examples for the students to practice with was very time-consuming before using AI. The examples were limited, and contexts were rather standard. AI made these diverse and made it possible to scale them up. There were no big surprises.” | ||
| Professor [15] “And because the way we do our notes and our exams, at least back then, were derivatives of what we had done in class, I think the AI was pretty good at handling that. So you gave it a new question that was very similar to something we had done in the notes or in the homework, and then the AI could adapt to that. It was exactly what the AI was meant to do: adapt things to a particular context with slight changes in numbers. So I think it worked really well for them to ask it to explain certain things.” | ||
| Uneven and context-dependent AI use | 13 (52%) | Uneven use; lower-than-expected uptake; optional versus required participation |
| Professor [20] “The other thing I noticed is that I think it was very skewed: some people would use it a lot. For the rest, it was a one-time thing, and that was it.” | ||
| Professor [3] “I learned that throughout the semester. At the beginning, I thought these tools were helpful, so I put them out there for students to use voluntarily. But I found that students don’t use them unless you give bonus points or tie them to a particular class requirement. So, for the two agents, I do give points, and I feel that it is a must for students of this age. If you don’t require them to use it, even though it’s beneficial, I think they’re optimizing their grades using the least amount of time. So probably not many people will use it.” | ||
| Professor [14] “The actual usage is much lower than I expected. [. . .] For the weekly assignment in particular, students just use their own AI tools. Their own AI tools are already good enough; the AI agents do not have to be trained on that week’s content to help them with the weekly assignments. That was beyond my expectation. But other than that, I think the students are using the practice exam AI very extensively.” | ||
| Technical friction and ongoing refinement | 9 (36%) | Prompt adjustments; technical limitations; refining how the tutor responds |
| Professor [25] “I changed the prompt several times, but it kept giving out details and answering discussion questions. So I ended up not using the case [. . .] chatbot.” | ||
| Theme | N (%) | Representative keywords |
| Professor [21] “Summarizing what happens in today’s class: this is where I think the AI agent is very useful. Usually, right after I finish class, I update [. . .] the agent by changing the prompts, so I can customize it and try to clarify things—maybe I wasn’t being very clear in class. [. . .] Or maybe I observe students asking me about certain points after class. I try to clarify those points using the agent, by highlighting them and providing short explanations. I write and update these myself, manually.” | ||
| Professor [9] “In about week five, I had to intervene and add a prompt to make sure it would never release the answers to any quiz material or other questions from the class. After that [. . .] usage has come down a little.” | ||
| Changes in human support use | 6 (24%) | Office hour attendance; email frequency; instructor-student interaction |
| Professor [6] “As I said, it was trained on course, the syllabus and materials, so students could ask questions about the course and logistics, 24/7. In my experience, I feel there has been a significant reduction in the number of students emailing me about course logistics or coming to my office hours.” | ||
| Professor [7] “I’m not sure whether this is a positive or negative thing from a time-use perspective. This is perhaps a positive or more efficient signal, but it also reduces interaction between instructors and students. In some way, they all turn to AI.” | ||
| Professor [14] “I don’t think so. Typically, for elective courses, people don’t have that many questions to bring to office hours. At least from what I heard from the TAs, I do not see an increase or decrease compared with previous years.” | ||
| Task-oriented AI use with limited exploration | 3 (12%) | Quiz-focused use; activity around deadlines; limited exploratory engagement |
| Professor [9] “They only used it to solve the quizzes. My objective was to have something they could talk to, engage with and ask [. . .] more sophisticated questions, and learn about something. That wasn’t the case for the majority of them.” | ||
| Professor [22] “The project agents are a huge hit. They have very high use, which tends to increase before exams. As assignment deadlines approach, students use them more and more.” | ||
| Professor [24] “That was, I think, very surprising to me, given that the conversations in class and around them must have been around AI. I saw big spikes around assignments, and that was a trend I continued to see across these two different implementations of AI over the two years of this experiment.” |
Table ED4. Table ED4. Faculty experiences during AI implementation. Panel B. What actually happened when you implemented it? How did students actually use or interact with the AI? Did they use it in the way you expected? Was there anything that worked differently than expected, surprised you, or caused you to change the intervention during the semester?
Note. Part 3 prompted accounts of student use, expectations, surprises and adjustments, not a comprehensive assessment of whether intended benefits were achieved. Human support accounts include reports of change and no change. N = 25 interviews. Each interview counts once per theme. Themes may overlap. Professor IDs refer to the revised participant register. Quotations have been lightly edited for clarity and readability, including removal of fillers, repetitions and false starts, without changing the intended meaning. [. . .] indicates omitted passages. Adjacent turns by the same speaker are joined.
References
- [1]Hamsa Bastani, Osbert Bastani, Alp Sungu, Haosen Ge, Özge Kabakcı, and Rei Mariman. Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences of the United States of America, 122(26):1–8, 2025. ISSN 10916490. doi: 10.1073/pnas.2422633122.DOI
- [2]Yizhou Fan, Luzhen Tang, Huixiao Le, Kejie Shen, Shufang Tan, Yueying Zhao, Yuan Shen, Xinyu Li, and Dragan Gašević. Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2):489–530, 2025. doi: 10.1111/bjet.13544.DOI
- [3]Jing Liu, Tracy Sweet, Marty Haoyuan Chen, Jonathan Engelberg, Megan Masters, Martyn Clark, Axel Persaud, Alia Lancaster, Jeffrey Hollingsworth, and Jennifer King Rice. The effects of course-integrated AI tutoring on student performance and engagement: A randomized university trial. EdWorkingPaper 26-1598, Annenberg Institute at Brown University, 2026.
- [4]Ad Hoc Committee on AI Use in Teaching, Learning, and Research Training. Report of MIT’s ad hoc committee on AI use in teaching, learning, and research training. Technical report, Massachusetts Institute of Technology, August 2026. URL https://aiandeducation.mit.edu/. Co-chairs E. Klopfer and S. Madden. Dated 13 August 2026; released 25 August 2026. Accessed 6 October 2026.
- [5]Kavita Bala, Alex Colvin, et al. Generative artificial intelligence for education and pedagogy. Committee report, Cornell University, July 2023. URL https://teaching.cornell.edu/generative-artificial-intelligence/cu-committee-report-generative-artificial-intelligence-education. Dated 18 July 2023. Accessed 6 October 2026.
- [6]Generative AI in Teaching and Learning (GAITL) Committee. Generative AI in teaching and learning (GAITL) committee final report and recommendations. Technical report, Brown University, July 2026. URL https://provost.brown.edu/sites/default/files/GAITL_Committee_Report_FNL.pdf. Co-chairs E. Kaldor and M. L. Littman. Provost’s letter dated 7 July 2026. Accessed 6 October 2026.
- [7]Nataliya Kosmyna, Eugene Hauptmann, Ye Tong Yuan, Jessica Situ, Xian-Hao Liao, Ashly Vivian Beresnitzky, Iris Braunstein, and Pattie Maes. Your brain on chatgpt: Accumulation of cognitive debt when using an ai assistant for essay writing task. arXiv preprint arXiv:2506.08872, 4:206, 2025.arxiv.org/abs/2506.08872
- [8]Zhiwei Liu, Haode Zuo, and Yongjing Lu. The impact of ChatGPT on students’ academic achievement: A meta-analysis. Journal of Computer Assisted Learning, 41(4):e70096, 2025. doi: 10.1111/jcal.70096.DOI
- [9]Allen Nie, Yash Chandak, Miroslav Suzara, Ali Malik, Juliette Woodrow, Matt Peng, Mehran Sahami, Emma Brunskill, and Chris Piech. The GPT surprise: Offering large language model chat in a massive coding class reduced engagement but may increase adopters’ exam performances. In Proceedings of the Twelfth ACM Conference on Learning @ Scale (L@S ’25), pages 376–380, New York, 2025. ACM. doi: 10.1145/3698205.3733960.DOI
- [10]Andrew Thoeni and Luke K. Fryer. AI chatbots in higher education: Comparing expectations to evidence. Computers in Human Behavior Reports, 22:101061, 2026. doi: 10.1016/j.chbr.2026.101061. URL https://doi.org/10.1016/j.chbr.2026.101061.
- [11]Greg Kestin, Kelly Miller, Anna Klales, Timothy Milbourne, and Gregorio Ponti. AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15(1):17458, 2025. ISSN 20452322. doi: 10.1038/s41598-025-97652-6.DOI
- [12]Ruiqi Deng, Maoli Jiang, Xinlu Yu, Yuyan Lu, and Shasha Liu. Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies. Computers and Education, 227(August 2024):105224, 2025. ISSN 03601315. doi: 10.1016/j.compedu.2024.105224. URL https://doi.org/10.1016/j.compedu.2024.105224.
- [13]Joshua Weidlich, Dragan Gašević, Hendrik Drachsler, and Paul A. Kirschner. ChatGPT in education: An effect in search of a cause. Journal of Computer Assisted Learning, 41(5):e70105, 2025. doi: 10.1111/jcal.70105.DOI
- [14]Helen Pearson. Universities are embracing AI: will students get smarter or stop thinking? Nature, 646(8086):788–791, 2025. doi: 10.1038/d41586-025-03340-w. News Feature.DOI
- [15]Helen Pearson. How to stay smart in the age of AI: the science of critical thinking. Nature, 657(8133):873–875, 2026. doi: 10.1038/d41586-026-02930-6. News Feature.DOI
- [16]Greg Kessler. Technology and the future of language teaching. Foreign Language Annals, 51(1):205–218, February 2018. ISSN 1944-9720. doi: 10.1111/flan.12318.DOI
- [17]Hyunjoo Moon and Greg Kessler. Exploring ai integration in south korean k-12 education: Teachers’ practices, motivations, and ethical considerations. English Language Teaching, 19(10):88, 2026. ISSN 1916-4742. doi: 10.5539/elt.v19n10p88.DOI
- [18]Frank Pasquale, David Rand, Phoebe Sengers, Marcus Smolka, Abraham Stroock, and Julia Thom-Levy. Subcommittee on technological change and artificial intelligence: Issue summary and core questions. Committee report, Cornell University, Committee on the Future of the American University, November 2025. URL https://provost.cornell.edu/initiatives/future-of-the-american-university/publications-reports/technological-change-ai/. Accessed 6 November 2026.
- [19]Matthew A. Kraft. Interpreting effect sizes of education interventions. Educational Researcher, 49(4):241–253, 2020. doi: 10.3102/0013189X20912798.DOI
- [20]Scott Freeman, Sarah L. Eddy, Miles McDonough, Michelle K. Smith, Nnadozie Oko- roafor, Hannah Jordt, and Mary Pat Wenderoth. Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23):8410–8415, 2014. doi: 10.1073/pnas.1319030111.DOI
- [21]Samantha M. Keppler, Jun Li, and Di (Andrew) Wu. Crowdfunding the front lines: An empirical study of teacher-driven school improvement. Management Science, 68(12):8809–8828, 2022. doi: 10.1287/mnsc.2022.4331. URL https://doi.org/10.1287/mnsc.2022.4331.
- [22]D. Nicol. From monologue to dialogue: improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5):501–517, 2010. doi: 10.1080/02602931003786559. URL https://doi.org/10.1080/02602931003786559.
- [23]J. L. Hsu, M. Rowland-Goldsmith, and E. B. Schwartz. Student motivations and barriers toward online and in-person office hours in STEM courses. CBE—Life Sciences Education, 21(4):ar68, 2022. doi: 10.1187/cbe.22-03-0048. URL https://doi.org/10.1187/cbe.22-03-0048.
- [24]S. A. Karabenick. Seeking help in large college classes: A person-centered approach. Contemporary Educational Psychology, 28(1):37–58, 2003. doi: 10.1016/S0361-476X(02)00012-7. URL https://doi.org/10.1016/S0361-476X(02)00012-7.
- [25]D. J. Nicol and D. Macfarlane-Dick. Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31(2):199–218, 2006. doi: 10.1080/03075070600572090. URL https://doi.org/10.1080/03075070600572090.
- [26]V. J. Shute. Focus on formative feedback. Review of Educational Research, 78(1): 153–189, 2008. doi: 10.3102/0034654307313795. URL https://doi.org/10.3102/0034654307313795.
- [27]M. T. H. Chi and R. Wylie. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4):219–243, 2014. doi: 10.1080/00461520.2014.965823. URL https://doi.org/10.1080/00461520.2014.965823.
- [28]Gerald F. Burch, Robert Giambatista, John H. Batchelor, Jana J. Burch, J. Duane Hoover, and Nathan A. Heller. A meta-analysis of the relationship between experiential learning and learning outcomes. Decision Sciences Journal of Innovative Education, 17(3):239–273, 2019. doi: 10.1111/dsji.12188. URL https://doi.org/10.1111/dsji.12188.
- [29]Yossi Ben-Zion, Turhan K. Carroll, Colin G. West, Jesse Wong, and Noah D. Finkelstein. Leveraging generative artificial intelligence for simulation-based physics experiments: A new approach to virtual learning about the real world. Physical Review Physics Education Research, 22(1):010109, January 2026. doi: 10.1103/s8dy-kqy5. URL https://doi.org/10.1103/s8dy-kqy5.
- [30]Alyssa Friend Wise and Yeonji Jung. Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2):53–69, 2019. doi: 10.18608/jla.2019.62.4. URL https://doi.org/10.18608/jla.2019.62.4.
- [31]Jennifer A. Fredricks, Phyllis C. Blumenfeld, and Alison H. Paris. School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1):59–109, 2004. doi: 10.3102/00346543074001059.DOI
- [32]Dan J. Wang, Neelam Modi Jain, Vanessa Burbano, Jorge Guzman, Daniel Keum, Soomi Kim, Bruce Kogut, and Nataliya Wright. How assigned AI use before class shapes active student engagement in class. Working Paper 7585318, SSRN, 2026. URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7585318. Version dated 7 October 2026; also available as arXiv:2610.10463.arxiv.org/abs/2610.10463
- [33]Robert L. Bray. A tutorial on teaching data analytics with generative AI. INFORMS Journal on Applied Analytics, 55(4):319–343, 2025. doi: 10.1287/inte.2023.0053.DOI
- [34]Robert M. Bell and Daniel F. McCaffrey. Bias reduction in standard errors for linear regression with multi-stage samples. Survey Methodology, 28(2):169–181, 2002.
- [35]Guido W. Imbens and Michal Kolesár. Robust standard errors in small samples: Some practical advice. The Review of Economics and Statistics, 98(4):701–712, 2016. doi: 10.1162/REST_a_00552.DOI





