WorksComputer Science
Perspicacious: An Educational Intelligence Platform for Instructional Designers
Abstract
Designing effective instruction requires instructional designers to create assessment questions aligned with clearly defined knowledge components (KCs). These KCs represent specific knowledge or skills that students must acquire. This process is often time-consuming and relies heavily on expert labor. Although AI for education research has produced a growing body of automated question-generation and KC-discovery methods, few have moved beyond research prototypes into user-friendly tools that instructional designers can readily use. To bridge the research-to-practice gap, we developed and evaluated an educational intelligence platform that unifies question generation and KC discovery behind a single interface to support instructional content development. Named "Perspicacious," our platform aims to help instructional designers "see through" complex data and algorithms to derive instructional insights. A cost model calibrated on 17 production runs revealed a cost-time trade-off: extracting KCs from a 576-question bank cost US $1.98-4.11 and took 36-158 minutes to complete, depending on the budget. A formative, partially Wizard-of-Oz study with eight instructional designers found strong perceived usability (mean SUS 78.8) and surfaced concrete requirements that we treat as prerequisites for deployment.
Code: https://github.com/the-oak-lab/KClusterInterface
Introduction
Recent advances in AI have reshaped and reinforced the role of education in society. While today’s AI can automate many routine tasks, research-backed instructional practices are still needed to develop minds that think critically and creatively. Among the key stakeholders involved in education, instructional designers play a crucial role in delivering effective instruction: they analyze learning needs, develop instructional content, and evaluate learning outcomes [7]—often behind the scenes. In particular, developing instructional content is a key process that requires two related tasks: (1) generating high-quality assessment questions based on high-level learning objectives and (2) categorizing these questions into knowledge components (KCs) [25] that represent discrete units of knowledge. Until recently, these tasks required deep content and instructional expertise, making them difficult to scale.
Scaling these tasks to support personalized instruction is often desired because the theory of mastery learning [6] predicts that most people can develop expertise in any domain if the instruction sets personal milestones. However, developing instructional content has traditionally been a laborious process highly dependent on manual work [36]. In a typical workflow, instructional designers first draft assessment questions and then engage subject experts in a think-aloud study [19] to identify the fine-grained KCs required to answer each question. This process is considered the gold standard for developing KC-aligned instructional content, but it does not scale well to large courses—not to mention the many fluid subjects on frontier AI techniques (e.g., agentic engineering) where new skills emerge rapidly. These courses often require frequent updates and continuous development to keep up with the evolving knowledge and the growing demand for personalization. This places a heavy burden on instructional designers who rely largely on manual processes.
Fortunately, decades of research have yielded a wealth of methods and tools that can automate question generation [10, 33] (Wei et al. 2026) and KC discovery [30, 34] (Wei et al. 2025), with the vision of significantly reducing the workload for instructional designers. Although these methods have been rigorously validated on benchmark datasets or in lab experiments, most remain research prototypes that are not yet ready to integrate into instructional design workflows. If an instructional designer needed to create and categorize thousands of questions by tomorrow, their best bet would still be “asking ChatGPT”; very few would read the latest research papers and try to apply the cutting-edge methods, only to stop at configuring the runtime environment.
This mile-wide gap between research and practice exists because researchers and instructional designers have different mindsets and priorities. Researchers often focus on building prototypes that improve performance on specific metrics, with no particular consideration for real-world deployment. In contrast, instructional designers prefer intuitive and out-of-the-box tools that seamlessly integrate into their existing workflows without requiring extensive technical expertise. As a result, improved algorithmic performance does not always translate into practical benefits for instructional designers, especially if it raises the technical barrier to use. For research output to benefit instructional designers, a user-centric integration is needed to streamline the last-mile adoption of AI tools in instructional design workflows.
In this paper, we introduce Perspicacious, an AI-powered educational intelligence platform that aims to bridge the gap. The name Perspicacious reflects our vision of providing instructional designers with a clear and insightful view of the instructional design process by helping them “see through” the intricate student data and modeling algorithms involved—much like how business-intelligence tools help analysts interpret transaction data. The platform aims to integrate instruction-related information, tools, and resources into a unified interface to streamline existing workflows—including question generation and KC discovery. We describe the design considerations, system architecture, and cost analysis of Perspicacious, and report the results of a formative user study with eight instructional designers. Our findings highlight the importance of usability, seamless integration into existing workflows, and clear explanations of AI-generated outputs to foster trust and adoption among instructional designers.
Related Work
Generating high-quality questions and identifying KCs from questions are two central tasks in instructional design. We briefly review the literature on AI-assisted question generation and KC discovery, highlighting the progress made in both areas, as well as the existing research-to-practice gap that Perspicacious aims to bridge.
AI-assisted question generation has been an active research area for decades [26]. Early approaches include rule-based methods [1, 20] (Wang et al. 2008) and sequence-to-sequence models using recurrent neural networks [18] (Du et al. 2017). More recently, LLMs have become the de facto standard for question generation. A large body of work [5, 21, 33] focuses on developing effective prompting strategies to generate questions that align with the revised Bloom’s taxonomy of cognitive complexity [3]. Some recent work has also explored the use of smaller language models for question generation, by fine-tuning [10] (Li et al. 2025) or extensive generation (Wei et al., 2026). A knowledge component (KC) was first conceptualized as “an acquired unit of cognitive function or structure that can be inferred from data” [25], but the idea of categorizing questions into discrete units of knowledge can be at least traced back to Knowledge Tracing [13]. KCs are important building blocks of adaptive learning systems that support mastery learning [6] and are fundamental to evidence-based instructional design that derives educational insights from student data.
AI and machine learning have been the backbone of many automated KC discovery methods. Many automated methods require student data to provide supervision signals. Early approaches include the Q-matrix method [4] and its sequels based on matrix factorization [16, 15], which search for an optimal question-KC mapping that best predicts student performance. Learning Factors Analysis [12] instead searches with [32] over merge and split operations on an existing KC model. Deep-learning-based variants learn the question-KC mapping end-to-end, either from the hidden states of a correctness predictor [35] or by extending the Q-matrix with a recurrent architecture [31]. LLMs have been applied to KC discovery without requiring student data. Examples include methods that simulate instructional experts [30], extract multimodal signals [29], and use next-token probabilities to detect patterns of question co-occurrence (Wei et al., 2025).
Instructional-design platforms exist at either end of the workflow. DataShop [24] gives researchers tools to fit and compare KC models, but presupposes the availability of student data; authoring tools such as CTAT [2] let non-experts build and deliver tutors, but take the questions and the KC model as given. Perspicacious sits between them, producing both from a list of learning objectives or an existing question bank.
Despite the impressive research progress so far, a significant research-to-practice gap remains. Very few (if any) of these automated methods have evolved from a research prototype into a user-friendly tool that instructional designers can readily use. Worse, most require more technical expertise than a typical instructional designer has. For example, even the seemingly simple approach to extract KCs using an LLM involves elaborate prompt engineering or fine-tuning that may overwhelm non-experts, not to mention that LLMs tend to produce an excessive number of KCs with near-duplicates [30]. Perspicacious allows instructional designers to upload questions in various formats and receive a KC report without ever knowing its internals—as laid out in its user-centric design considerations.
Design Considerations
Learning to use a new technology can cause stress and anxiety [8], even if it increases productivity. Instructional designers who are already managing a heavy workload may feel overwhelmed by a new platform, and the fear of making mistakes or not understanding it defeats the purpose of adopting it. We therefore grounded Perspicacious in three user-centric design considerations.
Low Technical Barriers
Although we expect our users to become more familiar with AI as more AI literacy programs unfold, using Perspicacious should not require any prior knowledge of AI or machine learning. The platform is designed to require minimal technical expertise. Users can upload a simple plain-text file of learning objectives to generate questions or upload existing questions in various formats (e.g., CSV, XLSX, or JSON) to receive a report mapping each question to a KC. The system handles all intermediate processing while keeping users informed about the progress.
Asynchronous Workflows
Instructional designers often have busy schedules and may be reluctant to devote a lot of time to Perspicacious, so the platform supports asynchronous workflows. Users upload files at their convenience and receive notifications when the job is done, allowing them to manage other duties and easing the fear that a new tool will disrupt their existing workflows.
Flexible Scalability
Users have varying expectations of the service quality. When analyzing a question bank, some users may prefer a minimal turnaround time regardless of the cost, while others may be more sensitive to budget constraints and willing to accept a longer processing time for lower costs. Perspicacious must adjust to different scalability requirements based on the user’s expected service quality.
System Architecture
The design considerations led us to build Perspicacious as a web application supported by a modular architecture. The web interface facilitates user interactions with two innovative, LLM-based components that would have been difficult for instructional designers to use directly: a KC extraction engine, which is deployed and characterized in § 5, and a question generation engine, which is implemented but not yet integrated into the production pipeline. We describe the algorithms behind each engine and the web interface. More implementation details are described in Appendix § 10.
KC Extraction Engine
The KC extraction engine is a more scalable implementation of KCluster (Wei et al., 2025), a novel KC extraction algorithm based on an LLM. Given a question bank, KCluster identifies clusters of similar questions that potentially belong to the same KC. It quantifies question similarity based on a novel measure of question co-occurrence. For every pair of questions, and , KCluster uses Phi-2 [22] as a native probability machine to evaluate the change in log-probability of given :
and defines a symmetric measure of question similarity:
The idea is that if the occurrence of significantly increases the probability of occurring (as measured by the conditional, ) over the baseline occurring rate , then and are congruent—they may come from the same textbook chapter, concern the same topic, or contain similar texts, all suggesting that they belong to the same KC. On the other hand, if barely changes or even decreases the likelihood of occurring, then and are incongruent and likely to belong to different KCs. KCluster then applies a clustering algorithm to systematically determine whether two questions belong to the same KC based on the congruity between all pairs of questions.
In the original formulation of KCluster, it requires evaluations to extract the concept labels for all questions and evaluations to calculate the congruity for every question pair—a total of evaluations that do not scale well if is large. To support asynchronous workflows and flexible scalability for Perspicacious, we re-implemented KCluster on Google’s Vertex AI platform to leverage its powerful distributed computing capabilities and asynchronous processing features (see Appendix § 10). In addition, Vertex AI allows us to estimate the computing resources to allocate based on a user’s desired completion time to accommodate workloads flexibly (more details in § 5).
To illustrate the engine’s output, the 64 questions generated for our user study (§ 6) produced eight KCs, and 63 of the 64 questions were grouped with the other questions written for the same objective (see Table 1 in the appendix). The exception, “What is the carbon cycle?”, was placed with questions about carrying capacity. Because the questions were generated from those objectives, this is only a sanity check; on real question banks, KCluster’s KC models predict student performance better than the best expert-designed models available (Wei et al., 2025).
Table 1. Breakdown of the 64 questions used in the study by the learning objective they were written for and the knowledge component Perspicacious assigned them to. Seven of the eight objectives map onto exactly one KC; only the carbon-cycle objective is split, with a single question (“What is the carbon cycle?”) placed under the carrying-capacity KC.
| Learning objective / discovered knowledge component | Questions |
| Define the demographic transition | 8 |
| identify the correct definition of the demographic transition | 8 |
| Describe causes of and problems related to overfishing | 8 |
| identify a likely cause of overfishing | 8 |
| Describe methods for mitigating human impact on forests | 8 |
| identify a method of mitigating human impact on forests | 8 |
| Describe sustainable agricultural and food production practices | 8 |
| identify the environmental and economic benefits of sustainable agricultural and food production systems | 8 |
| Describe the impact of carrying capacity on ecosystems | 8 |
| identify the impact of carrying capacity on ecosystems | 8 |
| Describe the impacts of human activity on wetlands and mangroves | 8 |
| identify the impact of human activity on wetlands and mangroves | 8 |
| Explain the environmental effects of excessive use of fertilizers and detergents on aquatic ecosystems | 8 |
| identify the environmental effects of excessive use of fertilizers and detergents on aquatic ecosystems | 8 |
| Explain the steps and reservoir interactions in the carbon cycle | 8 |
| identify the impact of human activities on the carbon cycle | 7 |
| identify the impact of carrying capacity on ecosystems | 1 |
| Total | 64 |
Question Generation Engine
The question generation engine replicates a recent “Generate-Then-Validate” pipeline (Wei et al., 2026) that uses an LLM to create high-quality questions from learning objectives. The pipeline proceeds in two main stages: expansive generation and selective validation.
In the expansive generation stage, Phi-2 is prompted to generate a large number of candidate questions for each learning objective, regardless of question quality. These questions are then validated and filtered through a three-step validation process. The first step is syntactic filtering, which removes any question that violates basic syntactic rules (e.g., containing incomplete options). Next, Phi-2 is used to evaluate the probability of the answer key given each question, and questions with low answer-key probabilities () are discarded. Finally, a similar notion of change in log-probabilities is extended to question-objective pairs to measure the relevance of each question () to each learning objective ():
The idea is again that if a learning objective elicits a large increase in log-probability of a question over its baseline occurring rate, the learning objective is likely to be relevant to the question. The pipeline removes any question whose original learning objective does not yield the highest relevance score among all competing learning objectives.
Web Interface
Users need to know none of these technical details thanks to the user-friendly web interface. The web interface is built with Django and deployed on Google Cloud Run in a Docker container. After signing in, users see a dashboard (Figure 3 in the appendix) summarizing their past and present tasks, from which they can track progress, download the generated questions or KC report, or run further KC analyses.

Figure 3. An intuitive dashboard showing an overview of a user’s KC extraction tasks with the user’s expected quality of service. When the batch job is completed, the web application retrieves the KC labels and values before applying the clustering algorithm to derive the KCs. The user is then notified to download a CSV report that maps each question to a KC.
The “New Analysis” button opens the upload page (Figure 4 in the appendix). For question generation, users can upload a plain-text file containing the target learning objectives. For KC extraction, we accept a wider range of formats that question banks may already come in, whereas the original KCluster algorithm requires a specific JSONL input format. To minimize user effort, Perspicacious accepts Microsoft Excel (.xlsx), CSV (.csv), and JSON (.json) formats, and automatically converts them into the correct JSONL structure. To reduce formatting errors, the interface provides (1) clear field descriptions explaining what information is required, (2) example files showing correctly formatted data, and (3) downloadable templates for each supported format. These aim to ensure that even first-time users can prepare correct input without extensive trial-and-error. Both engines are hosted inside the deployment’s own Google Cloud project, so uploaded questions never reach a third-party LLM service; retention and access policies for those uploads remain the hosting institution’s to set.

Figure 4. An informative upload page guiding the user to upload questions in a variety of supported formats
Cost and Scalability Analysis
The cost of running Perspicacious is determined by the computing resources allocated to a task: more resources shorten the turnaround time but raise the cost. To meet the varying expectations as described in our design considerations (Section 3), Perspicacious can flexibly allocate resources based on the user’s desired completion time.
We analyze the cost and scalability of the KC extraction engine. The engine uses Phi-2, a lightweight 2.7B-parameter LLM, which we host on the most affordable g2-standard-4 instances on Vertex AI. Each instance includes an NVIDIA L4 GPU with 24GB of memory. At the time of writing (August 2026), the cost of running one such instance is US $0.8129 per hour. If the actual completion time is minutes, the actual cost is given by:
The number of instances to allocate () can be determined by the user’s desired completion time. Let be the number of questions uploaded by the user. As mentioned previously, the total number of predictions to make is , which includes predictions for concept labels and predictions for values. Vertex AI processes these predictions in batches of size , which we typically set to 16 to fully utilize the GPU memory available. Each batch takes seconds to process, which we estimate empirically; overestimating over-provisions the job and inflates cost, while underestimating it starves the job and lengthens processing. Vertex AI also documents that each instance requires some time (15 minutes or so) to set up before any request is served. If the desired completion time is minutes, the number of instances to allocate can be estimated by:
The actual completion time varies with system load and resource availability: across runs at matched and with different question banks, completion times agreed to within 4.4–14.6%, and two question banks at agreed to 8.1%.
In practice, a user uploads questions and indirectly sets the number of instances via their desired completion time . A good measure of the system workload is then batches per instance, , which accounts for the different budget constraints conveyed by . To understand how the actual completion time relates to , we fit a linear regression model to the job completion statistics of 17 jobs. These jobs were completed for a previous project that extracted KCs from 13 datasets curated by Cao et al. [11], spanning science courses at four schools.
Figure 1a shows that the linear model fits well to the data () across different . It estimates the processing speed seconds per batch and the fixed start-up overhead minutes, some 30% above the documented figure; we use this fitted value in Equation (5). Substituting in Equation (4) with the estimated linear model gives an informative decomposition of the actual cost :
At , the median and mode size of the 13 datasets analyzed, this simplifies to . In other words, extracting KCs from 576 questions costs an irreducible $1.78 plus an overhead proportional to . Under the same linear model, the actual cost is a rational function of with a vertical asymptote at : . Figure 1b shows such a cost curve at across nine jobs run at various . We observe a diminishing return for adding more instances as approaches the irreducible floor at . Extracting
(a) Actual completion time against batches per instance, over 17 production KC-extraction runs. The fit gives seconds per batch and a fixed overhead of minutes.
(b) Cost against actual completion time for a 576-question bank. Markers are the nine measured runs, each at a given ; the curve is drawn solid over the measured range and dashed beyond it.
Figure 1. Cost and scalability analysis of the KC extraction engine.
KCs from a 576-question bank cost between $1.98 and $4.11 and took 36 to 158 minutes, depending on the budget. After Perspicacious is deployed, we can collect more usage data to better estimate the resources needed for a given job and provide greater cost transparency for users.
User Study
To evaluate its usability and pedagogical utility, we conducted a user testing study of Perspicacious’s ability to enable instructional designers to integrate two existing AI-based methods: KCluster (Wei et al., 2025) for KC discovery and Generate-Then-Validate (Wei et al., 2026) for question generation. This study goes beyond algorithmic evaluation to examine real-world user interaction, adoption barriers, and the interpretability of outputs in an instructional context. The study was conducted between January 2026 and June 2026.
Participants
We recruited eight instructional designers from within our professional network. All participants regularly work with assessment questions and have varying levels of experience with tools that extract KCs and generate questions. Although participants had professional experience in instructional design, they were not domain experts in the specific subject area used in the study materials; accordingly, their evaluations focused on usability, alignment, and pedagogical plausibility, rather than on fine-grained content accuracy. We refer to our participants as P1–P8 in order of session date.
Procedure
Participants were invited to use the platform in a think-aloud session lasting about 40 minutes. They completed two tasks:
Upload a plain-text file of eight learning objectives to generate questions. The learning objectives were chosen from the same Environmental Science curriculum used by Wei et al. (2026).
Download a CSV file of the 64 generated questions and re-upload it to generate KCs for each question.
After each task, participants reviewed a subset of the generated questions and KC assignments and rated them in terms of clarity and perceived usefulness. Finally, participants completed a post-study survey consisting of usability questions and additional open-ended questions asking how they might integrate the platform into their workflows, as well as any remaining challenges or points of confusion. All experimental procedures were approved by our institution’s IRB.
Evaluation Scope
Our user study evaluated the interface and the end-to-end workflow rather than engine performance, and was conducted as a partial Wizard-of-Oz evaluation [14, 23]. Account creation, file upload and validation, format conversion, the task dashboard, and email notification were live. The questions and KC assignments returned during a session, however, were prepared in advance with the same Generate-Then-Validate (Wei et al., 2026) and KCluster (Wei et al., 2025) pipelines rather than computed in real time, because a KC extraction job takes about 20 minutes to start (§ 5)—too long for two think-aloud tasks, output review, and a survey to be done in a 40-minute session. The question generation engine was not yet integrated. Participants were explicitly told that parts of the system were simulated and that outputs might not reflect the fully integrated version. The results below therefore characterize the workflow and practitioners’ judgments of output quality, not end-to-end latency or reliability when the platform goes live.
Measures
Quantitative measures included task completion rate and usability scores, assessed using the 10-item System Usability Scale (SUS) [9]. Qualitative data consisted of thematic analyses of interviews focusing on ease of use, clarity of outputs, and suggestions for improvement.
Results
All eight participants completed both tasks. For P4, the dashboard did not update before time ran out and the facilitator supplied the prepared output; P8 met an unsupported file format and recovered by exporting to CSV. In the post-study survey, seven of the eight participants anticipated using the platform in their own work, while one did not.
Usability
Participants rated each SUS item on a five-point Likert scale, and we scored the items according to the standard procedure (see Appendix § 11). For our study, participants found the web application easy to use and learn (raw scores 26–37, corresponding to standardized SUS scores of 65–92.5; M = 78.8, SD = 10.3). The average score falls in the “B+” range on the curved grading scale of SUS benchmarks, indicating good perceived ease of use and learnability [27]. Figure 5 in the appendix shows the overall and item-level score distributions.

Figure 5. System Usability Scale (SUS) evaluation of the platform. The left panel displays the overall distribution of standardized SUS scores (0–100) using a box-and-whisker plot overlaid with individual participant data points. The system achieved a mean standardized SUS score of 78.8 (SD = 10.3), placing it above the industry-average usability baseline of 68.0 (indicated by the dashed gold line). The right panel provides an item-level usability footprint. To allow direct comparison, Likert-scale responses (1–5) are mathematically adjusted to a uniform 0–4 scale (where 4 represents optimal usability) following standard SUS scoring protocols. Error bars represent the standard error of the mean.
Qualitative Feedback
To understand the perceived utility of the platform, we analyzed the think-aloud transcripts and post-session surveys of the eight instructional designers. Our findings center on participants’ existing workflows, the friction in current AI-assisted practices, the platform’s practical utility, and the conditions they require for adoption.
Business as Usual
Participants described assessment design as a highly manual process under persistent time constraints. They use general LLMs such as ChatGPT and Gemini for brainstorming, but conversationally rather than integrated into established workflows. P3 reported they “use ChatGPT every week for brainstorming questions,” while P5, a learning engineer, wished their team had “a system better than generic ChatGPT.” These ad hoc uses have limits: generic LLMs produced “trivial distractors” at scale (P3), requiring item-by-item revision, and some participants wanted more robust ways to derive KCs from questions of varying quality. Practitioners struggle not with access to LLMs but with the lack of tools that integrate generation, validation, and refinement into a coherent workflow.
Platform Utility and Usability
Participants responded positively to the platform’s ability to provide access to validated assessment-generation methods through a simple visual interface. Rather than requiring Python scripts or command-line tools, practitioners could directly invoke the Generate-Then-Validate pipeline and obtain assessment items they perceived as more rigorous than typical conversational LLM output. P8, who creates assessment items daily, remarked that “this would be very helpful,” while P2 described the interface as “clear and nice from a usability standpoint.” Participants also valued the quality of generated answer choices and distractors, noting that they appeared less superficial than those produced by ChatGPT and could support meaningful instructional discussion. The low technical barrier was similarly emphasized, with P3 reporting that they could have learned to use the system independently within minutes because the templates were easy to understand.
Trust as a Precondition for Adoption
Two participants made adoption conditional on trusting the output. P1 would use the platform “if I can trust the quality,” naming distrust of “the generated results” as the only reason they would not; P2 cited distrust of “AI hallucination” and asked to ground generation in uploaded source material. Neither engine currently exposes why a question or KC was produced, suggesting that provenance and verification are as important for adoption as usability.
Discussion
Participants’ strong response to low-barrier access to the Generate-Then-Validate and KCluster pipelines suggests that the research-to-practice gap is not just a question of algorithmic performance, but also of whether effective methods can be embedded in workflows that practitioners can adopt.
Improving Cognitive Fidelity of KCs
Five participants noted that the generated KCs all used the same action verb “identify” (see Table 1 in the appendix). Although the underlying concepts were generally judged to be appropriate, this repetition reduced the usefulness of the KCs for distinguishing between lower- and higher-level learning goals. Using a wider range of action verbs could make the generated KCs more informative for instructional planning and better support alignment with cognitive frameworks such as the revised Bloom’s taxonomy [3].
Requirements for Deployment
Deployment is more within reach than what our prototype may have suggested. The KC extraction engine already runs in production on Vertex AI—the 17 jobs analyzed in § 5 processed 13 real question banks from four schools—and account creation, upload, validation, the dashboard, and notification all ran live in the study. Integrating the question generation engine is the one remaining engineering step, and our participants set four requirements for that release.
The first is steerability. Participants valued automated generation while requesting control over difficulty, reading level, question type, and item refinement: practitioners appear to want more steerable automation rather than autonomous automation. The goal is not to remove the instructional designer, but to accelerate routine work while preserving their agency. The second is expressiveness: several wanted explanatory, option-specific feedback deliverable to students, and others wanted formats beyond multiple choice (e.g., open-ended, scenario-based, short-answer, and essay questions) to cover a wider range of learning objectives. The third is provenance. Because adoption was conditional on trust, the release must show why each question or KC was produced, so that instructional designers can verify rather than assume.
The fourth comes from our cost model. Because every job pays a fixed start-up overhead of about 20 minutes (§ 5), small jobs are dominated by provisioning: the model puts the study’s 64-question bank at 21 minutes on a single instance, 92% of it spent waiting for that instance to start, against 13% for a full 576-question course bank. This is precisely the scale at which a new user is likely to try the platform, so keeping a warm instance available, or batching small jobs together, would make first contact with Perspicacious feel immediate rather than slow.
Conclusion and Future Work
We introduced Perspicacious, an educational intelligence platform that supports both automated question generation from learning objectives and KC extraction from existing assessment items, extending KCluster and Generate-Then-Validate into an end-to-end workflow with low technical barriers for instructional designers. A formative user study demonstrated strong usability and positive perceptions of the platform’s pedagogical utility. Future work will focus on refining the system based on user feedback, such as increasing control over question generation parameters and enhancing KC expressiveness, and on follow-up evaluations with subject experts and classroom instructors to assess content accuracy and instructional impact in authentic educational settings.
Acknowledgments
This research was supported by the US National Science Foundation under Grant No. 2301130 and a Google Academic Research Award to Paulo F. Carvalho.
System Implementation
The Perspicacious web application is hosted on Google Cloud Platform. When a user uploads a file, it is uploaded to a designated bucket on Google Cloud Storage, and a Cloud Run job corresponding to a task on the dashboard Figure 3 is then created to begin the appropriate task. The job first verifies that the uploaded file satisfies the requirements and then converts the file into the required input format, which is then processed and sent to the appropriate (KC extraction or question generation) engine hosted on Vertex AI (now named “Gemini Enterprise Agent Platform”). A relational database is maintained via Cloud SQL to store user and task information. The web application uses multithreading to monitor the status of each task. Once a task is completed, the user is notified via email to download the generated questions or KCs. Figure 2 illustrates an example workflow when a user initiates a KC extraction task.
Figure 2. An example workflow initiated by a user to extract KCs from questions
KC Extraction Engine
The KC extraction engine runs a more scalable implementation of the KCluster algorithm (Wei et al., 2025). The KCluster algorithm first extracts concept labels from each question and then computes in Equation (1) for every pair of questions and . With reference to the original implementation2, we built a Docker image that packs label extraction and calculation as a Custom Prediction Routine and registered it as a Model on Vertex AI. When a user uploads questions, the web application prepares a JSONL file of lines, each of which is a prediction request containing the prompt for either label extraction or calculation ( requests for label extraction and requests for calculation). The web application then initiates a BatchPredictionJob to serve the requests. This process is fully managed and optimized by Vertex AI, which applies MapReduce to parallelize the computation and allocate sufficient resources commensurate
Question Generation Engine
Although we are still in the process of fully integrating the question generation engine into Perspicacious (see § 6.3), we plan to pack a scalable implementation of the Generate-Then-Validate pipeline (Wei et al., 2026) in a Docker image and register it as another Model on Vertex AI, similar to the KC extraction engine. When a user uploads a set of learning objectives, the web application will prepare a JSONL file containing prediction requests for the generation stage. After initiating a BatchPredictionJob for the generation stage, the web application retrieves the generated questions and prepares another JSONL file for the validation stage. A second BatchPredictionJob is then initiated for the validation stage. Once the validation is completed, the user is notified to download a CSV file containing the final set of questions.
SUS Administration
Participants completed the System Usability Scale (SUS) following their interaction with Perspicacious. The SUS is a standardized 10-item usability questionnaire. We reproduce the SUS questions administered to participants below for transparency and reproducibility.
The SUS items are sourced from and scored according to the standard SUS procedure described by Brooke [9]. Responses to positively worded items were adjusted by subtracting 1, while negative worded item responses were subtracted from 5. We summed the resulting values to obtain the raw scores (0–40) and multiplied them by 2.5 to produce the standardized scores (0–100).
Participants answered the following questions on a Likert Response Scale from 1 to 5, where 1 represents Strongly Disagree and 5 represents Strongly Agree:
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well-integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
References
- [1]Adamson, D., Bhartiya, D., Gujral, B., Kedia, R., Singh, A., & Rosé, C. P. (2013). Automatically generating discussion questions. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial intelligence in education (pp. 81–90). Springer Berlin Heidelberg.
- [2]Aleven, V., McLaren, B. M., Sewall, J., & Koedinger, K. R. (2009). A new paradigm for intelligent tutoring systems: Example-tracing tutors. International Journal of Artificial Intelligence in Education, 19(2), 105–154.
- [3]Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of bloom’s taxonomy of educational objectives. Longman.
- [4]Barnes, T. (2005). The Q-matrix Method: Mining Student Response Data for Knowledge. Retrieved August 6, 2024, from https://aaai.org/papers/ws05-02-006-the-q-matrix-method-mining-student-response-data-for-knowledge/
- [5]Blobstein, A., Izmaylov, D., Yifal, T., Levy, M., & Segal, A. (2023). Angel: A new generation tool for learning material-based questions and answers. Proceedings of the NeurIPS ’23 Workshop on Generative AI for Education (GAIED), 1–13. https://gaied.org/neurips2023/files/9/9_paper.pdf
- [6]Bloom, B. S. (1968, May). Learning for Mastery. Instruction and Curriculum. Regional Education Laboratory for the Carolinas and Virginia. Topical Papers and Reprints. Number 1 (tech. rep.). Regional Education Laboratory for the Carolinas, Virginia, Mutual Plaza (Chapel Hill, and Duke Sts. Retrieved August 14, 2025, from https://eric.ed.gov/?id=ED053419
- [7]Branch, R. M. (2009). Instructional design: The addie approach. Springer. https://doi.org/10.1007/978-0-387-09506-6
- [8]Brod, C. (1984). Technostress : The human cost of the computer revolution.
- [9]Brooke, J. (1996). Sus: A quick and dirty usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & I. L. McClelland (Eds.), Usability evaluation in industry (pp. 189–194). Taylor & Francis.DOI
- [10]Bulathwela, S., Muse, H., & Yilmaz, E. (2023). Scalable educational question generation with pre-trained language models. Artificial Intelligence in Education: 24th International Conference, AIED 2023, Tokyo, Japan, July 3–7, 2023, Proceedings, 327–339. https://doi.org/10.1007/978-3-031-36272-9_27
- [11]Cao, M., Ling, J., Pavlik, J., Philip, I., & Carvalho, P. F. (2026). What should be spaced? variability constrains spacing effects in classroom learning. OSF Preprint. https://osf.io/preprints/psyarxiv/g97u4_v1
- [12]Cen, H., Koedinger, K., & Junker, B. (2006). Learning Factors Analysis – A General Method for Cognitive Model Evaluation and Improvement. In M. Ikeda, K. D. Ashley, & T.-W. Chan (Eds.), Intelligent Tutoring Systems (pp. 164–175). Springer. https://doi.org/10.1007/11774303_17
- [13]Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4), 253–278. https://doi.org/10.1007/BF01099821
- [14]Dahlbäck, N., Jönsson, A., & Åhrenberg, L. (1993). Wizard of oz studies — why and how [Special Issue: Intelligent User Interfaces]. Knowledge-Based Systems, 6(4), 258–266. https://doi.org/10.1016/0950-7051(93)90017-N
- [15]Desmarais, M. C., Beheshti, B., & Naceur, R. (2012). Item to Skills Mapping: Deriving a Conjunctive Q-matrix from Data. In S. A. Cerri, W. J. Clancey, G. Papadourakis, & K. Panourgia (Eds.), Intelligent Tutoring Systems (pp. 454–463). Springer Berlin Heidelberg.DOI
- [16]Desmarais, M. C., & Naceur, R. (2013). A Matrix Factorization Method for Mapping Items to Skills and for Enhancing Expert-Based Q-Matrices. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial Intelligence in Education (pp. 441–450). Springer. https://doi.org/10.1007/978-3-642-39112-5_45
- [18]Duan, N., Tang, D., Chen, P., & Zhou, M. (2017, September). Question generation for question answering. In M. Palmer, R. Hwa, & S. Riedel (Eds.), Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 866–874). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1090
- [19]Ericsson, K. A., & Simon, H. A. (1993, April). Protocol analysis: Verbal reports as data. The MIT Press. https://doi.org/10.7551/mitpress/5657.001.0001
- [20]Heilman, M., & Smith, N. A. (2010, June). Good question! statistical ranking for question generation. In R. Kaplan, J. Burstein, M. Harper, & G. Penn (Eds.), Human language technologies: The 2010 annual conference of the north American chapter of the association for computational linguistics (pp. 609–617). Association for Computational Linguistics. https://aclanthology.org/N10-1086/
- [21]Hwang, K., Wang, K., Alomair, M., Choa, F.-S., & Chen, L. K. (2024). Towards automated multiple choice question generation and evaluation: Aligning with bloom’s taxonomy. In A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, & I. I. Bittencourt (Eds.), Artificial intelligence in education (pp. 389–396). Springer Nature Switzerland.
- [22]Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C. C. T., Chen, W., Giorno, A. D., Eldan, R., Gopi, S., Gunasekar, S., Javaheripi, M., Kauffmann, P., Lee, Y. T., Li, Y., Nguyen, A., de Rosa, G., Saarikivi, O., Salim, A., … Zhang, Y. (2023, December). Phi-2: The surprising power of small language models.
- [23]Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst., 2(1), 26–41. https://doi.org/10.1145/357417.357420
- [24]Koedinger, K. R., Baker, R. S. J. d., Cunningham, K., Skogsholm, A., Leber, B., & Stamper, J. (2010). A data repository for the edm community: The pslc datashop. In C. Romero, S. Ventura, M. Pechenizkiy, & R. S. J. d. Baker (Eds.), Handbook of educational data mining. CRC Press.DOI
- [25]Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012). The knowledge-learning-instruction framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5), 757–798. https://doi.org/10.1111/j.1551-6709.2012.01245.x
- [26]Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020). A Systematic Review of Automatic Question Generation for Educational Purposes. International Journal of Artificial Intelligence in Education, 30(1), 121–204. https://doi.org/10.1007/s40593-019-00186-y
- [27]Lewis, J. R., & Sauro, J. (2018). Item benchmarks for the system usability scale. Journal of Usability Studies, 13(3), 158–167.DOI
- [29]Moon, H., Davis, R. L., Nezhaei, S. P., & Dillenbourg, P. (2025, July). Using large multimodal models to extract knowledge components for knowledge tracing from multimedia question information. In C. Mills, G. Alexandron, D. Taibi, G. L. Bosco, & L. Paquette (Eds.), Proceedings of the 18th international conference on educational data mining (pp. 342–353). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.15870171
- [30]Moore, S., Schmucker, R., Mitchell, T., & Stamper, J. (2024). Automated generation and tagging of knowledge components from multiple-choice questions. Proceedings of the Eleventh ACM Conference on Learning @ Scale, 122–133. https://doi.org/10.1145/3657604.3662030
- [31]Pardos, Z. A., & Dadu, A. (2018). dAFM: Fusing Psychometric and Connectionist Modeling for Q-matrix Refinement. Journal of Educational Data Mining, 10(2), 1–27. https://doi.org/10.5281/zenodo.3554689
- [32]Russell, S. J., & Norvig, P. (2020). Artificial intelligence : A modern approach (4th edition) [Section: XVII, 1115 Seiten : Illustrationen]. Pearson.
- [33]Scaria, N., Dharani Chenna, S., & Subramani, D. (2024). Automated educational question generation at different bloom’s skill levels using large language models: Strategies and evaluation. In A. M. Olney, I.-A. Chounta, Z. Liu, O. C. Santos, & I. I. Bittencourt (Eds.), Artificial intelligence in education (pp. 165–179). Springer Nature Switzerland.DOI
- [34]Shabana, K. M., & Lakshminarayanan, C. (2023). Unsupervised concept tagging of mathematical questions from student explanations. In N. Wang, G. Rebolledo-Mendez, N. Matsuda, O. C. Santos, & V. Dimitrova (Eds.), Artificial intelligence in education (pp. 627–638). Springer Nature Switzerland.DOI
- [35]Shi, Y., Schmucker, R., Chi, M., Barnes, T., & Price, T. (2023). KC-Finder: Automated Knowledge Component Discovery for Programming Problems (tech. rep.) (ERIC Number: ED630850). International Educational Data Mining Society. Retrieved October 19, 2024, from https://eric.ed.gov/?id=ED630850DOI
- [36]Stamper, J. C., & Koedinger, K. R. (2011). Human-Machine Student Model Discovery and Improvement Using DataShop. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Artificial Intelligence in Education (pp. 353–360). Springer. https://doi.org/10.1007/978-3-642-21869-9_46


