WorksComputer Science
FlashScript: Compiling Recurring Agent Workflows into Reusable Neuro-Symbolic Programs
Abstract
Many agent tasks recur with new data or small changes, yet agents regenerate their reasoning and actions on each run, producing different results. Distilling a prior run into a skill helps the agent avoid earlier mistakes, but it still re-runs the full process each time, so runs can vary and incur that cost. We compile a successful agent run into a reusable neuro-symbolic program (NSP) with a set of guidelines we call FlashScript. Actions and data transformations become code, instance-specific values become parameters, and decisions that need model judgment become LLM calls. We identify three layers where these calls belong and three techniques that replace most of them with code. The compilation does not remove nondeterminism; it localizes it, so the deterministic program and its neural calls can be reviewed and tested separately. Across 120 WorkBench analytics tasks with Claude Haiku 4.5, the NSPs were correct and made identical tool calls on every run, using about 70 times fewer input tokens and a third of the runtime of a skill-guided agent whose runs were inconsistent. In a composite version of the tasks, the skill-guided agent's accuracy fell to 23%, and its output inconsistency increased, while the NSPs remained correct and identical across all runs. The NSPs successfully generalized to all 108 unseen instances of the same templates. Our findings suggest that NSPs from agent trajectories can be considered cumulative software assets for end-users to test, maintain, and reuse in their recurring workflows.
Introduction
AI agents are increasingly performing recurring tasks on behalf of end users. For example, an analyst asks an agent to update a weekly report using data from several sources without writing the code to retrieve, combine, and format it. By translating natural-language requests into tool actions, agents advance the long-standing vision of end-user software engineering: letting people automate their own specialized needs [1]. Yet these agentic workflows do not behave like conventional software. An agent reconstructs the procedure on every run, so repeated executions of the same request can diverge in path and outcome [2, 3]. A successful execution holds valuable procedural knowledge: which tools to use, what went wrong, and which decisions produced a desirable result [4, 5]. Currently, this knowledge is buried within a session: a user may inspect, correct, and approve a run, but that effort never carries over to future executions. Users must therefore keep inspecting an already reviewed procedure.
Prior work has attempted to preserve successful agent behavior in several forms. Experience-based memory and workflow methods extract demonstrations, guidelines, or natural-language routines from past trajectories and retrieve them to guide later executions [4, 5, 6]. These methods help agents reuse experience, but the model must still interpret that guidance and decide how to execute it, so runs still vary. At the other extreme, programming-by-demonstration and robotic process automation record and replay concrete action sequences [7, 8, 9, 10]; they preserve execution more directly but generalize poorly when the inputs or environment change. More recent methods compile agent traces into reusable artifacts: procedural skills in SKILL-DISCO, programmatic skills via neuro-symbolic learning, and mostly deterministic workflows in TraceCompiler [11, 12, 13, 14, 15]. These show the promise of trajectory compilation but leave open where the model’s judgment must remain, and they do not address inconsistency. What is missing is a consistent yet generalizable process that a user can rerun on each recurrence of their workflow.
In this paper, we explore how a recurring agent task can be preserved as a reusable NSP (Figure 1-bottom), with guidelines we call FlashScript. A completed run is a natural starting point because users often reach it through several turns of feedback and manual fixes with an agent. Our method builds on that interaction to create an NSP. The compilation turns actions into code, instance-specific values into parameters, and represents each decision that requires model judgment as a typed call llm<T>, which returns a value of declared type , at a fixed point in the code. We identify three layers where these calls belong, the input, computation, and output layers, and three techniques that replace most of them with code: caching observed values behind a guard, codifying judgments as rules, and clubbing independent questions into a call. An NSP would consistently follow the same steps, invoking the model only when needed to answer scoped questions.

Figure 1. Compiling the agent’s prior runs (top) into a neuro-symbolic program (bottom) using FlashScript. The sheet selection process becomes a lookup with a neural fallback ①, the scam judgment a yes-or-no call for each ②, and the closing message a call that phrases changes computed by code ③. Colors mark the layer of each neural call.
We evaluated the resulting NSPs on 120 WorkBench analytics tasks [16] using Claude Haiku 4.5, against a vanilla agent and an agent with task-specific skills. The NSPs were correct and made identical tool calls on every run, using about 70 times fewer input tokens and about a third of the runtime of either agent. The skilled agent was correct in all three runs on 96% of tasks, but on composite tasks, combining up to ten of these requests as subtasks, its accuracy fell to 23%, while the NSPs stayed correct and identical in all runs. The NSPs also solved all 108 unseen instances of the templates (100% accuracy), against 96% for the agent with skill.
NSPs can turn recurring agent work into specific software that end users can develop, bringing software engineering practices to programs including LLM calls [1], [17]. They do not eliminate nondeterminism, but they localize it. Each neural call has an explicit input format, output type, and defined use in the NSP. This structure facilitates the separate testing and maintenance of the deterministic code and neural components.
Motivating Example
Imagine an analyst who maintains a workbook of available rental listings for his clients each week. Prices may change, some listings may be rented, and new rentals may become available. Scam listings are also a big concern in large cities; therefore, he wants to help his clients avoid them. So every Monday morning, he asks an agent to “update LA rentals and flag scams” (Figure 1-top). The workbook has a separate sheet for each city. The agent first infers that “LA” in the query refers to Los Angeles and selects the corresponding sheet. It updates the workbook by fetching and merging new listings, then flags listings that request a money transfer before the first visit as potential scams. Finally, it summarizes the changes.
What if he wants to repeat this next week or apply the same process for New York City? He has already spent time clarifying the request and manually editing the workbook to get the output he wants for Los Angeles. Ideally, the next run should be: (C1) consistent: the same output format, styling, and process as he reviewed and reached in the previous run; (C2) generalizable: it could handle variations in new data or in another city and reach the correct solution; (C3) cheap: reasonable costs for repeated runs, since the workflow has been refined and his corrections incorporated once, so the system should have already figured the solution out.
Our preliminary results show that a skill-guided agent does not fully meet these needs. In our sandbox, it generalizes to the New York City task, but across three runs it varies in how it names added columns and marks removed listings, taking an average of 38 seconds and using 42k tokens per run. By contrast, the NSP in Figure 1-bottom flagged the same listings and produced identical sheets in 6.7 seconds using 6.7k tokens.
FlashScript
FlashScript is our authoring guidance for writing these NSPs from prior agent runs. Programs that combine code and invocations to learned models have a long history [18], and LLMs have made them more common. Models can serve as semantic predicates in synthesized programs [19], as functions in generated programs [20, 21, 22], and as operators in data pipelines [23]. We derive such an NSP from a given user’s session with an AI agent. We turn a completed agent trajectory into a script with an additional operation, llm<T>, which sends a prompt to a model and returns a value of a declared type, such as a string, Boolean, or option. The run’s actions become code, while its reasoning becomes model calls to be made when a future run requires judgment.
Fixing the Actions (C1). Each of the neural calls in NSPs is bounded. The NSP selects its inputs, checks that the response has the expected type [24, 25, 26], and makes the call at a fixed point in the code. The NSP’s control flow determines when the model is queried, for what, and how its response is used. As a result, variation across runs is confined to those calls.
Locating the Neural Part (C2). The calls fall into three layers, based on how information moves through the workflow. Input layer calls turn unstructured text into typed values, such as identifying which sheet “L.A.” refers to. Computation layer calls make semantic decisions that are difficult to express as rules, such as whether a listing is a scam. Output layer calls render computed results in natural language for the task owner.
A program with no model calls is a fixed script: it is consistent, but limited only to the inputs it was generated from. And a program that relies solely on model calls is equivalent to the reason-action agent loop. NSPs sit between these two ends. They keep actions as code and use the model calls only where expected. Their generality depends on how many decisions they leave to model calls: delegating more handles more variation but reduces consistency and efficiency.
Efficient Decision Making (C3). Unlike an agent that makes an LLM call at every step, each reading the growing history, an NSP pays only for its limited expected number of model calls. Its expected cost depends on how often each call is made and how large it is. We propose three techniques that can make calls less frequent or smaller, further reducing costs. Caching keeps a value observed in the run and checks a condition before reusing it in a future run. The model is called only if that condition fails. For example, the sheet lookup in Figure 1 checks the SHEETS dictionary and falls back to an LLM call choosing from existing sheets. Codifying replaces a repeatable judgment with a heuristic rule. Writing the rule once costs less than asking the model every run and gives the same answer each time. Unlike program-aided reasoning [27, 28], the model does not make the decision by executing generated code. In Figure 1, a regex such as (send|pay).*(Venmo|Zelle).*before.* could flag the requests for suspicious payment before a visit, indicating potential scams. Clubbing puts independent questions in one call or runs them in parallel [29, 30]. These typed questions can also go to smaller classification models [31, 32], since the generative capacity of an LLM is not required.
Preliminary Results
We implemented NSPs as Python scripts with neural call using the same model as the agent. We compared the NSPs with a vanilla agent and a skill-guided agent on the WorkBench benchmark [16] and evaluate whether they meet consistency (C1), generalization (C2), and cost-effectiveness (C3) criteria.
Setup. WorkBench’s 690 tasks came from 69 request templates with ten instances each. We took its 12 Analytics templates (120 tasks), in which the agent must read data, compute a value, compare it to a threshold, and decide whether and what to plot. For example, the template “If {metric} is more than {threshold} {higher or lower} than {date}, make a line plot of it since then” has the instance “If engaged users is more than 10% lower than November 24, make a line plot of it since then.” Other task categories are mostly single-record edits, leaving little room for runs to differ. We first run the vanilla agent three times on every task. Then for each template, WorkBench’s grader selected the best run and a coding agent (Claude Code Opus 5.5) wrote the skill from that run. The coding agent also wrote each NSP from FlashScript instructions (§ 3) that provide the prompt template, the tool descriptions, and the same agent trajectory used to write the skill (the R + T condition in Table II). We ran the skill-guided agent and the NSP three times on every task. All runs used Claude Haiku 4.5, chosen for its favorable performance–cost trade-off, at temperature 1, a common setting that reflects typical LLM generation behavior. To examine how performance changes with task size, we combined requests from different templates into single prompts (“Please do all of the following: 1. . . . 2. . . .”), using only instances not used to create the skill or NSP. We created ten composition tasks of varying lengths. Their subtasks are independent, allowing the benchmark grader to evaluate each one separately. Because errors can propagate when subtasks depend on one another, our setup likely understates the drop in accuracy on realistic tasks.
| R | T | R+T | |
| Accuracy, same instance | 1.00 | 1.00 | 1.00 |
| Accuracy, unseen instances | 0.89 | 1.00 | 1.00 |
| Accuracy, composite tasks | 0.53 | 1.00 | 1.00 |
| Same outcome, composite tasks | 0.99 | 1.00 | 1.00 |
| Runs that raise an error | 8.6% | 0% | 0% |
| Neural calls per run | 1.3 | 1.8 | 1.5 |
Table II. PERFORMANCE METRICS FOR NSPS WRITTEN FROM THE REFERENCE RUN (R), THE REQUEST TEMPLATE (T), AND BOTH (R+T). COMPOSITE VALUES ARE AVERAGED OVER THE FOUR LENGTHS.
Consistency (C1). The vanilla agent’s runs were inconsistent: only 82% of its repeated runs ended in the same state as the first run (Table I). The skill reduced most of this inconsistency but did not remove it completely (99%). The
| Agent | Agent + skill | NSP | |
| Accuracy | |||
| Correct in all three runs | |||
| Same outcome as first run | |||
| Same tool calls as first run | |||
| Input tokens | |||
| Output tokens | |||
| Runtime (s) |
Table I. Performance metrics on repeated runs. Values are mean ± std. The “correct in all three runs” row is the fraction of the tasks correct in every repeat.
NSP made the same tool calls in every run, and across runs only the wording of its reply varies. On composite tasks, agents become less consistent (Figure 2b–Figure 2f): with ten subtasks, the skilled agent reproduces the outcome of its first run in only 10% of cases while working on the exact same task, receiving every subtask’s skill and appearing to mix them up, adding plots no subtask requested and omitting some that were. The NSP, by contrast, produces the exact same result in every run.
Figure 2. Tasks made of 1, 3, 5, and 10 independent subtasks; ten tasks per length, three runs each. Bands show one standard deviation across the repeats.
Generalizability (C2). On repeated tasks, the vanilla agent was often correct (accuracy 0.89), and the skill reached 0.97. They were correct in all three runs on 79% and 96% of tasks, respectively (Table I). Meanwhile, the NSP was correct on every single run (accuracy 1.0). Among the tasks, eight templates needed a decision about the data, such as whether a metric crossed a threshold or which date in the data “Saturday” refers to. These templates account for 34 of the vanilla agent’s 39 errors and all 11 of the skilled agent’s. The skilled agent fails in two ways. When the skill was correct, the agent still made mistakes, for example by wrongly summing daily values in its reasoning (2+3+6+3+8+5+⋯ = 148 visits). In other cases, the skill inherits a heuristic that was not generalizable beyond the reference run. One template asks to compare a metric’s percent growth, but the reference run compared their sums instead; both agreed there, so the grader accepted it, and the skill kept the shortcut, dropping accuracy to 0.67 against 0.87 for the vanilla agent on that task.
On composite tasks (ten per length, three runs each), the NSP was correct in every run, compared with about 23% for the skilled agent (Figure 2-a). Across 108 unseen instances, the NSP solved every task and repeated the first instance’s tool sequence, against 96% and 98% for the skilled agent and 90% and 78% for the vanilla agent. The NSP also made no plot errors on composite tasks, while the skilled agent added 1.6 extra plots and omitted 0.6 plots per run compared with the anchor tasks, the same composites assembled from the first instance of each template.
Cost (C3). The NSP used about 70 times fewer input tokens and about a third of the runtime of either agent (Table I). Across all NSPs there are 69 neural calls, 49 guarded input calls and 20 reply calls, averaging 1.5 per run. On composite tasks its runtime grows from 1.4 to 6.0 seconds, against about 26 to 30 for the agents. All three reduction techniques from Section III helped: caching served 37 of the 49 guarded input calls from a stored value instead of the model, codifying turned every computation-layer decision into a rule, and clubbing combined independent questions into one call, cutting a ten-subtask task from 5.8 model questions to 2.0.
We also studied how an NSP’s quality depends on the generation instructions and on the trajectory. Our first instructions asked the agent only to write an NSP from a template or a trajectory; those NSPs achieved 0.96 and 0.88 accuracy, respectively, but failed on reworded requests and edge cases. Accuracy reached 1.00 once we asked the agent to plan first, encode decisions as rules, and place calls according to their layer (available in supplementary materials2). With these instructions, we wrote NSPs from three inputs: the template only (T), the run alone (R), and the template with the skill’s reference run (R+T) (Table II). Only using the run makes the NSP too specific. It is correct on the instance it is generated from, but fails on unseen instances. Using the template alone already yields good generalization. It identifies the variables and plans for the variants. Adding the run to the template strengthens the NSP’s caches by setting more inclusive guards, reducing the number of calls from 1.8 to 1.5. The run also exposes the agent’s decisions to review. For example, the plan caught the reference run’s shortcut for summing and corrected it. In short, correctness comes from the template and the planning instructions, while the run contributes efficiency through tighter caches and auditability by exposing the reference run’s decisions for review.
Discussion
NSPs also changes what users retain after a successful run: reusable software that can be parameterized, combined with other NSPs, and maintained over time.
User Interaction with NSPs. Instead of having an agent solve the same task from scratch each time, the user keeps an
NSP they reuse. An NSP can expose user-facing parameters for the task. For a scam check, the user could choose a heuristic from a menu or describe what they consider a scam as a text input. The NSP includes that preference in its LLM scam-check call, grounding the model’s judgment in the user’s criteria. The user can change the preference and see its effect on the output within seconds.
Software Lifecycle and Testing. Because the compiled control flow is fixed and separates deterministic code from model calls, making LLM-based systems easier to analyze and validate [33], NSPs can follow the same lifecycle as other software. Developers can inspect the control flow before deployment, see which steps run and where the model is called, and remove the “black-box” risk of an agent. Testing then splits by component: the symbolic shell takes the usual unit, integration, and regression tests; the neural calls, whose outputs vary across runs and models, need contract, repeated-sampling, and metamorphic tests; and the seam must check the shell handles any value of a call’s return type . As a workflow settles, refactoring, such as rule codification, replaces neural calls with deterministic code.
Benchmarking Nuanced Decisions. Agent benchmarks favor tasks with clear, verifiable outcomes, so they rarely capture the nuanced decisions involved in recurring work. Real tasks may require models to infer constraints, interpret vague terms such as “suspicious,” weigh personal preferences, or frame open-ended problems [34, 35, 36, 37, 38, 39, 40, 41, 42]. A dataset of such workflows would clarify the boundary of when an NSP neural calls.
Future Plans
We will build a benchmark of recurring workflows that test the nuanced decisions an agent must make before acting. Most benchmarks focus on whether the agent uses the right tools, even if it misunderstood the task. Our benchmark will test whether it can infer unstated constraints, define vague terms such as “suspicious” [42], weigh personal preferences, and decide what to investigate when there is no clear path or success measure. Each workflow will include successive instances, with real users’ judgments as ground truth. We will test how well caches and guards handle changing data, and measure each model call’s accuracy and consistency in a NSP separately from task execution.
Additionally, we will study interfaces that let end-user developers inspect and customize NSPs without reading code. Users could set text or menu parameters, edit steps or model guidance, and see the effect on outputs instantly. Further user studies will explore which parameters to expose, how users want to set them for an NSP, and what they expect NSPs to generalize across recurring runs.
Finally, we will make NSP coverage visible for the user. For each new request, model calls will identify its requirements and check which ones the NSP can handle. Covered requirements will run as code, and a fallback agentic step will handle the rest and can update the NSP. Over time, the NSP can cover more scenarios and data structures, so more of each request can run as reviewed code.
References
References
- [1]A. J. Ko, R. Abraham, L. Beckwith, A. Blackwell, M. Burnett, M. Erwig, C. Scaffidi, J. A. Lawrence, H. Lieberman, B. Myers, M. B. Rosson, G. Rothermel, M. Shaw, and S. Wiedenbeck, “The state of the art in end-user software engineering,” ACM Computing Surveys, vol. 43, no. 3, pp. 1–44, Apr. 2011.
- [2]S. Yan, X. Wen, S. Ruan, Y. Zhang, J. Mi, Y. Sun, H. Qu, and R. Sheng, “InconLens: Interactive visual diagnosis of behavioral inconsistencies in LLM-based agentic systems,” arXiv preprint arXiv:2603.28106, 2026.arxiv.org/abs/2603.28106
- [3]A. Mehta, “When agents disagree with themselves: Behavioral consistency as an uncertainty signal for LLM agents,” arXiv preprint arXiv:2602.11619, 2026.arxiv.org/abs/2602.11619
- [4]A. Zhao, D. Huang, Q. Xu, M. Lin, Y.-J. Liu, and G. Huang, “ExpeL: LLM agents are experiential learners,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024.arxiv.org/abs/2308.10144
- [5]Z. Z. Wang, J. Mao, D. Fried, and G. Neubig, “Agent workflow memory,” in Proceedings of the 42nd International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, vol. 267. PMLR, 2025, pp. 63 897–63 911. [Online]. Available: https://proceedings.mlr.press/v267/wang25b.htmlarxiv.org/abs/2409.07429
- [6]Y. Fu, D.-K. Kim, J. Kim, S. Sohn, K. Logeswaran, K. Bae, and H. Lee, “Autoguide: Automated generation and selection of context-aware guidelines for large language model agents,” Advances in Neural Information Processing Systems, vol. 37, pp. 119 919–119 948, 2024.arxiv.org/abs/2403.08978
- [7]G. Leshed, E. M. Haber, T. Matthews, and T. Lau, “CoScripter: Automating & sharing how-to knowledge in the enterprise,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), 2008.
- [8]T. J.-J. Li, A. Azaria, and B. A. Myers, “SUGILITE: Creating multi-modal smartphone automation by demonstration,” in Proceedings of the CHI Conference on Human Factors in Computing Systems, 2017.
- [9]S. E. Chasins, M. Mueller, and R. Bodik, “Roussillon: Scraping distributed hierarchical web data,” in Proceedings of the ACM Symposium on User Interface Software and Technology (UIST), 2018.DOI
- [10]W. M. P. van der Aalst, M. Bichler, and A. Heinzl, “Robotic process automation,” Business & Information Systems Engineering, vol. 60, no. 4, 2018.
- [11]J.-J. Shao, H. Yin, Y. Lyu, X. Yu, L.-Z. Guo, I. Tsang, J. Kwok, and Y.-F. Li, “Lifting traces to logic: Programmatic skill induction with neuro-symbolic learning for long-horizon agentic tasks,” arXiv preprint arXiv:2605.01293, 2026.arxiv.org/abs/2605.01293
- [12]Z. Guo, D. Qi, H. Gu, P. Cheng, and Y. Xiong, “Skill-disco: Distilling and compiling agent traces into reusable procedural skills,” arXiv preprint arXiv:2606.26669, 2026.arxiv.org/abs/2606.26669
- [13]S. E. Yadouni and G. Li, “Tracecompiler: Skill-guided mining and compilation of llm agent traces into mostly deterministic workflows,” arXiv preprint arXiv:2608.02680, 2026.
- [14]B. Li, “Compucus: Computer-using agents that get faster on repeated tasks,” arXiv preprint arXiv:2606.17929, 2026.arxiv.org/abs/2606.17929
- [15]H. Hasanbeig, N. Y. Jeppu, A. Abate, T. Melham, and D. Kroening, “Symbolic task inference in deep reinforcement learning,” Journal of Artificial Intelligence Research, vol. 80, pp. 1099–1137, 2024.
- [16]O. Styles, S. Miller, P. Cerda-Mardini, T. Guha, V. Sanchez, and B. Vidgen, “WorkBench: A benchmark dataset for agents in a realistic workplace setting,” arXiv preprint arXiv:2405.00823, 2024.arxiv.org/abs/2405.00823
- [17]B. Mahmoudi, Z. Chenail-Larcher, N. Moha, Q. Stiévenart, and F. Avel-laneda, “Specification and detection of LLM code smells,” in Proceedings of the 48th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), 2026, pp. 181–185.
- [18]S. Chaudhuri, K. Ellis, O. Polozov, R. Singh, A. Solar-Lezama, and Y. Yue, “Neurosymbolic programming,” Foundations and Trends in Programming Languages, vol. 7, no. 3, 2021.
- [19]G. Verbruggen, V. Le, and S. Gulwani, “Semantic programming by example with pre-trained models,” Proceedings of the ACM on Programming Languages, vol. 5, no. OOPSLA, pp. 1–25, 2021.
- [20]T. Gupta and A. Kembhavi, “Visual programming: Compositional visual reasoning without training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.arxiv.org/abs/2211.11559
- [21]D. Surís, S. Menon, and C. Vondrick, “ViperGPT: Visual inference via python execution for reasoning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.arxiv.org/abs/2303.08128
- [22]Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu, “Binding language models in symbolic languages,” in International Conference on Learning Representations, 2023.arxiv.org/abs/2210.02875
- [23]L. Patel, S. Jha, M. Pan, H. Gupta, P. Asawa, C. Guestrin, and M. Zaharia, “Semantic operators: A declarative model for rich, AI-based data processing,” arXiv preprint arXiv:2407.11418, 2024.arxiv.org/abs/2407.11418
- [24]L. Beurer-Kellner, M. Fischer, and M. Vechev, “Prompting is programming: A query language for large language models,” Proceedings of the ACM on Programming Languages, vol. 7, no. PLDI, 2023.
- [25]O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Vardhamanan, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, M. Zaharia, and C. Potts, “DSPy: Compiling declarative language model calls into state-of-the-art pipelines,” 2024.
- [26]B. T. Willard and R. Louf, “Efficient guided generation for large language models,” arXiv preprint arXiv:2307.09702, 2023.arxiv.org/abs/2307.09702
- [27]L. Gao, M. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig, “PAL: Program-aided language models,” in International Conference on Machine Learning, 2023, pp. 10 764–10 799.arxiv.org/abs/2211.10435
- [28]W. Chen, X. Ma, X. Wang, and W. W. Cohen, “Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks,” Transactions on Machine Learning Research, 2023.arxiv.org/abs/2211.12588
- [29]Z. Cheng, J. Kasai, and T. Yu, “Batch prompting: Efficient inference with large language model APIs,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2023.arxiv.org/abs/2301.08721
- [30]S. Kim, S. Moon, R. Tabrizi, N. Lee, M. W. Mahoney, K. Keutzer, and A. Gholami, “An LLM compiler for parallel function calling,” in International Conference on Machine Learning, 2024.arxiv.org/abs/2312.04511
- [31]L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to use large language models while reducing cost and improving performance,” Transactions on Machine Learning Research, 2024.arxiv.org/abs/2305.05176
- [32]Y. Li, Y. Miao, R. Krishnan, and R. Padman, “JEV-as-a-judge: Accept when confident, escalate when unsure,” arXiv preprint arXiv:2609.26550, 2026.arxiv.org/abs/2609.26550
- [33]V. Scotti, J. Keim, T. Hey, A. Metzger, A. Koziolek, and R. Mirandola, “A road map for tamed interactions with large language models,” ACM Transactions on Software Engineering and Methodology, vol. 35, no. 9, pp. 1–40, Aug. 2026. [Online]. Available: https://doi.org/10.1145/3832776
- [34]H. A. Simon, Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan, 1947.
- [35]——, “The structure of ill-structured problems,” Artificial Intelligence, vol. 4, no. 3–4, pp. 181–201, 1973.
- [36]R. L. Keeney and H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Trade-offs. John Wiley & Sons, 1976.
- [37]H. A. Simon, “Rational choice and the structure of the environment,” Psychological Review, vol. 63, no. 2, pp. 129–138, 1956.pubmed.ncbi.nlm.nih.gov/13310708
- [38]E. Hüllermeier and R. Slowinski, “Preference learning and multiple criteria decision aiding: Differences, commonalities, and synergies—part I,” 4OR, vol. 22, no. 2, pp. 179–209, 2024.DOI
- [39]S. Bankes, “Exploratory modeling for policy analysis,” Operations Research, vol. 41, no. 3, pp. 435–449, 1993.DOI
- [40]R. J. Lempert, S. W. Popper, and S. C. Bankes, Shaping the Next One Hundred Years: New Methods for Quantitative, Long-Term Policy Analysis. RAND Corporation, 2003.DOI
- [41]P. Checkland, Systems Thinking, Systems Practice. John Wiley & Sons, 1981.
- [42]S. Baradaran, A. Nazari, and M. Raghothaman, “LLM-integrated declarative program analysis,” in Proceedings of the 15th ACM SIGPLAN International Workshop on the State of the Art in Program Analysis, 2026, pp. 41–47.DOI
