WorksComputer Science
UAV-SwarmAgent: Scalable Multi-UAV Reconnaissance and Task Allocation via Code Generation and Vision-Language Reasoning
Abstract
Supervising a multi-UAV reconnaissance mission usually means plotting waypoints and watching every drone. We present UAV SwarmAgent, a hierarchical multi-agent system in which an operator states the mission in one natural-language sentence and a ground-station agent writes and executes code that turns it into per-UAV routes. The agent grounds inspection targets in an aerial map with a vision-language model, splits them among N UAVs by balancing route length, and dispatches worker agents that verify incidents with a compact onboard decision model. In simulation over 4 incident types and 3 foundation models, the swarm located the incident in 244 of 255 missions (95.7%). A paired scaling study with one to 5 UAVs shows that flight time follows an Amdahl-type law. The second UAV delivers half of the attainable saving and the speedup is bounded at 1.52x. We discuss what this predictability, and the false alarms we observed, imply for an operator who supervises the swarm, and outline the user study needed to test it.
CCS Concepts • Computer systems organization → Robotics; • Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Multi-agent systems.
Introduction
Unmanned aerial vehicles (UAVs) are routinely used for time-critical reconnaissance such as wildfire assessment, search and rescue, and environmental monitoring [11, 16]. Adding drones shortens the mission, but it also multiplies the work of the person in charge: each UAV needs waypoints, and each video stream needs attention. Research on supervisory control has shown that the number of robots one operator can handle is limited by how long a robot can be neglected relative to how long it takes to service it [3, 12], and that workload and situation awareness degrade as teams grow [2, 6].
Large language and vision-language models offer a different division of labour. They can read a satellite image, follow an underspecified request, and write programs that call tools [8, 17, 19]. Recent aerial systems use them to navigate from language instructions [7, 9] or to generate a single-drone mission from a map [14, 15]. What is missing for human-swarm interaction is the step from one instruction to a multi-drone mission whose duration the operator can anticipate.
This late-breaking report presents UAV-SwarmAgent, a system in which the operator writes one sentence (“I need to check the houses; one of them might be on fire”) and receives the locations of detected incidents with the supporting camera frames (Figure 1). Our contributions are:
Figure 1. UAV-SwarmAgent turns one operator sentence into a multi-UAV reconnaissance mission: the ground-station agent grounds inspection targets on an archival map, allocates them to the UAVs, and each UAV agent verifies the incident onboard and reports its location with the supporting frames.
a two-tier agent architecture in which a ground-station agent plans by writing code and delegates incident verification to one agent per UAV;
a simulation study of 255 missions over 4 incident types and 3 foundation models, with mission-level success and frame-level false alarms reported separately;
a paired scaling study with one to 5 UAVs that yields a simple law for mission time, which an operator can use to decide how many drones to send.
We did not involve human participants; the implications for operators in Section 6 are hypotheses for the study we outline.
Related Work
Human-swarm interaction. Beyond the supervisory-control limits cited above, language has been proposed as a way to command groups at the level of intent, for example for formation flight and choreography [5, 10].
Vision-language models for UAVs. Benchmarks for aerial vision-and-language navigation evaluate a single drone that follows instructions in a simulator [7, 9, 18, 21]. Mission-generation systems ground language goals on satellite imagery and plan a route for one UAV [13, 14]. UAV-CodeAgents [15] plans such missions with ReAct agents that ground targets by pixel pointing; UAV-SwarmAgent builds on that design and adds route allocation across drones and an onboard verifier. Training-free multi-agent pipelines improve aerial image understanding by routing sub-tasks to specialised tools [20].
Agents that act by writing code. ReAct interleaves reasoning with tool calls [19]; expressing actions as executable code lets one step compose several tools [8, 17]. UAV-SwarmAgent applies this paradigm to swarm reconnaissance and, unlike the work above, measures how mission time changes with the number of drones.
System
Agents
UAV-SwarmAgent is built on the open-source smolagents library and has two tiers (Figure 2). The Airspace Management Agent (AMA) runs at the ground station and works in a ReAct loop whose actions are short Python programs executed in a sandbox: it infers which structures or areas are worth inspecting, grounds them as waypoints on the archival map, shows them to the operator, and dispatches the swarm. One UAV agent per drone runs the onboard verifier on the camera frames collected along its route.
Figure 2. The two agent tiers (left), their tools (right), and the nine steps of a mission, with outputs of the structure-fire example of Figure 1.
We keep geometry out of the language model’s hands. Route allocation, flight simulation, and delegation are a single tool call, dispatch_swarm, rather than code the model writes itself. In a prototype where the model wrote this code, one model silently rescaled the waypoint coordinates before dispatch, which shifted every route. Fixed tools make the behaviour of the swarm independent of which model plans the mission.
Grounding inspection targets
The AMA sends the map, letterboxed to a 1000 1000 grid, to a vision-language model and asks for waypoints. The prompt distinguishes two kinds of targets. Objects such as buildings or vehicles receive one point at their centre. Areas such as open water or forest receive an even spread of points, because one point in the middle of a lake does not make a drone survey it (Figure 4). Returned points are mapped back to map pixels; at most 15 are kept. Answers in which several object points lie on one image axis at equal spacing are discarded as fabricated.

Figure 4. One example mission per scenario (). Buildings and roads are grounded as objects, one waypoint each; forest and water are grounded as areas and covered with an even spread of waypoints. Insets show the reported frame with the highest score; dashed lines are the return legs to the base station. The collision and oil-spill examples use the refined verifier questions.
Allocating routes
Waypoints are assigned to UAVs that start and land at one base station. Because the mission ends when the last drone returns, we balance the longest route rather than the total distance: waypoints are taken farthest-first and each goes to the UAV whose closed route stays shortest after receiving it; each UAV then visits its waypoints in nearest-neighbour order. Mission flight time is the makespan
where is the length of route in pixels, m/px is the ground sampling distance of the maps, and m/s is the cruise speed.
Verifying incidents onboard
Each UAV carries a nadir camera with a 128 128 px footprint (about 170 m on the ground). A frame is scored every 16 m along the route by Valen-Preview-0923, a 2B-parameter vision-language decision model that returns the probability of “yes” for a scenario-specific question (for example, whether flames or smoke are visible) in a single forward pass, without generating text. At this size the verifier can run locally on the drone, on an embedded NVIDIA Jetson computer; in our experiments it ran on a laptop GPU. A frame is reported when the probability exceeds a threshold set per scenario (0.55–0.74). Frames of all UAVs are batched on one GPU. Reported frames return to the AMA with their coordinates.
Evaluation Setup
Scenarios and maps. We use satellite maps of rural settlements derived from the UAV-VLA benchmark [14], each covering about km, and five coastal scenes collected from public web map imagery. Every mission pairs an archival reference map, on which targets are grounded, with a target map of the same area that contains the incident and supplies the camera frames (Figure 3); grounding never uses the target map. Incidents are synthetic and composited into the target map: flames on one building (structure fire, 30 maps), a burn scar with a flame front and a smoke plume inside a forest (wildfire, 5 maps), vehicles stopped at a junction (vehicle collision, 5 maps), and a dark slick on coastal water (oil spill, 5 maps). The operator request is one fixed sentence per scenario and never names a location.

Figure 3. Benchmark maps: the archival reference map used for grounding (top) and the target map of the same area with a synthetic incident, over which the drones fly (bottom; incident boxed).
Models. The AMA and the UAV agents use one of 3 models through an OpenAI-compatible gateway: gemini-3.8-flash, gpt-6.1-sol, or qwen3.8-max. The onboard verifier is the same in all runs and executes locally on a laptop GPU (NVIDIA RTX 4050, 6 GB).
Protocol. The scaling study runs the structure-fire scenario on all 30 maps with UAVs using gemini-3.8-flash (150 missions). The design is paired: every swarm size sees the same maps and the same base station, placed at a random point of the map border per map. The cross-model study fixes and runs every scenario with every model (135 missions, of which the 30 Gemini fire missions are shared with the scaling study). In total we report 255 missions. For the collision and oil-spill scenarios the verifier questions were later refined and the Gemini missions repeated; Table 1 gives the precision of that repeat and the mission counts of the first runs.
| Scenario | Maps | Gemini | GPT | Qwen | Prec. |
| Structure fire | 30 | 28/30 | 30/30 | 28/30 | 0.68 |
| Wildfire | 5 | 5/5 | 5/5 | 5/5 | 0.57 |
| Vehicle collision | 5 | 3/5 | 4/5 | 4/5 | 0.74 |
| Oil spill | 5 | 5/5 | 5/5 | 5/5 | 0.95 |
| All scenarios | 41/45 | 44/45 | 42/45 | ||
| Agent time, median (s) | 152 | 180 | 135 |
Table 1. Missions in which the swarm located the incident, by scenario and model ( UAVs). Prec.: median share of reported frames that show the incident.
Measures. A mission is successful when at least one reported frame shows the incident, that is, the incident centre lies inside the frame’s px camera footprint. Wildfires and oil spills are larger than the footprint, so for them the frame only has to overlap the area changed by the incident. Precision is the share of a mission’s reported frames that satisfy the same criterion. Flight time is the simulated makespan of Eq. (1). Agent time is the wall-clock time of planning, simulation, and onboard inference. Total time is their sum, a conservative figure because inference would overlap with flight on a real drone.
Analysis. Speedups are geometric means of per-map ratios . Confidence intervals are 95% percentile intervals from 10,000 bootstrap resamples of maps [4]. The effect of swarm size is tested with Friedman’s test and Page’s test for ordered alternatives, with maps as blocks. Scaling laws are fitted by least squares on log time with a separate level per map.
Use of generative AI. An AI coding assistant (Claude) helped write parts of the software, the analysis scripts, and the plotting code; the authors checked all results.
Results
Mission success
The swarm located the incident in 244 of 255 missions (95.7%). Table 1 breaks down the cross-model study: all 3 models completed the planning workflow in every mission, and 127 of 135 missions were successful. The scaling study adds 145 successes in 150 missions. The models differed more in latency than in outcome (last row of Table 1). With five maps per non-fire scenario, differences between models in those rows are within what chance alone produces.
Scaling with swarm size
Flight time decreased monotonically from 505 s with one UAV to 363 s with 5 (28.1%; Friedman and Page tests, ). As reference, photographing each map completely in parallel strips one footprint wide would take 2432 s and 719 s, 4.8 and 2.0 times longer; targeting buys this time at the risk of missing an incident that no waypoint is near. Figure 5 shows that the gain flattens quickly. An Amdahl-type law [1], in which a share of the single-UAV flight time is divided among the drones and the rest is not,
Figure 5. Scaling study (structure fire, 30 maps per swarm size): mean flight, agent, and total mission time with 95% bootstrap confidence intervals; lines are the fitted laws.
fits with (95% CI ). The fixed part has a physical reading: every drone must fly from the base station to the area of interest and back, however the waypoints are divided. The law bounds the flight-time speedup at (CI ); the second UAV already delivers 50% of the attainable saving and 5 UAVs deliver 80%. A power law fits equally well over this range (AIC difference 0.8); we prefer (2) because its parameters are interpretable.
Agent time grew slowly with swarm size, by 6.7 s per UAV (CI [0.8, 12.0]) from an intercept of 140 s. Total mission time therefore fell from 657 s to 533 s (18.9%), and almost all of that gain came from the second UAV: beyond two UAVs, consecutive swarm sizes do not differ significantly in total time (Wilcoxon signed-rank tests, Holm-corrected).
False alarms and missed incidents
Mission-level success hides what the operator would actually receive. A mission produced a median of 37 reported frames, and not all of them show the incident: median precision ranged from 0.57 for the wildfire scenario to 0.95 for the oil spill scenario (Table 1). Of the 11 failed missions, 5 failed at the planning stage: the vision-language model placed no waypoint near the incident, so no route passed over it. In the other 6 a drone flew over the incident but the verifier did not report it. The missions of the first kind used at most 12 of the 15 waypoints allowed, so asking the model for more waypoints is a direct remedy.
Discussion
From plotting to checking. In UAV-SwarmAgent the operator’s input does not grow with the number of drones: one sentence produced routes for up to 5 UAVs. What remains for the human is verification. The waypoint overlay shown before dispatch and the reported frames returned afterwards are the two places where an operator can catch an error. Whether operators catch such errors, and at what workload, is the first question for a user study.
Swarm size as a predictable choice. (2) shows that in base-station reconnaissance more drones do not help proportionally: the commute to the area does not parallelise, so a second drone is well spent and a fifth buys little. Because the law has two parameters with physical meaning, it could show to the operator as a forecast (“two UAVs: about 426 s; three: about 393 s”) before launch. We expect such a forecast to support calibrated reliance better than an unexplained recommendation, which remains to be tested.
False alarms are an interaction problem. A system that locates the incident in most missions can still hand the operator dozens of frames to inspect. Ranking reports by score, clustering neighbouring frames into one report, and letting the operator adjust are interface decisions that matter as much as detector accuracy.
Limitations. The study is a simulation with synthetic incidents and a kinematic flight model without wind, batteries, or obstacles. Verifier questions and thresholds were chosen on the benchmark maps, with no held-out set. Each configuration was run once, although language-model outputs vary between runs, and operator requests were fixed sentences. No human participants were involved, so every statement about operators is a hypothesis.
Conclusion and Next Steps
UAV-SwarmAgent turns one natural-language instruction into a multi-UAV reconnaissance mission. In 255 simulated missions the swarm located the incident 95.7% of the time with 3 different foundation models, and flight time followed a simple law that makes the benefit of each additional drone predictable. Next, we will compare instruction-based commanding with manual waypoint planning in a user study that measures planning time, workload, and error detection, and we will move from simulation to physical multirotors.
References
References
- [1]Gene M. Amdahl. 1967. Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities. In Proceedings of the AFIPS Spring Joint Computer Conference, 483–485.
- [2]Jessie Y. C. Chen and Michael J. Barnes. 2014. Human–Agent Teaming for Multirobot Control: A Review of Human Factors Issues. IEEE Transactions on Human-Machine Systems 44, 1 (2014), 13–29.DOI
- [3]Jacob W. Crandall, Mary L. Cummings, M. Della Penna, and E. de Visser. 2005. Validating human-swarm interaction models with human-in-the-loop experiments. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans 35, 4 (2005), 438–449.
- [4]Bradley Efron and Robert J. Tibshirani. 1994. An Introduction to the Bootstrap. Chapman & Hall/CRC.
- [5]Aoran Jiao, Tanmay P. Patel, Sanjmi Khurana, Anna-Mariya Korol, Lukas Brunke, Vivek K. Adajania, Utku Culha, Siqi Zhou, and Angela P. Schoellig. 2023. Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design. arXiv:2312.01059 [cs.RO]arxiv.org/abs/2312.01059
- [6]Andreas Kolling, Phillip Walker, Nilanjan Chakraborty, Katia Sycara, and Michael Lewis. 2016. Human Interaction With Robot Swarms: A Survey. IEEE Transactions on Human-Machine Systems 46, 1 (2016), 9–26.DOI
- [7]Jungdae Lee, Taiki Miyanishi, Shuei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, and Nakamasa Inoue. 2024. CityNav: Language-Goal Aerial Navigation Dataset with Geographic Information. arXiv:2406.14240 [cs.RO]
- [8]Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as Policies: Language Model Programs for Embodied Control. In IEEE International Conference on Robotics and Automation (ICRA), 9493–9500.arxiv.org/abs/2209.07753
- [9]Shubuo Liu, Hongsheng Zhang, Yuanqai Qi, Peng Wang, Yanning Zhang, and Qi Wu. 2023. AerialVLN: Vision-and-Language Navigation for UAVs. In 2023 IEEE/CVF Int. Conf. on Computer Vision (ICCV). 15338–15348.arxiv.org/abs/2308.06735
- [10]Artem Lykov, Sausar Karaf, Mikhail Martynov, Valerii Serpiva, Aleksey Fedoseev, Mikhail Konenkov, and Dzmitry Tsetserukou. 2024. FlockGPT: Guiding UAV Flocking with Linguistic Orchestration. In 2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), 485–488.arxiv.org/abs/2405.05872
- [11]Mingyang Lyu, Yibo Zhao, Chao Huang, and Hailong Huang. 2023. Unmanned Aerial Vehicles for Search and Rescue: A Survey. Remote Sensing 15, 13 (2023), 3266.DOI
- [12]Dan R. Olsen and Michael A. Goodrich. 2004. Metrics for evaluating human-robot interactions. Proc. of the 2004 ACM/IEEE Conf. on Human-Robot Interaction (2004), 80–87.
- [13]Oleg Sautenkov, Aibek Akhmetkazy, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Grik Tadevosyan, Artem Lykov, and Dzmitry Tsetserukou. 2025. UAV-VLPA*: A Vision-Language-Path-Action System for Optimal Route Generation on a Large Scales. arXiv:2503.02544 [cs.RO]
- [14]Oleg Sautenkov, Yasheerah Yaqoot, Artem Lykov, Muhammad Ahsan Mustafa, Grik Tadevosyan, Aibek Akhmetkazy, Miguel Altamirano Cabrera, Mikhail Martynov, Sausar Karaf, and Dzmitry Tsetserukou. 2025. UAV-VLA: Vision-Language-Action System for Large Scale Aerial Mission Generation. In Proc. of the 2025 ACM/IEEE Int. Conf. on Human-Robot Interaction. 1588–1592.arxiv.org/abs/2501.05014
- [15]Oleg Sautenkov, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Faryal Batool, Jeffrin Sam, Artem Lykov, Chih-Yung Wen, and Dzmitry Tsetserukou. 2025. UAV-CodeAgents: Scalable UAV Mission Planning via Multi-Agent ReAct and Vision-Language Reasoning. arXiv:2505.07236 [cs.RO]arxiv.org/abs/2505.07236
- [16]Jinya Su, Xiaoyong Zhu, Shihua Li, and Wen-Hua Chen. 2023. AI meets UAVs: A survey on AI empowered UAV perception systems for precision agriculture. Neurocomputing 518 (2023), 242–270.DOI
- [17]Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable Code Actions Elicit Better LLM Agents. In International Conference on Machine Learning (ICML).
- [18]Fanglong Yao, Yuanchang Yue, Youzhi Liu, Xian Sun, and Kun Fu. 2025. Aero-Verse: UAV-Agent Benchmark Suite for Simulating, Pre-training, Finetuning, and Evaluating Aerospace Embodied World Models. arXiv:2408.15511 [cs.RO]arxiv.org/abs/2408.15511
- [19]Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In International Conference on Learning Representations (ICLR).arxiv.org/abs/2210.03629
- [20]Haoyu Zhang, Shuoxun Zhang, Peng Ye, Lin Zhang, Jiakang Yuan, Shenghong Yi, Yuening Wang, and Tao Chen. 2026. Advancing MLLM-based UAV Image Understanding and Reasoning: A Benchmark and a Training-Free Multi-Agent System. arXiv:2608.11738 [cs.CV]arxiv.org/abs/2608.11738
- [21]Weichen Zhang, Yuxuan Liu, Xuzhe Wang, Xuecheng Chen, Chen Gao, and Xinlei Chen. 2024. Embodied City: Embodied Aerial Agent for City-level Visual Language Navigation Using Large Language Model. In 2024 23rd ACM/IEEE Int. Conf. on Information Processing in Sensor Networks (IPSN). 265–266.DOI


