GeoFaham: Domain-Specialized Multi-Agent Geospatial Analytics for Disaster Response

Introduction

Geospatial decision-making during disasters often requires executing complex, multi-source workflows rather than retrieving a single fact or generating an isolated map. This challenge is especially pronounced for policymakers and emergency-response teams in the Global South, where access to technical expertise, computational resources, and integrated geospatial infrastructure can be limited [80] (Chen, Shirazi, and Wang 2021). Answering an operational question may involve identifying the correct disaster extent, retrieving relevant infrastructure and road networks, selecting satellite imagery from appropriate pre- and post-event periods, performing spatial and raster operations, and producing reusable outputs such as maps, tables, GeoJSON layers, or GeoTIFFs. For example, determining whether hospitals near a flood remain potentially accessible requires combining hazard boundaries, healthcare facilities, and transportation networks while preserving valid spatial reference and distance calculations. Although GIS experts can manage these dependencies explicitly, natural-language systems must coordinate them automatically. A failure at any step such as selecting the wrong spatial extent, time window, or coordinate system can produce a plausible but operationally incorrect result.

Recent advances in large language models (LLMs) have enabled agentic systems that can autonomously plan, invoke tools, and adapt to intermediate observations (L. Wang et al. 2024; Xi et al. 2025). While such systems have shown strong performance on text-centric multi-step tasks, applying them to geospatial analytics introduces three key challenges. (i) Domain complexity: geospatial reasoning requires specialized knowledge of spatial databases and coordinate systems, satellite imagery and spectral information, map-derived data, and raster analysis. (ii) Data heterogeneity: operational pipelines combine vector layers, multi-band raster imagery, tabular summaries, and temporal products, each requiring distinct tools, formats, and validation procedures. (iii) Scale and context: geospatial artifacts can be large; GeoJSON layers may contain thousands of features, while raster products can reach gigabytes. Passing complete intermediate results through LLM context windows is therefore impractical, even though downstream correctness depends on preserving metadata such as coordinate reference systems, band names, acquisition timestamps, and spatial bounds.

General-purpose coding agents provide a promising execution framework for addressing these challenges. By decomposing tasks, generating and executing code, inspecting outputs, and revising plans based on intermediate observations, they offer a natural foundation for geospatial decision support. A straightforward approach is therefore to equip such an agent with geospatial libraries, spatial databases, map APIs, and satellite imagery catalogs, and ask it to solve geospatial workflows end-to-end. However, disaster-response workflows introduce failure modes that are not captured by successful code execution alone. An answer may appear fluent and its code may execute successfully, while the underlying analysis remains incorrect because the agent selected the wrong disaster footprint, used an inappropriate spatial boundary, searched imagery outside the relevant event window, computed distances in an invalid CRS, or relied on an intermediate artifact that was planned but never produced.

To address these challenges, we introduce GeoFaham (Figure 1), a domain-specialized multiagent system for geospatial disaster analytics. Rather than relying on a single generalist agent with broad tool access, GeoFaham decomposes geospatial workflows into four specialized roles defined by data source and operation type: structured spatial data querying, map feature retrieval, satellite imagery discovery, and raster analysis. A central Orchestrator coordinates these agents through a dual-ledger control loop. The outer loop extracts structured task facts and constructs a dependency-aware plan, while the inner loop tracks execution progress, verifies that required artifacts have been produced, and replans when execution stalls or a dependency cannot be satisfied.

A key feature of GeoFaham is artifact-aware collaboration. Agents do not pass large coordinate arrays, raster data, or complete imagery payloads through the LLM context. Instead, full outputs are persisted in a working memory called the Response Store, while the Orchestrator exchanges compact response identifiers and typed metadata. Agents reason over schemas, summaries, spatial bounds, temporal ranges, and band metadata, while deterministic geospatial engines operate on the full artifacts. This design reduces context overhead, reduces the risk of critical spatial and temporal metadata loss, and keeps downstream computation grounded in the full artifacts.

Existing geospatial benchmarks evaluate remote-sensing tool use and geoprocessing workflows ([70, 35, 38, 98]), but do not fully capture cross-source workflows that require coordination across heterogeneous geospatial data, intermediate artifact handling, and recovery from spatial or temporal execution failures. We therefore introduce GeoFaham-Bench for evaluating multi-step geospatial disaster workflows. GeoFaham-Bench contains 40 natural-language tasks grounded in 17 real-world disaster records spanning fires, floods, hurricanes, cyclones, earthquakes, and tornadoes. Each task captures an operational workflow relevant to GIS analysts, humanitarian workers, or government decision-makers. The tasks require integration across disaster databases, OpenStreetMap features ([60]), satellite imagery catalogs, and raster products, together with operations such as spatial joins, aggregation, ranking, and artifact generation. Because multiple valid execution traces may satisfy the same task, GeoFaham-Bench uses a non-rigid evaluation protocol that checks spatial, temporal, methodological, and output-level invariants rather than enforcing an exact execution trace.

We evaluate GeoFaham on GeoFaham-Bench against three strong general-purpose coding-agent baselines: Codex ([58]), Claude Code ([3]), and OpenCode ([59]). GeoFaham achieves an overall score of 0.940, compared with 0.879 for Codex, the best-performing baseline, while using approximately 7.5 times fewer reported tokens. Its largest gains occur in data selection correctness and spatial-temporal logic, improving over the best baseline by 11.9 and 13.9 points, respectively. These results suggest that GeoFaham’s advantage does not arise simply from better code generation, but from its combined domain-specialized, validation-aware, and artifact-preserving design.

Our contributions are as follows.

1. We introduce GeoFaham, a domain-specialized multi-agent architecture for geospatial disaster analytics that decomposes workflows into bounded specialist roles and coordinates them through validation-aware orchestration.

2. We introduce a Response Store for artifact-aware collaboration that enforces a boundary between LLM reasoning and geospatial computation: agents exchange compact identifiers and typed metadata, while raw geometries, rasters, and imagery payloads remain outside the LLM context.

3. We introduce GeoFaham-Bench, a 40-task benchmark grounded in 17 real-world disaster records spanning fires, floods, hurricanes, cyclones, earthquakes, and tornadoes. It evaluates multi-step geospatial reasoning, cross-source integration, raster-vector computation, and artifact generation using a non-rigid evaluation protocol.

4. We demonstrate that GeoFaham outperforms strong general-purpose coding-agent baselines while using substantially fewer tokens, with the largest gains concentrated in data selection and spatial-temporal reasoning.

Related Work

GeoFaham sits at the intersection of (i) LLM-agent reasoning and tool use, (ii) multi-agent orchestration with typed memory, and (iii) tool-augmented agents for geospatial, GIS, and Earth-observation (EO) workflows. We summarize each and then position GeoFaham along five axes.

LLM Agents and Multi-Agent Orchestration

Modern LLM agents interleave reasoning with tool calls via chain-of-thought [86] and the ReAct loop [93], learn to invoke APIs [69, 65], and self-correct through verbal critique [51, 73] or tree search [92]. Surveys [83, 21, 23] (Xi et al. 2025) place these capabilities within now-standard agent taxonomies. Multi-agent systems decompose tasks across specialized roles [87, 78]: CAMEL (G. Li et al. 2023) elicits cooperative role-play; MetaGPT [24] encodes human SOPs and structured artifacts; AgentVerse [10] studies dynamic composition; ChatDev [63] structures software-engineering hand-offs; AutoGen (Wu et al. 2024) provides a conversational programming model; HuggingGPT [71] pioneers controller–specialist dispatch; Magentic-One [17] adds a generalist Orchestrator with explicit ledger state. AutoAgents [7], MegaAgent [85], DyLAN (Z. Liu et al. 2024), Mixture-of-Agents [82], AgentScope [18], and Internet-of-Agents [11] extend topology and scale. A parallel planning thread—Plan-and-Solve [84], LLM-Planner [75], LLMCompiler [37], ADaPT [62], ReWOO (Xu et al. 2023)—separates plan generation, parallel dispatch, and recursive replanning. GeoFaham’s dual-ledger design combines an upfront dependency-aware plan in the spirit of Plan-and-Solve, parallel-friendly structure as in LLMCompiler, and ADaPT-style replanning gated by a stall counter and validation predicates.

For memory and context, CoALA [76] provides the working/episodic/semantic/procedural taxonomy we adopt; Voyager (G. Wang et al. 2024) demonstrates a persistent skill library; TaskWeaver [64] indexes code artifacts across turns; Mixture-of-Agents [82] explicitly concatenates prior-tier outputs. None of these addresses geospatial artifacts (layers with CRS, STAC items with temporal windows, rasters with bounds and band schemas)—a gap GeoFaham’s typed Response Store targets.

Code-Generating and Data-Science Agents

GeoFaham’s Raster Agent generates sandboxed Python over xarray, rasterio, and geopandas, building on the code–LLM lineage [9, 4, 68, 30] (Li, Allah, et al. 2023). Data-science benchmarks measure increasingly realistic workflows: DS-1000 [41] on NumPy/pandas, ARCADE [94] on multi-turn notebook state, InfAgent-DABench (Hu et al. 2024) on CSV analysis, DSBench [34] on Kaggle-style end-to-end pipelines, BLADE [20] on analytical decision diversity, Data-Copilot [101] on autonomous workflows, and Spider2-V [5] on full data-science environments. NL→SQL work [102] (Spider (T. Yu et al. 2018; ), BIRD (Li, Hui, et al. 2023) shows that LLMs collapse on real-world schemas with domain-specific operators, motivating the dedicated spatial-SQL line below.

Autonomous GIS, Earth Observation Copilots, and Geospatial Benchmarks

LLM–Geo (Li and Ning 2023) introduced the autonomous-GIS pattern; follow-ups specialize it to data retrieval [57], spatial analysis (GIS Copilot [1]), shapefile processing (ShapefileGPT [47]), AutoGPT-style chains (GeoGPT [103]), OSM-augmented prompting (GeoLLM [53]), realistic copilot environments (GeoLLM-Engine [74]), four-agent plan-retrieve-codedebug pipelines (MapCoder [31]), PostGIS NL2SQL (GeoSQL-Eval [25]), and staged multi-agent spatial Text-to-SQL [36]. EO foundation and vision-language models supply the imagery side—RemoteCLIP [48], GeoChat [39], SkyEyeGPT [97], LHRS-Bot [56], EarthGPT [100], RSGPT (Hu et al. 2025), VHM [61], EarthMarker [99], Prithvi-EO [32, 77], SatMAE [12], Scale-MAE [67]—all targeting tasks (flood mapping, burn-scar detection, damage assessment) that recur in GeoFaham [52].

A rapidly growing set of geospatial benchmarks evaluates these systems. Spatial reasoning and map QA: SpartQA [55] and StepGame [72] probe textual spatial reasoning; MapEval [15] and MapQA (Z. Li et al. 2025) ground questions in real map APIs and OSM. Remote-sensing agents: ThinkGeo [70] measures imagery-grounded tool use; UnivEARTH [35] finds execution failures of generated Earth-Engine code dominate end-to-end error; AutoGEEval [27] grades GEE Python-API code generation; GEO-Bench [40] and GEOBench-VLM [14] standardize EO foundation- and VLM-task suites. GIS practitioner workflows: GeoBenchX [38] scores tool-calling against a fixed function set with intentionally unsolvable probes; GeoAnalystBench [98] measures expert-validated geoprocessing workflows; GeoCode-Bench [26] evaluates 8,000+ geospatial-code questions across eight capability levels; GeoAgentBench (B. Yu et al. 2026) ships a dynamic sandbox with 117 atomic GIS tools. For methodological context, generalist agent benchmarks [54, 33, 90, 105] (Liu et al. 2024) frame the evaluation paradigm, and our use of an LLM-as-judge follows Zheng Zheng et al. [104].

Disaster-Response and Humanitarian Datasets

GeoFaham is built on 17 real-world disaster records with associated damage and flood-extent imagery. The community datasets that established these formats are foundational: xBD [22] standardized four-class building damage across 45,362 km² of paired pre/post imagery; BRIGHT [8] extended damage assessment to all-weather SAR+optical and serves as the 2025 IEEE GRSS Data-Fusion-Contest dataset; FloodNet [66] provides post-Harvey UAV segmentation and VQA; SpaceNet [79] established building-footprint and road-network protocols. On the language side, CrisisMMD [2] introduced multimodal crisis-tweet classification and HumSet [16] provides multilingual humanitarian extraction from UN OCHA reports. Cutter Cutter, Boruff, and Shirley 2003 and Goodchild [19] ground the demand for practitioner-relevant geospatial decision support.

Positioning GeoFaham

We group prior work into five categories: (1) remote-sensing agent benchmarks, (2) GIS-practitioner benchmarks, (3) autonomous GIS systems, (4) general-purpose multi-agent frameworks, and (5) data-science agent benchmarks. Below, we summarize what each category covers and the gap that GeoFaham addresses. Remote-sensing agent benchmarks. [70, 35, 14] (Hou, Shen, et al. 2025). These benchmarks test how well an agent works with Earth observation data from a single source. They do not involve several cooperating agents or the passing of intermediate results between them. GeoFaham combines several data sources in one workflow, lets agents share typed outputs (e.g., rasters and vector layers) through a shared memory, and evaluates the quality of the geospatial method, not only the final answer. GIS-practitioner benchmarks [38, 98, 26 B. Yu] et al. 2026). These benchmarks check whether a single agent calls the right tools from a fixed set of functions. They do not test whether an agent can keep track of intermediate outputs or recover when a step fails, for example when imagery does not cover the area of interest or layers use different projections. GeoFaham uses an orchestrator that tracks progress, checks each result before moving on, and replans when a step fails. Autonomous GIS systems ([103, 1, 47, 74, 36] (Li and Ning 2023)). These systems automate GIS tasks, but each mostly works with one type of data and does not check intermediate results before continuing. GeoFaham coordinates four specialist agents (Maps, STAC, Vector, and Raster) through one Orchestrator. A shared Response Store keeps every intermediate output so that later steps can reuse it. General-purpose multi-agent frameworks ([17, 24, 63, 64, 37, 62] (Wu et al. 2024; Chen et al. 2024)). These frameworks provide generic ways for agents to plan and work together, but they are not designed for geospatial data. GeoFaham builds on the ledger-based planning of Magentic-One and adapts it to geospatial work: agents exchange geospatial outputs, results are checked for the correct coordinate reference system (CRS), spatial coverage, and geometry type before they are used, and each task is routed according to how its area of interest is defined (database extent, a fetched boundary, or a user-provided geometry). To our knowledge, no general-purpose framework lets agents find earlier geospatial outputs by their properties. Data-science agent benchmarks ([41, 94, 34, 5] (Hu et al. 2024; Gu et al. 2024)). These benchmarks focus on tabular data, such as CSV files and databases. GeoFaham extends this setting to geospatial data (rasters, vector layers, and STAC items), with tasks built from 17 real-world disaster records. Its evaluation rubric also includes geospatial criteria that data-science benchmarks do not cover, such as spatio-temporal reasoning, the quality of geometric operations, geometry data integrity, and the use of a metric CRS for distance and area calculations.

System Architecture

Figure 1 provides an overview of the GeoFaham architecture. GeoFaham decomposes geospatial workflows along data-source and operation-type boundaries. A central Orchestrator maintains global task state and execution control, coordinating four domain-specialized agents: the Vector Agent for spatial database queries and vector operations; the Maps Agent for map-derived feature retrieval; the STAC Agent for satellite imagery discovery and coverage validation; and the Raster Agent for code-based raster analysis. The architecture follows three principles. First, bounded specialization: each agent has a narrow action space, domain-specific prompt, and standardized output contract. Second, state-grounded orchestration: intermediate geospatial artifacts become part of the evolving task state, grounding planning, dispatch, recovery, and final synthesis in executed evidence rather than a static plan. Third, artifact-aware memory: large geospatial outputs remain outside the LLM context, while agents exchange compact identifiers and typed metadata. Together, these principles constrain execution while preserving the flexibility required for open-ended geospatial workflows. While GeoFaham is instantiated and evaluated for disaster response in this work, these architectural principles are not disaster-specific and can support other multi-source geospatial analysis settings.

Orchestrator

The Orchestrator is responsible for interpreting the user request, planning a step-by-step workflow, dispatching agents, tracking progress, recovering from failures, and producing the final answer. It uses a dual-ledger control loop ([17]). The outer loop builds a task ledger for fact extraction and planning, while the inner loop maintains a progress ledger for execution monitoring and replanning, as shown in Figure 2.

Outer loop: Task ledger. Given a user query, the Orchestrator extracts a structured task state that records the user intent, requested output, disaster context, temporal constraints, area of interest (AOI), target entities, required spatial operations, and reusable artifacts from the current session. For example, the query “Which hospitals are near the 2023 Lahaina fire footprint?” is converted into explicit fields describing the event, hazard footprint, target feature class, distance constraint, and expected output. This stage reduces errors that arise before execution, such as selecting the wrong disaster record, using an administrative boundary instead of a hazard footprint, or searching imagery over an inappropriate temporal window. Using the extracted task state and the documented capabilities of each agent, the Orchestrator constructs a dependency-aware plan. Each step specifies the responsible agent, semantic goal, required inputs, expected output type, and dependencies on earlier steps. Agent instructions express goals rather than low-level procedures. For example, the Orchestrator may request healthcare facilities within a disaster footprint without prescribing the exact map tags or implementation details. This makes the workflow auditable while allowing each specialist to apply domain-specific reasoning.

Inner loop: Progress ledger. During execution, the progress ledger records completed and pending steps, response identifiers, artifact metadata, failures, and unresolved dependencies. At each iteration, the Orchestrator determines whether the task is complete, whether execution has stalled, which agent should act next, and what instruction it should receive. Before dispatching an agent, the Orchestrator performs three checks. The plan-adherence check verifies that all prerequisite steps have been completed. The input-validity check verifies that every required artifact exists as a registered response rather than only as an expected output in the plan. The geospatial-compatibility check verifies that the geometry types, coordinate reference systems, bounds, temporal coverage, and schemas of the required artifacts support the intended operation. For example, a distance computation requires a suitable projected CRS or geodesic method, while raster analysis requires imagery with the necessary bands and sufficient spatial coverage. A planned output is therefore not treated as a realized artifact. A downstream agent cannot use a flood footprint, road layer, or imagery set until the corresponding response identifier exists in the Response Store and its metadata passes validation. This prevents hallucinated handoffs in which an agent proceeds even though a previous step failed, returned an empty result, or produced an incompatible artifact.

Replanning. When repeated tool errors, empty outputs, missing artifacts, insufficient imagery coverage, or unresolved dependencies cause execution to stall, control returns from the inner loop to the outer loop. The Orchestrator updates the task state with the observed failure and revises the plan. Replanning may change the spatial scope, request a different boundary, broaden a temporal window, switch imagery collections, decompose a broad request into smaller feature classes, or route a step to another specialist. This provides a structured recovery mechanism rather than relying on repeated retries. The Orchestrator limits each agent’s visibility to the information required for its assigned step. This keeps prompts focused, reduces token usage, and prevents unrelated intermediate state from interfering with execution. The Orchestrator maintains global workflow control, while the Response Store provides shared access to realized artifacts.

Response Store

GeoFaham separates LLM reasoning from geospatial data handling through the Response Store, a persistent artifact registry shared by all agents. After each tool call, the complete output is stored and registered with a unique identifier, source, summary, artifact path, and typed metadata. Vector metadata includes geometry type, coordinate reference system (CRS), bounds, feature count, and property schema. Imagery and raster metadata includes temporal coverage, bands, resolution, bounds, CRS, and summary statistics. Only compact identifiers and metadata pass through the Orchestrator. Full artifacts remain outside the LLM context and are resolved by downstream agents when needed. This avoids repeatedly summarizing large outputs and preserves information required for validation and later computation. The Response Store also separates language reasoning from deterministic spatial execution. For example, the Vector Agent generates spatial queries over symbolic artifact identifiers. The execution layer resolves these identifiers to the corresponding geometries, executes the operation, and registers the result as a new response. Agents may also retrieve prior responses by source or data type, supporting artifact reuse, follow-up questions, and traceable final answers, as illustrated in Figure 3.

Specialized Agents

Each specialist has a bounded tool set and returns outputs through a shared response schema, giving the Orchestrator a uniform interface across heterogeneous data sources.

Vector Agent. The Vector Agent performs spatial database queries and vector operations, including joins, buffering, distance filtering, aggregation, and artifact generation. It can combine database records with vector artifacts from other agents without exposing raw coordinates to the LLM. Instead, the agent receives response identifiers and typed metadata and generates spatial SQL over symbolic placeholders. At execution time, these placeholders are resolved to the artifacts and loaded into temporary spatial tables. The database engine performs the exact geometry operations, and the resulting artifact is registered in the Response Store.

Maps Agent. The Maps Agent retrieves map-derived features such as roads, facilities, boundaries, waterways, and infrastructure. Its area of interest may be a place name, bounding box, or registered spatial artifact. Each call retrieves one feature class, producing cleaner schemas and more predictable geometry types for downstream operations.

STAC Agent. The STAC Agent searches satellite imagery and raster catalogs using spatial, temporal, collection, cloud-cover, and band constraints. It validates scene coverage before raster analysis and may broaden the temporal window, switch collections, or return a structured failure when suitable imagery is not found.

Raster Agent. The Raster Agent generates sandboxed workflows for spectral indices, change detection, flood and burn analysis, classification, raster-to-vector conversion, and zonal statistics. It may inspect intermediate statistics and correct execution errors before registering outputs and their spatial metadata in the Response Store.

Execution Semantics and Final Synthesis

A GeoFaham workflow forms a typed artifact graph in which executed steps produce registered artifacts consumed by later steps. Natural-language statements are not treated as substitutes for artifacts. For example, a claim that schools lie within a flood extent must be supported by registered school and flood layers and an executed spatial join. Final synthesis begins after the required dependencies are satisfied or remaining failures are identified. The Orchestrator reports the findings, references generated artifacts, and states limitations such as missing map features, partial imagery coverage, or failed operations. The resulting answer remains traceable to concrete geospatial outputs.

Implementation

GeoFaham is implemented as a closed multi-agent team using a conversational agent framework (Wu et al. 2024), with custom orchestration, artifact storage, and geospatial execution layers. Vector outputs are stored as GeoJSON, raster outputs as Cloud-Optimized GeoTIFFs or derived layers, and imagery searches as registered item sets with temporal and band metadata. The file-backed, session-scoped Response Store supports reuse, recovery, and traceable handoffs. The implementation is model-agnostic. Models can be assigned to the planner, progress tracker, and specialists without changing the shared artifact interface, allowing stronger models to be reserved for planning, spatial query generation, and raster programming.

GeoFaham-Bench

Evaluating geospatial agents requires more than checking the final answer. A fluent response may be grounded in the wrong disaster footprint, imagery window, coordinate system, or intermediate artifact. Exact-match evaluation is also unsuitable because valid solutions may use different tools, execution orders, spatial predicates, or raster thresholds while satisfying the same spatial, temporal, methodological, and output requirements.

To evaluate these capabilities, we introduce GeoFaham-Bench, a benchmark containing 40 natural-language tasks across 17 real-world disaster records. The benchmark spans 10 task categories covering damage assessment, flood analysis, infrastructure and point-of-interest analysis, satellite imagery search, raster computation, spatial aggregation, and artifact generation. The task set emphasizes challenging multi-step workflows, with 24 of 40 tasks (60%) categorized as hard. We use the benchmark for two evaluations: an architecture-neutral comparison with external coding agents and a process-centric analysis of model assignments within GeoFaham. Detailed task and disaster statistics are provided in Appendix 1.2.

Task Design

Tasks are written as user-facing disaster-response questions rather than tool-specific instructions, allowing systems to follow different valid workflows. They represent operational needs of GIS analysts, humanitarian workers, and government decision-makers, such as identifying exposed facilities, ranking damaged regions, retrieving event-relevant imagery, estimating affected area, or producing map artifacts. Task difficulty is defined by workflow complexity. Harder tasks require additional data sources, temporal reasoning, raster computation, or artifact handoffs. Each task also specifies the required evidence type, so reporting a statistic without a requested supporting artifact is considered incomplete.

Ground Truth Construction

GeoFaham-Bench uses non-rigid ground truth that specifies required invariants, acceptable variations, expected-result checks, and common failure modes. This accommodates equivalent spatial predicates, execution orders, and raster-analysis thresholds, as well as small variations caused by dynamic sources such as OpenStreetMap and satellite catalogs. References are constructed through human-in-the-loop execution. For each task, a geospatial expert defines the disaster context, spatial and temporal scope, required evidence, and expected outputs. An LLM proposes a step-by-step workflow, tool calls, and implementation details, while the expert reviews each step and validates the intermediate results. Incorrect data sources, parameters, spatial operations, or outputs are corrected before proceeding. Corrections are limited to factual and validity errors, while the reasoning strategy and implementation choices remain with the system.

Evaluation Protocols

We use structured LLM-as-judge rubrics derived from validated ground truth. Because external coding agents and GeoFaham expose different levels of execution evidence, we use separate cross-system and internal protocols. Both organize fine-grained evaluation dimensions into broader categories, then combine them into an overall score. All systems receive the same tasks, data sources, geospatial libraries, execution environment, and output requirements. They differ only in agent architecture, prompting, and orchestration strategy.

The cross-system judge evaluates each run using the evidence exposed by that system, including generated code, logs, structured results, and output artifacts. Failed or timed-out runs are evaluated using the evidence produced before failure. The protocol scores 13 dimensions grouped into five: Task Outcome ($S_{\mathrm{out}}$), Geospatial Method ($S_{\mathrm{meth}}$), Reproducibility ($S_{\mathrm{repr}}$), Compliance ($S_{\mathrm{comp}}$), and Robustness ($S_{\mathrm{rob}}$). The overall cross-system score ($S_{\mathrm{c}}$) is computed as:

Task Outcome measures answer accuracy and completeness. Geospatial Method evaluates data selection, spatiotemporal logic, geometric operations, geometry integrity, and implementation correctness. Reproducibility measures structured outputs, method traceability, and artifact quality, while Compliance and Robustness assess benchmark-rule adherence, error handling, and confidence calibration. Geometry-derived quantities must be traceable to deterministic spatial engines rather than inferred from coordinate text. The complete dimension definitions, scoring criteria, and diagnostic flags are provided in Appendix 2.1.

For internal ablations, the judge additionally inspects task and progress ledgers, agent instructions, tool calls, intermediate artifacts, failures, and replanning steps. It scores 11 dimensions grouped into four categories: Planning ($S_{\mathrm{plan}}$), Execution ($S_{\mathrm{exec}}$), Compliance ($S_{\mathrm{comp}}$), and Output ($S_{\mathrm{out}}$). The overall internal evaluation score ($S_i$) is computed as:

Planning evaluates workflow construction and agent instructions. Execution measures tool selection, parameter accuracy, code logic, and result summarization. Compliance captures prompt adherence and error recovery, while Output measures final-answer accuracy and completeness. This protocol is used only to study which model assignments and roles affect performance within GeoFaham. The dimension-level rubric is provided in Appendix 2.2.

Experiments

Experimental Setup

Systems. We compare GeoFaham with three general-purpose coding-agent configurations: Codex with GPT-5.1, Claude Code with Claude Sonnet 4.6, and OpenCode with GPT-5.1. All systems receive the same 40 benchmark tasks, source constraints, and geospatial execution environment. Each system may generate and execute code, queries, and tool calls to solve the tasks. For internal ablations, we evaluate four model-assignment configurations on the same tasks. C1 assigns GPT-4o to all roles. C2 upgrades only the Orchestrator planner to GPT-5.1. C3 additionally assigns GPT-5.1 to the progress tracker, Vector Agent, and Raster Agent, while retaining GPT-4o for the Maps and STAC agents. C4 assigns GPT-5.1 to all roles. For the cross-system comparison, GeoFaham uses the C3 tiered model assignment, with fresh runs conducted alongside the external coding agents under the same evaluation setup.¹ We use GPT-5.2 as the LLM judge for both the cross-system and internal evaluation protocols.

System	$S_{\mathrm{out}}$	$S_{\mathrm{meth}}$	$S_{\mathrm{repr}}$	$S_{\mathrm{comp}}$	$S_{\mathrm{rob}}$	$S_{\mathrm{c}}$
GeoFaham	0.897	0.951	0.982	0.999	0.933	0.940
Codex	0.794	0.890	0.974	0.986	0.896	0.879
Claude Code	0.738	0.863	0.902	0.974	0.873	0.837
OpenCode	0.622	0.828	0.865	1.000	0.838	0.780

Evaluation. Cross-system comparisons use the architecture-neutral protocol, which scores task outcome, geospatial method, reproducibility, compliance, and robustness. Internal ablations use the process-centric judge, which also examines planning, agent instructions, tool calls, execution state, and recovery. The two protocols answer different questions: whether GeoFaham outperforms external agents, and which model assignments improve performance within it.

Efficiency. We report average wall-clock duration and token usage per task. Token counts are obtained from system logs and include input, output, and reasoning tokens where exposed by the system. They are reported as efficiency indicators and do not affect correctness scores.

Cross-System Comparison

Table 1 reports the aggregate results. GeoFaham achieves an overall score of 0.940, outperforming Codex, Claude Code, and OpenCode by 6.1, 10.3, and 16.0 points, respectively. Codex is the strongest baseline with a score of 0.879. The largest category-level gains over Codex occur in task outcome, where GeoFaham scores 0.897 versus 0.794, and geospatial method, where it scores 0.951 versus 0.890. Compliance is high across all systems, indicating that agents generally follow instructions without violating benchmark rules. Figure 4 provides a dimension-level breakdown. The largest gains occur in spatial-temporal logic and data selection correctness. GeoFaham scores 0.918 and 0.972 on these dimensions, exceeding the strongest baseline by 13.9 and 11.9 points, respectively. These dimensions measure whether a system selects the correct evidence, preserves the intended spatial and temporal scope, and orders dependent operations correctly. In contrast, implementation correctness is comparable across systems: GeoFaham scores 0.922, while the strongest baseline scores 0.934. Geometry integrity is also consistently high, ranging from 0.956 to 0.997. These results indicate that GeoFaham’s advantage lies in selecting, coordinating, and validating geospatial operations rather than code generation alone. GeoFaham also achieves the highest artifact-quality score of 0.976, consistent with its artifact-aware execution and Response Store. Dimension-level scores are provided in the supplementary material.

Model-Assignment Ablation

Table 2 reports aggregate performance across the four internal configurations. C1, which assigns GPT-4o to all roles, achieves an overall score of 0.808. Upgrading only the Orchestrator planner in C2 raises the score to 0.914, an improvement of 10.6 points. This identifies planning as a major control point in the workflow. C3 additionally upgrades the progress tracker, Vector Agent, and Raster Agent while retaining GPT-4o for Maps and STAC. C3 achieves the highest mean score of 0.964, while C4, which assigns GPT-5.1 to every role, performs comparably with a score of 0.959. These results suggest that stronger models are most valuable in roles requiring planning, execution monitoring, spatial query generation, and raster programming. Constrained retrieval agents can use less expensive models without reducing overall performance. Dimension-level results further show that upgrading the planner primarily improves workflow construction and agent instructions, while stronger execution-oriented agents improve parameter accuracy, code logic, and final answer quality. Full per-dimension results are provided in the supplementary material.

Config	$S_{\mathrm{plan}}$	$S_{\mathrm{exec}}$	$S_{\mathrm{comp}}$	$S_{\mathrm{out}}$	$S_{\mathrm{i}}$
C1	0.849	0.832	0.825	0.680	$0.808 \pm0.12$
C2	0.944	0.906	0.912	0.885	$0.914 \pm0.07$
C3	0.983	0.958	0.961	0.952	$0.964 \pm0.03$
C4	0.980	0.957	0.958	0.934	$0.959 \pm0.04$

Error Recovery Analysis

Figure 5 distinguishes error occurrence from recovery. C1 encounters errors in 61.3% of runs and recovers 89.8% of them, resulting in 93.8% net success. C2 has the same error rate of 61.3%, but recovers all observed errors and achieves 100% net success. Here, net success denotes completion without an unrecovered execution error, not perfect benchmark correctness. This suggests upgrading the planner improves failure diagnosis and recovery, although it does not reduce errors. C3 and C4 reduce error occurrence to 33.8% and 28.8%, respectively, and recover all observed errors. Stronger execution-oriented agents therefore complement planning by preventing more failures before requiring recovery.

Efficiency Analysis

Table 3 reports cross-system runtime and token usage. GeoFaham uses an average of 106,344 tokens per task, compared with 795,352 for Codex, 285,100 for Claude Code, and 376,042 for OpenCode. These values correspond to reductions of approximately 7.5×, 2.7×, and 3.5×, respectively. OpenCode is the fastest system at 128.37 seconds per task but achieves the lowest overall score. GeoFaham takes 169.64 seconds per task, compared with 214.92 seconds for Codex and 254.23 seconds for Claude Code. Thus, GeoFaham’s lower token usage does not come at the cost of accuracy or substantially longer execution.

System	Duration sec.	Avg. Tokens Per Task	Median Tokens Per Task	Total Tokens (M)
GeoFaham	169.64	106,344	86,974	4.29995
Codex	214.92	795,352	741,656	31.81
ClaudeCode	254.23	285,100	227,596	11.40
OpenCode	128.37	376,042	317,846	15.04

Conclusion

We presented GeoFaham, a domain-specialized multi-agent system for geospatial disaster analytics. GeoFaham coordinates specialized agents for vector operations, map feature retrieval, imagery discovery, and raster analysis through validation-aware orchestration and artifact-preserving memory. We also introduced GeoFaham-Bench, a 40-task benchmark grounded in 17 real-world disaster records for evaluating multi-step geospatial reasoning, cross-source integration, and artifact generation under non-rigid evaluation. GeoFaham outperforms strong general-purpose coding-agent baselines, with the largest gains in data selection and spatial-temporal reasoning rather than implementation correctness. Internal ablations further show that strong models are most valuable for planning, progress tracking, spatial query generation, and raster analysis, while constrained retrieval roles can use less expensive models. These results highlight that reliable geospatial agents require not only executable code, but also explicit spatial and temporal grounding, validation of artifact dependencies, and persistent management of intermediate outputs.

Additional Benchmark Detail

Disaster Coverage

GeoFaham-Bench includes disaster records spanning fires, floods, hurricanes, cyclones, earthquakes, and tornadoes across multiple geographic regions. The complete set of benchmark disaster records is listed in Table 4. The hazard type provides the event context used to determine the relevant temporal window, area of interest, imagery selection, and downstream analysis requirements. For example, fire tasks may involve burn-severity analysis or aggregation of damaged buildings, flood tasks may require flood-footprint intersection and exposure analysis, and earthquake or storm tasks may focus on building damage and critical infrastructure impact. The authoritative disaster layers consist primarily of building-level damage assessments, flood-extent maps, or both. OpenStreetMap features and satellite imagery are retrieved separately to support additional spatial and temporal analysis. The benchmark also covers different regions and administrative structures, introducing variation in place names, boundary availability, and OpenStreetMap coverage. The selected events are recent enough to support satellite-imagery searches over event-centered temporal windows. For each disaster record, we prepare supporting context for task design, including the area of interest, event metadata, available damage or flood layers, relevant map-feature inventories, and a short event overview. Questions are constructed from this grounded context so that the referenced locations, entities, data sources, and expected evidence can be verified during execution.

ID	Type	Location	Country	Date	Available Data
1	Tornado	Saint Louis	USA	2025-05-16	Building damage
2	Hurricane	Black River	Jamaica	2025-10-29	Building damage
3	Hurricane	Montego Bay	Jamaica	2025-10-31	Building damage
4	Flood	Porto Alegre	Brazil	2024-05-09	Building damage
5	Cyclone	Mamoudzou	Mayotte	2024-12-18	Building damage
6	Flood	Valencia	Spain	2024-10-31	Flood map
7	Hurricane	Grenada	Grenada	2024-07-02	Building damage
8	Cyclone	Aceh	Indonesia	2025-11-29	Building damage, flood map
9	Flood	Nairobi	Kenya	2024-05-01	Flood map
10	Flood	Busia	Kenya	2024-05-06	Flood map
11	Fire (Eaton)	Pasadena	USA	2025-01-10	Building damage
12	Fire (Palisades)	Los Angeles	USA	2025-01-08	Building damage
13	Fire (Palisades)	Los Angeles	USA	2025-01-09	Building damage
14	Fire (Palisades)	Los Angeles	USA	2025-01-10	Building damage
15	Earthquake	Mandalay	Myanmar	2025-03-29	Building damage
16	Fire	Lahaina	USA	2023-08-08	Building damage
17	Cyclone	Colombo	Sri Lanka	2025-11-30	Building damage, flood map

Task Design

Questions are designed from five stakeholder perspectives: GIS analysts, humanitarian aid workers, government officials, infrastructure planners, and emergency coordinators. These personas capture complementary operational needs, ranging from precise spatial analysis and map-layer generation to infrastructure assessment, facility prioritization, and situational decision support. The 40 benchmark tasks span 10 broad task categories, including point-of-interest impact analysis, infrastructure analysis, building-damage assessment, flood analysis, satellite-imagery analysis, spatial aggregation, and cross-event comparison. Table 5 summarizes the task distribution.

Task Category	Tasks (#)	Share (%)
Point-of-interest impact analysis	11	27.5%
Infrastructure analysis	8	20.0%
Clustering and aggregation	7	17.5%
Building-damage assessment	4	10.0%
Comparative analysis	4	10.0%
Flood analysis	2	5.0%
Burn-intensity analysis	1	2.5%
Land-cover impact analysis	1	2.5%
Population and humanitarian analysis	1	2.5%
Satellite-imagery analysis	1	2.5%

Difficulty is defined by workflow complexity rather than answer length. Simple tasks require one data source and direct retrieval or aggregation, medium tasks require two sources or a spatial operation, and hard tasks require three or more sources, temporal reasoning, raster computation, or artifact handoff between intermediate stages. Of the 40 tasks, 24 (60%) are hard, 12 (30%) are medium, and 4 (10%) are simple, emphasizing multi-step workflows over direct retrieval. Two tasks are explicitly multi-turn, comprising four follow-up turns in total. Each task also specifies the expected evidence type, including textual statistics, GeoJSON layers, raster outputs, tables, or combined artifacts.

Reference Specifications and Judge Inputs

For each benchmark task, the validated ground truth is represented as a structured specification containing the disaster context, area of interest, event date, target feature type, buffer distance, temporal window, expected data sources, required spatial predicates, CRS constraints, expected output type, acceptable numerical tolerances, and known failure modes. This specification defines the spatial, temporal, methodological, and output-level conditions that a valid solution must satisfy, without requiring systems to reproduce a single implementation. The cross-system judge evaluates each run using the evidence exposed by that system. For GeoFaham, this includes run metadata, conversation history, agent messages, Response Store entries, and generated artifacts. For external coding agents, it includes the task prompt, stdout, stderr, structured result output, generated files, scripts, SQL, and workspace artifacts. Failed or timed-out runs are evaluated using the evidence produced before failure, allowing supported intermediate results to receive partial credit while penalizing unsupported or hallucinated claims. The evaluation also checks geospatial-specific failure modes, including selection of the wrong evidence source, misidentification of the disaster or area of interest, violation of event-date constraints, inappropriate spatial predicates, and metric computations performed without a suitable projected CRS or geodesic method. Geometry-derived values must be traceable to deterministic spatial engines such as PostGIS, GeoPandas, Shapely, GDAL, or rasterio; values inferred directly from coordinate text are penalized regardless of numerical plausibility. The judge additionally records diagnostic flags for run completion, structured-output validity, timeout, missing artifacts, forbidden-source usage, unsupported claims, and inappropriate CRS usage. These flags support failure-mode analysis independently of the aggregate score.

Detailed Evaluation Rubrics

Both evaluation protocols score each dimension on a five-point scale: 1.00 indicates a correct, complete, and well-supported result; 0.75 a mostly correct result with minor omissions; 0.50 partial correctness with important gaps; 0.25 major flaws with some useful evidence; and 0.00 a missing, unsupported, incorrect, or non-compliant result. Alternative workflows receive full credit when they satisfy the same validated spatial, temporal, methodological, and output-level requirements.

Architecture-Neutral Cross-System Rubric

The cross-system rubric evaluates GeoFaham and external coding agents using evidence available for every system. It does not reward a particular architecture, orchestration strategy, memory mechanism, or tool interface. The rubric contains 13 dimensions grouped into five categories.

Task Outcome

Answer Accuracy. Measures whether the final answer agrees with the expected findings. Numeric counts, rankings, distances, areas, classifications, and affected entities must fall within the tolerances or consensus ranges specified by the ground truth. Alternative valid workflows may produce slightly different values, but the reported result must remain consistent with the executed analysis. Unsupported caveats do not compensate for an incorrect answer.

Answer Completeness. Measures whether the response addresses every requested part of the task, including required statistics, rankings, comparisons, explanations, and output artifacts. A correct textual answer remains incomplete when the task also requests a supporting GeoJSON, raster, table, or map artifact that was not produced.

Geospatial Method

Data Selection Correctness. Measures whether the run selected the correct disaster record, event, area of interest, tables, attributes, imagery collections, map features, and endpoint data. It penalizes using an administrative boundary in place of a hazard footprint, selecting the wrong event, querying irrelevant data, or omitting a required source.

Spatial-Temporal Logic. Measures whether the spatial and temporal reasoning answers the task. Checks include event-date consistency, pre- and post-event windows, buffer distances, spatial joins, predicate semantics, ordering of dependent operations, and handling of missing data. This dimension evaluates the logic of the workflow rather than the numerical quality of the geometric implementation.

Geometric Operation Quality. Measures whether appropriate geometric functions and spatial methods were used. Metric distance, area, and length calculations should use a suitable projected CRS, geodesic method, or geography representation rather than planar operations on latitude–longitude degrees. The judge also checks spatial-predicate choice, valid-geometry handling, spatial-index-friendly formulations, and preservation of coordinate precision. Degree-based metric calculations, inappropriate predicates, ignored invalid geometries, missing CRS transformations, and consequential precision loss are penalized. If no non-trivial geometric operation is required, this dimension receives full credit.

Geometry Data Integrity. Measures whether geometric and numeric results were produced by an executed deterministic spatial engine rather than inferred by the language model from coordinate text. Distances, areas, counts, containment relations, and intersections should be traceable to deterministic spatial engines such as PostGIS, GeoPandas, Shapely, GDAL, or rasterio; values inferred directly from coordinate text are penalized regardless of numerical plausibility. The judge additionally records diagnostic flags for run completion, structured-output validity, timeout, missing artifacts, forbidden-source usage, unsupported claims, and inappropriate CRS usage. These flags support failure-mode analysis independently of the aggregate score. penalized. This criterion is applied equally to all architectures. If no geometry or coordinate computation is involved, the dimension receives full credit.

Implementation Correctness. Measures whether the generated SQL, Python, shell commands, or tool calls execute successfully and produce the result claimed in the answer. Invalid code, failed queries treated as successful, empty outputs interpreted as valid zero results, and discrepancies between executed outputs and reported values are penalized.

Reproducibility and Evidence

Structured Result Evidence. Measures whether sufficiently complete and machine-readable evidence is available for the final output. For external agents, this may consist of a parsed result JSON or an equivalent structured representation. For GeoFaham, the final answer, conversation trace, Response Store entries, and associated metadata provide equivalent evidence. Systems are not required to use the same schema, but missing, incomplete, or unparsable final-output evidence lowers the score.

Method Traceability. Measures whether an evaluator can determine which data sources, queries, code, spatial operations, and intermediate steps produced the answer. The available evidence should make the core method understandable without requiring undocumented operations to be inferred.

Artifact Quality. Measures whether requested GeoJSON files, raster products, maps, tables, scripts, SQL files, or other computed outputs were produced and are relevant, plausible, and usable. Missing, empty, irrelevant, or answer-inconsistent artifacts receive lower scores.

Compliance

Benchmark Rule Compliance. Measures adherence to the benchmark’s data-use and execution rules. External coding agents may use only the supplied benchmark inputs, read-only database access, approved geospatial libraries, and listed service endpoints. They may not use ground-truth files, prior evaluated runs, undisclosed web search, GeoFaham internals, or database writes. GeoFaham may use its own agents and Response Store because these are components of the evaluated system; it is judged instead on task-specific source and execution compliance.

Robustness. Error Handling. Measures how the system responds to failed queries, service errors, empty results, invalid code, insufficient imagery coverage, timeouts, or other execution problems. Credit is given for diagnosing the failure, revising the workflow, or returning a supported partial answer. If no error occurs and the evidence confirms successful execution, this dimension receives full credit.

Evidence-Aware Confidence. Measures whether the certainty expressed in the final answer is consistent with the quality and completeness of the available evidence. Unsupported results presented with high confidence are penalized, as are generic uncertainty statements that do not identify the actual limitation. Appropriate qualifications may describe incomplete map coverage, dynamic data sources, partial imagery coverage, failed operations, or other evidence limitations.

Aggregation

Task Outcome averages Answer Accuracy and Answer Completeness. Geospatial Method averages Data Selection Correctness, Spatial–Temporal Logic, Geometric Operation Quality, Geometry Data Integrity, and Implementation Correctness. Reproducibility averages Structured Result Evidence, Method Traceability, and Artifact Quality. Compliance contains Benchmark Rule Compliance, while Robustness averages Error Handling and Evidence-Aware Confidence. Their respective category weights are 35%, 30%, 15%, 10%, and 10%. Runtime, token consumption, and monetary cost are reported separately and do not affect the evaluation score.

GeoFaham-Internal Process-Centric Rubric

The internal rubric is used only for model-assignment ablations within GeoFaham. Unlike the architecture-neutral protocol, it may inspect the task ledger, progress ledger, agent instructions, tool calls, Response Store entries, intermediate artifacts, detected failures, replanning decisions, and final outputs. It contains 11 dimensions grouped into four categories.

Planning and Orchestration

Plan Correctness. Measures whether the selected agent sequence and dependency structure form a valid workflow. Alternative sequences receive full credit when they are logically sound. The score is reduced when an agent is assigned an operation outside its capabilities or when dependent steps are executed in an invalid order.

Plan Completeness. Measures whether the plan includes all steps needed to answer the question, such as disaster identification, area-of-interest resolution, data retrieval, spatial or raster analysis, artifact generation, and final synthesis. Missing a critical dependency lowers the score. Additional useful steps are not penalized unless they introduce harmful redundancy or unsupported operations.

Instruction Quality. Measures whether the Orchestrator provides clear, self-contained, semantic instructions that specify the intended goal rather than unnecessary low-level procedures. Instructions should include the required disaster, spatial, temporal, and output context, reference realized artifacts through valid response identifiers, and remain within the assigned agent’s capabilities.

Execution

Tool Selection. Measures whether each specialist selects tools appropriate for its assigned operation. Alternative tools receive full credit when they produce the same supported result. Tools that cannot perform the required operation or that bypass necessary geospatial computation are penalized.

Parameter Accuracy. Measures the correctness of tool and query parameters, including event dates, temporal windows, imagery collections, cloud-cover filters, bounding boxes, feature tags, buffer distances, thresholds, spatial predicates, attribute names, band selections, and output settings. Small variations are accepted when they are reasonable and preserve the task semantics.

Code Logic. Measures whether generated SQL or Python is functionally correct. The judge checks spatial joins, filters, aggregation, CRS handling, raster band logic, nodata treatment, masks, thresholds, and output generation. Different implementations receive equivalent credit when they produce the same valid result.

Result Summarization. Measures whether specialist outputs are summarized accurately and in a form useful to the Orchestrator. Summaries should communicate the main result, relevant metadata, limitations, and response identifiers needed for later steps without contradicting the stored artifact.

Compliance and Recovery

Prompt Compliance. Measures adherence to the system’s orchestration and agent rules. This includes semantic instructions, correct artifact passing, use of valid response identifiers, appropriate agent boundaries, CRS-safe spatial operations, spatial-index-aware queries, vectorized raster processing, and avoidance of anti-patterns such as treating a planned artifact as if it had already been produced.

Error Recovery. Measures whether execution failures are detected, diagnosed, and handled appropriately. Recovery may involve correcting a query, changing an invalid parameter, broadening an imagery window, switching collections, requesting a different boundary, rerouting a step, or reporting a supported limitation. If no errors occur, this dimension receives full credit. Repeated unsuccessful retries or continuation with invalid artifacts receive lower scores.

Final Output

Final Answer Accuracy. Measures whether the final findings agree with the validated reference within the permitted tolerances. Alternative valid workflows may produce different but reasonable values when those values are internally consistent with the executed analysis and supported by generated artifacts.

Final Answer Completeness. Measures whether every part of the user request is addressed and whether the response includes the required statistics, explanations, caveats, and artifact references. A correct but partial response receives a reduced score.

Aggregation

Planning averages Plan Correctness, Plan Completeness, and Instruction Quality. Execution averages Tool Selection, Parameter Accuracy, Code Logic, and Result Summarization. Compliance averages Prompt Compliance and Error Recovery, while Output averages Final Answer Accuracy and Final Answer Completeness. Their respective category weights are 27%, 38%, 17%, and 18%.

Diagnostic Flags

The cross-system judge records binary diagnostic flags independently of the numerical scores. These indicate whether the run completed successfully, produced valid structured output, used a forbidden source, attempted a database write, omitted a required artifact, reported an unsupported result, encountered a service failure or timeout, passed raw coordinates through an LLM, relied on language-model reasoning for a numeric geometric result, lost coordinate precision, used an appropriate CRS for metric computation, or failed to handle invalid geometry. The internal judge additionally records whether errors occurred, whether the system recovered, whether replanning occurred, whether replanning was necessary and successful, whether the run was slow but correct, and whether it contained redundant steps. These flags support failure-mode and recovery analysis but do not directly modify the weighted rubric score.

Benchmark Scope

GeoFaham-Bench focuses on realistic geospatial disaster-response workflows rather than exhaustive coverage of all GIS operations. Its 40 tasks support comparison across diverse hazards, data sources, and workflow types, but do not cover every remote-sensing sensor, humanitarian dataset, or disaster scenario.

The benchmark also relies on dynamic external sources such as OpenStreetMap and satellite catalogs. Ground truth therefore uses numeric tolerances and semantic checks where exact counts may change. This reflects the operational setting, in which robust systems must select valid data, execute sound geospatial operations, preserve artifacts, and provide evidence-backed outputs under realistic data variability.

Additional Experimental Detail

Dimension-Level Analysis

Table 6 reports the complete cross-system dimension-level results. The largest gains occur in data selection correctness and spatial–temporal logic, the two dimensions most directly connected to GeoFaham’s architecture. GeoFaham scores 0.972 on data selection correctness, improving over Codex, Claude Code, and OpenCode by 11.9, 16.0, and 15.3 points, respectively. For spatial–temporal logic, GeoFaham scores 0.918 compared with 0.779 for Codex, 0.716 for Claude Code, and 0.661 for OpenCode. Implementation correctness shows a different pattern. Codex scores 0.916 and Claude Code scores 0.934, compared with 0.922 for GeoFaham. Geometry data integrity is high across all systems, ranging from 0.956 to 0.997. These results show that the general-purpose coding agents frequently produce executable geospatial implementations, whereas GeoFaham’s strongest relative gains occur in evidence selection, preservation of spatial and temporal scope, and coordination across dependent operations. GeoFaham also obtains the highest artifact-quality score of 0.976, consistent with the intended role of the Response Store in preserving and transferring intermediate outputs.

Ablation Dimension Analysis

Table 7 shows how individual capabilities change across model-assignment configurations. Upgrading only the Planner from C1 to C2 produces the largest initial gains in the planning dimensions: Plan

Dimension	GeoFaham	Codex	Claude	OpenCode
Task Outcome
Category Score	0.897	0.794	0.738	0.622
Answer Accuracy	0.876	0.766	0.669	0.586
Answer Completeness	0.918	0.821	0.806	0.657
Geospatial Method
Category Score	0.951	0.890	0.863	0.828
Data Selection Correctness	0.972	0.853	0.812	0.819
Spatial–Temporal Logic	0.918	0.779	0.716	0.661
Geometric Operation Quality	0.944	0.909	0.871	0.863
Geometry Data Integrity	0.997	0.995	0.984	0.956
Implementation Correctness	0.922	0.916	0.934	0.843
Reproducibility
Category Score	0.982	0.974	0.902	0.865
Structured Result Evidence	0.972	0.996	0.881	0.869
Method Traceability	1.000	0.990	0.995	0.993
Artifact Quality	0.976	0.936	0.830	0.735
Compliance
Category Score	0.999	0.986	0.974	1.000
Benchmark Rule Compliance	0.999	0.986	0.974	1.000
Robustness
Category Score	0.933	0.896	0.873	0.838
Error Handling	0.977	0.904	0.973	0.880
Evidence Aware Confidence	0.890	0.889	0.774	0.796

Correctness increases from 0.861 to 0.948, Plan Completeness from 0.833 to 0.951, and Instruction Quality from 0.854 to 0.934. This pattern suggests that the C1-to-C2 improvement is primarily associated with stronger orchestration. C3 further increases Plan Correctness and Plan Completeness to 0.994 and 0.995, respectively. Execution dimensions also improve when stronger models are assigned to the Vector and Raster Agents. Tool Selection rises from 0.924 in C1 to 0.998 in C3, while Parameter Accuracy and Code Logic reach 0.942 and 0.932. These results indicate that planning establishes the workflow structure, but execution-time decisions, including SQL formulation, raster-band handling, nodata treatment, and CRS transformations, continue to depend on the capabilities of the specialist agents. Final Answer Accuracy consequently increases from 0.674 in C1 to 0.857 in C2 and 0.951 in C3, with C3 providing the strongest combination of planning and execution performance.

Recovery and Behavioral Analysis

Table 8 reports behavioral indicators that complement the Error Recovery scores in the main paper. Errors denotes runs containing at least one execution failure, while Recovered denotes runs in which the system successfully handled the observed failure. Replan identifies runs in which the Orchestrator revised the workflow, and Replan OK identifies runs in which that revision succeeded. Slow and

Config	Errors	Recovered Runs	Replan	Replan OK	Slow	Redundant
C1	61.3%	55.0%	27.5%	22.5%	13.8%	38.8%
C2	61.3%	61.3%	31.2%	28.7%	18.8%	42.5%
C3	33.8%	33.8%	3.8%	3.8%	1.2%	16.2%
C4	28.8%	28.8%	5.0%	5.0%	1.2%	17.5%

Dimension	C1	C2	C3	C4
Planning
Plan Correctness	0.861	0.948	0.994	0.992
Plan Completeness	0.833	0.951	0.995	0.988
Execution
Tool Selection	0.924	0.978	0.998	0.998
Parameter Accuracy	0.797	0.879	0.942	0.935
Code Logic	0.767	0.821	0.932	0.931
Result Summarization	0.852	0.958	0.966	0.965
Compliance
Prompt Compliance	0.811	0.909	0.952	0.953
Error Recovery	0.838	0.916	0.970	0.963
Output
Final Answer Accuracy	0.674	0.857	0.951	0.928
Final Answer Completeness	0.686	0.914	0.953	0.940

Redundant identify runs judged correct but inefficient or containing unnecessary repeated operations, respectively. C2 replans most frequently, with replanning observed in 31.2% of runs and successful replanning in 28.7% of runs. C3 and C4 replan less frequently, at 3.8% and 5.0%, respectively, which is consistent with their lower execution-error rates. C2 also has the highest slow-run rate, at 18.8%, and the highest redundant-call rate, at 42.5%. C3 and C4 reduce the slow-run rate to 1.2% and the redundant-call rate to 16.2% and 17.5%, respectively. Overall, the stronger configurations improve reliability primarily by preventing execution failures, while retaining the ability to recover when failures occur.

Configuration-Level Efficiency Analysis

Token consumption does not increase monotonically with model strength. C1 and C2 consume an average of 124,606 and 129,259 tokens per task, respectively, whereas C3 reduces average consumption to 104,146 tokens while achieving the highest overall evaluation score. C4 uses a similar average of 105,668 tokens but performs slightly below C3. The lower token consumption of C3 coincides with fewer retries, redundant calls, and replanning loops, suggesting that selective assignment of stronger models to high-impact roles can improve both effectiveness and execution efficiency.

Config	Mean	Median	Prompt	Completion
C1	124,606	80,232	122,567	2,039
C2	129,259	93,800	127,156	2,103
C3	104,146	92,844	99,606	4,540
C4	105,668	76,938	99,738	5,930

The GeoFaham Interface

GeoFaham is exposed through a browser-based interface that combines a conversational assistant with an interactive geospatial workspace. Users can submit natural-language questions or define an area of interest directly on the map, inspect intermediate and final outputs, and issue follow-up queries that reuse previously generated artifacts. The conversation panel presents textual answers, tables, and artifact notifications. The map displays vector layers, raster products, satellite imagery, and user-drawn regions, with controls for layer visibility, basemaps, raster visualization, and feature inspection. Generated outputs remain available in the active workspace and can be referenced in subsequent turns. Figure 6 and Figure 7 illustrate this artifact-centered interaction across example GeoFaham sessions.