OpenDraco: A Platform for Engineering and Experimenting with Configurable Multi-Agent Architectures for Software Engineering

Introduction

Large Language Models (LLMs) are increasingly used in agentic solutions for Software Engineering (SE) tasks, including automated program repair (APR) [1], [2]. Multi-agent systems (MAS) address these tasks by coordinating specialized agents, such as bug localizers, patch generators, and validators [3], [4]. How these agents communicate is an important design choice: different topologies determine how information flows between agents and may affect both task outcomes and computational cost [5], [6].

Developing a MAS is only part of the challenge. Researchers investigating alternative architectures also need to adjust configurations, run them on comparable tasks, evaluate their outputs, and track the results. Existing frameworks and visual development environments simplify the construction and execution of agent workflows. However, comparing MAS configurations for SE tasks requires managing benchmark instances and evaluation results. Without an integrated workflow, such experiments become harder to organize and reproduce[7].

We present OpenDraco, an open-source platform designed to support this experimentation process. OpenDraco represents agents as nodes and their communication topology as edges, while keeping prompts, tools, models, and parameters configurable. Each system is captured in an exportable, versioned JSON specification that is translated into an executable LangGraph workflow¹. Researchers can reuse agent definitions, modify their communication structure, and execute alternative configurations using local or cloud-based agents. By storing specifications and results, OpenDraco enables researchers and practitioners to reproduce experiments, compare configurations, and iteratively refine MASs.

We demonstrate this workflow through APR, integrating OpenDraco with the SWE-bench evaluation harness [2]. We evaluate a sequential baseline and seven reconstructed configurations on synthetic and real-world bugs using the same locally hosted model. On a 23-instance development subset, the baseline and hyperagent_star each resolve two issues, while the baseline requires 39.78% less wall-clock time and 47.32% fewer tokens. The baseline also resolves 12 of 77 selected SWE-bench Lite test instances. These experiments illustrate how OpenDraco can be used to execute and compare heterogeneous MAS configurations under a common evaluation setup. Additional proof-of-concept workflows illustrate its applicability beyond APR.

Image not extracted

The source code and documentation of OpenDraco, together with the replication artifacts, are available on GitHub [8] and Figshare [9]. A demonstration video of OpenDraco is available on YouTube [10].

Related Work

The organization of agents and their communication channels is an important design dimension of LLM-based MAS. Early approaches relied on predefined interaction patterns, including sequential pipelines, hierarchical structures, and ensembles. Li et al. [3], for example, investigate how aggregating multiple agents can improve task performance. More recent studies examine communication topology directly. Yang et al. [11] identify topology as an important research direction, while GPTSwarm and G-Designer investigate the optimization of agent interaction graphs [5], [6]. AFlow instead focuses on automatically generating agent workflows [12]. These approaches contribute methods for constructing or optimizing MAS designs. Our focus is complementary: providing an environment in which researchers can configure, execute, and evaluate alternative designs for SE tasks.

Several development platforms already provide mechanisms for building and inspecting agent workflows. Flowise [13], Langflow [14], and Crew Studio [15] offer visual interfaces for composing agents and their interactions. AutoGen Studio [16] also supports declarative JSON specifications, reusable components, and interactive evaluation, while LangGraph Studio [17] provides graph visualization and execution debugging. Thus, visual modeling, reusable agents, and declarative workflow specifications are not unique to OpenDraco.

OpenDraco differs primarily in its focus on comparative experimentation for SE. Rather than proposing a new topology optimization method or general-purpose workflow builder, it combines configurable agents and communication structures with versioned specifications, execution, task-specific evaluation, and result comparison. Its APR instantiation integrates reusable agents and topology templates with SWE-bench evaluation, allowing researchers to modify, reuse, and compare MAS configurations and their outcomes and costs without reimplementing orchestration logic.

OpenDraco Architecture

Image not extracted

Fig. 1 shows the architecture of OpenDraco. The framework follows an $n$-tier design. Each tier is described below.

User Workflow

Image not extracted

Fig. 2 summarizes the workflow followed by a user when configuring and executing a MAS with OpenDraco. OpenDraco can be accessed through its web interface or command line, and deployed locally through Docker. Users select or define a topology, configure each agent’s role, prompt, tools, model, and parameters, and save the configuration for reuse and versioning. They then execute it on selected task instances, evaluate the outputs, and inspect the results. Finally, they can modify or replace the configuration and rerun it on the same instances to compare MAS designs under the same setting.

Frontend Layer

OpenDraco provides an Angular-based interface. The Topology page connects to the Agentic System Modeler (Section III-C1), enabling users to define MASs as agent nodes and communication edges and configure their models, prompts, parameters, and tools. Configurations are stored as importable/exportable JSON files and Git-versioned. The Inference page executes selected instances, streams agent activity, and stores outputs. The Evaluation page runs task-specific evaluators; for APR, it uses the SWE-bench harness to apply patches and execute tests. The Results page reports outputs, patches, evaluation results, wall-clock time, token usage, passed tests, and resolution status.

Backend Layer

The backend is implemented in Python and exposes its services through a FastAPI server, which is consumed by the frontend and can also be accessed by third-party components.

Agentic System Modeler

Agentic System Modeler: This module supports the modeling of MAS and produces a structured specification (e.g., JSON) containing: a) each agent, defined by its model, prompts, parameters (e.g., temperature), and tools; and b) the topology, i.e., how the agents are connected. It further provides: i) a validation mechanism, which checks the specification against LangGraph’s constraints and detects structural anomalies (e.g., unreachable agents or the absence of a path from START to END). Cyclic flows are supported and bounded at runtime by a revisit limit; ii) an agent catalog, which organizes agents by attributes such as role to support reuse across different systems; iii) configuration history, which maintains a Git history for each configuration file, allowing changes to be tracked and previous versions to be restored; and iv) persistence, which stores specifications in the file system and loads them when needed for reuse or sharing.

Agentic System Executor

This module converts a modeled MAS, represented as a JSON specification (Section III-C1), into a LangGraph StateGraph and runs it on a given problem instance to produce a prediction (e.g., a patch for a SWE-bench instance). It comprises: i) a graph builder that creates the LangGraph representation; ii) a typed shared state for agent communication; iii) an agent runtime that calls the model with its configured parameters, prompt, tools, and current state; and iv) a workflow runner that schedules nodes and supports fan-out and conditional edges. Each agent follows a common reasoning-and-acting loop [18], alternating between LLM calls and tool invocations until it produces a final response. Outputs are normalized before being passed downstream. A shared registry exposes the tool catalog and enforces per-agent allow lists defined by the tools list.

Evaluation Runner

This module evaluates outputs produced by the Agentic System Executor, including correctness or quality against ground-truth solutions. For APR, OpenDraco provides three modes: i) the official SWE-bench harness in Docker, ii) remote execution via sb-cli, and iii) a local evaluator for repositories outside SWE-bench—including those containing our synthetic instances—that clones the repository, applies the generated patch, and runs its tests.

Additional Layers

The Storage Layer persists execution registries, predictions, reports, metrics, configurations, logs, and results as local files, enabling direct inspection and editing. The Inference Layer supports Ollama² backends running on localhost (the default) or another host, and cloud-based services through OpenAI-compatible APIs (e.g., OpenRouter).

Customization of OpenDraco for APR

We instantiate OpenDraco for APR by specializing three components: i) The agent catalog, comprising agents derived from 16 publicly available repositories identified in our previous study [19]. These repositories implement solutions reported on the SWE-bench leaderboard, and the agents are classified by role (e.g., Bug Localizer or Patch Synthesizer). The catalog includes only artifacts covered by permissive upstream licenses (MIT, BSD, or Apache-2.0) and records their provenance. ii) The evaluation pipeline, through integration with the SWE-bench harness. iii) The UI, which exposes the APR agent catalog and predefined APR topology templates, provides access to the SWE-bench evaluation, and reports APR-specific results and metrics.

Evaluation and Preliminary Results

Evaluation Goals

Our preliminary evaluation addresses four questions: whether OpenDraco can represent and execute existing agentic APR topologies end-to-end; whether the modeled systems can repair real-world bugs from SWE-bench Lite; whether the evaluation pipeline scales to a larger set of real-world instances; and how temperature affects run-to-run variability. We use synthetic bugs for the initial end-to-end assessment.

Evaluation Metrics and Setup

We use three instance sets: five synthetic APR instances spanning five difficulty tiers, the 23-instance dev subset of SWE-bench Lite, and a 77-instance subset of the SWE-bench Lite test split, restricted to instances whose gold patch is shorter than 600 characters. The synthetic set allows us to evaluate all eight configurations end-to-end before running the more expensive real-world experiments.

We evaluate eight MAS configurations that vary in their topology and/or agent definitions. Collectively, they cover four topology patterns: chain (sequential), star (via a central agent), tree (hierarchical), and hybrid (combined). First, we define a chain baseline following a conventional APR sequence by connecting a bug-localization agent, a patch-generation agent, and a validation agent.

The remaining seven configurations draw on agentic APR systems reported on the SWE-bench leaderboard. Two configurations, tree and hybrid, are inspired by the communication structures of Prometheus and AgentScope, respectively, while using our own agent definitions. The other five reconstruct systems by combining agents from the catalog according to their communication structures. To assess run-to-run variability, we execute the chain baseline five times at each temperature (0/1) on the synthetic set, using a different random seed for each run and recording the resolved rate, wall-clock time, and token usage.

Configuration	Ag. (ed.)	Wall-clock	Resolved	Total tokens
Configuration comparison: synthetic set (5 difficulty tiers)		–	+
chain (baseline)	4 (3)	29 min 41 s		226,605
tree	7 (6)	31 min 41 s		172,084
hybrid	6 (7)	35 min 21 s		254,316
experepair_star	5 (7)	41 min 18 s		197,781
hyperagent_star	5 (7)	29 min 35 s		229,188
joycode_star	5 (7)	39 min 15 s		256,908
linqxi_star	5 (7)	42 min 22 s		300,563
openhands_star	6 (9)	17 min 32 s		82,840
Real-world repair: SWE-bench Lite (23-instance dev subset)
chain (baseline)	4 (3)	4 h 37 min	2/23	2,178,309
hyperagent_star	5 (7)	7 h 40 min	2/23	4,134,877
openhands_star	6 (9)	3 h 15 min	0/23	1,348,773
Evaluation scalability: SWE-bench Lite (77-instance test subset)
chain	4 (3)	16 h 4 min	12/77	6,344,990
Run-to-run variability: chain on the Synthetic set
chain - temperature 0	4 (3)	30–32 min	3/5 (all)	206.0–206.5k
chain - temperature 1	4 (3)	25–34 min	2/5 to 4/5	198.7–276.8k

Preliminary Results

Table I reports the results. All configurations use the same model (qwen3.5:9b) to ensure a consistent comparison. On the synthetic set, openhands_star resolves all five instances and also has the shortest wall-clock time and lowest token usage among the eight configurations. On the 23-instance development subset, both chain and hyperagent_star resolve two instances. However, chain uses 39.78% less time and 47.32% fewer tokens. Given this effectiveness–cost trade-off, we select chain for the larger 77-instance test subset. It resolves 12 of the 77 instances (15.58%).

At temperature 0, chain resolves the same three synthetic instances in all five runs. At temperature 1, the number of resolved instances ranges from two to four, and both runtime and token usage vary more across runs. This variability suggests that configurations using nonzero temperature should be evaluated over repeated runs.

Discussion

Framework Extensibility

To assess OpenDraco beyond APR, we modeled and executed proof-of-concept workflows through its UI (Section III-B), including a translation workflow that reads a text file, translates it, and evaluates the output with BLEU against a reference. This required only task-specific agents, topologies, tools, and evaluators, without changes to the framework core. OpenDraco can support new tasks through configuration without framework modification.

Limitations of the Experiments

Reconstruction Fidelity: The reconstructed APR configurations approximate the original systems in terms of agents, prompts, tools, and communication structures, but not their exact runtime behavior. Thus, our results characterize their OpenDraco instantiations, and differences cannot be attributed to topology alone.

Resources: Due to the 6 GB VRAM of the GPU, all agents were executed sequentially, including independent branches. Reported wall-clock times therefore do not capture potential parallelization gains.

Model Choice: All experiments use the locally hosted qwen3.5:9b, limiting generalizability to other models. We chose an open local model to support reproducibility, keep source code local, and avoid proprietary paid APIs.

Evaluation Scope: Except for the variability study, each configuration was executed once. Moreover, only three configurations were evaluated on real-world instances, with chain alone evaluated on the larger subset, limiting conclusions about variability and configuration differences.

Conclusion and Future Work

This paper presented OpenDraco, an open-source platform for modeling, executing, and evaluating configurable LLM-based MAS for SE. OpenDraco captures agents, prompts, tools, models, parameters, and communication topology in a versioned, shareable configuration, allowing MAS experiments to be modified and reproduced without changing the framework core. A preliminary APR evaluation shows that OpenDraco supports end-to-end experimentation and comparison of configurations by effectiveness, time, and token usage. Future work will support runtime topology evolution, evaluate additional local and cloud-based models over repeated runs, and extend the agent catalog, topology templates, and evaluators to other SE tasks. We also plan a user study on modeling and reproducibility effort, and to incorporate sustainability metrics into OpenDraco.

Acknowledgements

ChatGPT was used only to improve language and readability. All authors reviewed and validated the final manuscript and take full responsibility.