Haptic Shared Control for Multi-Robot Systems

Rapporteurs avant soutenance :

Fabio MORBIDI — Maître de conférence, Université de Picardie Jules Verne

Francesco CHINELLO — Professeur associé, Université d’Aarhus

Composition du Jury :

Présidente : Maud MARCHAL — Professeure, INSA/IRISA Rennes

Examinateurs : Maud MARCHAL — Professeure, INSA/IRISA Rennes

Olivier SIMONIN — Professeur, INSA/Inria Lyon

Rapporteurs : Fabio MORBIDI — Maître de Conférence, Université de Picardie Jules Verne

Francesco CHINELLO — Professeur associé, Université d’Aarhus

Dir. de thèse : Paolo ROBUFFO GIORDANO — Directeur de recherche CNRS, IRISA/Inria Rennes

Co-encadrant : Claudio PACCHIEROTTI — Chargé de recherche CNRS, IRISA/Inria Rennes

Acknowledgement

Faire une thèse, c’est à la fois une aventure scientifique et une expérience humaine intense, et elle n’aurait clairement pas été possible sans l’aide, le soutien et la présence de beaucoup de personnes. Je tiens ici à remercier tous ceux qui ont contribué, de près ou de loin, à l’aboutissement de ce travail.

Je remercie bien sûr mes directeurs de thèse, Paolo Robuffo Giordano et Claudio Pacchierotti, pour leur accompagnement tout au long de ces années, pour leurs conseils, leur disponibilité et la confiance qu’ils m’ont accordée.

Je remercie les rapporteurs de cette thèse, Fabio Morbidi et Francesco Chinello, pour le temps consacré à la relecture attentive du manuscrit et pour leurs remarques constructives, ainsi que les membres du jury, Maud Marchal et Olivier Simonin, pour avoir accepté d’examiner ce travail et de participer à ma soutenance.

Ces années de doctorat n’auraient pas été les mêmes sans l’environnement de travail offert par l’équipe Rainbow. Merci à toutes celles et ceux qui en font partie pour l’accueil, les échanges scientifiques, mais aussi pour les moments plus informels qui ont largement contribué à rendre cette expérience agréable. Un merci tout particulier aux doctorants et post-doctorants avec qui j’ai partagé le quotidien du labo, les discussions, les galères, mais aussi les réussites et les moments de détente.

Merci à mes parents pour leur soutien et leurs encouragements, sans lesquels rien n’aurait été possible. Merci à mes frères pour leur présence, même à distance, et merci à ma tante pour son soutien indéfectible.

Enfin, un immense merci à mes amis pour leur présence, vous avez été essentiels pour garder l’équilibre, surtout dans les périodes plus difficiles. Aux amis de Rennes, vous rencontrer a été un des plus beaux cadeaux de cette aventure. Aux amis de longue date, quitter momentanément Rennes pour vous retrouver a toujours été un bol d’air et une source de motivation.

À toutes celles et ceux qui ont, d’une manière ou d’une autre, contribué à rendre ce parcours plus humain et plus agréable : merci sincèrement.

Résumé étendu en français

Contexte et problématique

Les systèmes multi-robots offrent des avantages significatifs en termes d’efficacité, de robustesse et de scalabilité pour des applications telles que les missions de recherche et sauvetage, la surveillance ou l’exploration d’environnements dangereux. Cependant, le contrôle intuitif d’une flotte de robots par un opérateur humain unique pose des défis majeurs. Les algorithmes de maintenance de connectivité décentralisée permettent de garantir la communication inter-agents tout en distribuant le contrôle, mais les travaux antérieurs reposent sur des hypothèses simplificatrices : capteurs omnidirectionnels et référentiel commun partagé par tous les robots. Cette thèse vise à lever ces limitations en proposant des méthodes de commande partagée avec retour haptique adaptées à des contraintes réalistes de capteurs embarqués.

Objectifs et contributions

Cette thèse poursuit trois objectifs principaux :

• étendre l’algorithme de maintenance de connectivité décentralisée pour intégrer des capteurs à champ de vision limité (FOV) et des mesures exprimées dans le repère propre de chaque robot,

• développer une méthode d’estimation décentralisée de l’orientation relative entre robots,

• proposer plusieurs architectures de contrôle partagé à échelle variable avec retour haptique informatif. Les contributions sont validées par des simulations réalistes et une étude utilisateur impliquant 20 participants.

Méthodologie

La méthodologie repose sur la théorie des graphes algébriques pour modéliser les interactions inter-agents. L’algorithme de maintenance de connectivité consiste en une

Résultats principaux

L’étude utilisateur compare les trois méthodes de contrôle avec et sans retour haptique dans des scénarios adaptés. Pour le contrôle d’agent unique (S), le retour haptique réduit significativement le temps de complétion et améliore l’efficacité perçue. Pour le contrôle par ancrage (P), l’efficacité perçue s’améliore avec haptique sans différence significative sur le temps. Pour le contrôle par barycentre (C), l’intuitivité est significativement améliorée et la zone explorée augmente.

Un cas d’usage intégré démontre des transitions fluides entre modes de contrôle durant une mission complexe, avec maintien de la connectivité tout au long de l’exécution. Les résultats confirment que le retour haptique améliore les performances objectives et subjectives, avec des effets variables selon l’échelle de contrôle.

Discussion et perspectives

Cette thèse lève des hypothèses simplificatrices des travaux antérieurs en considérant des contraintes réalistes de capteurs (FOV limité, mesures en repère propre) et propose une méthode d’estimation d’orientation décentralisée sans hypothèse de rigidité du graphe. Les architectures de contrôle à échelle variable offrent flexibilité et adaptabilité aux exigences des missions.

Les perspectives incluent l’extension du modèle dynamique pour considérer l’inclinaison des quadrotors affectant le FOV réel, l’intégration de mécanismes de récupération de connectivité après déconnexion temporaire, et l’ajout de capacités d’évitement d’obstacles avec occlusions visuelles. Le retour tactile multimodal pourrait compléter le retour de force, et des études à long terme exploreraient les effets d’apprentissage. Le déploiement sur équipes hétérogènes (aériens et terrestres) et l’intégration de SLAM collaboratif décentralisé pour environnements inconnus constituent des perspectives majeures. La maintenance de connectivité, associée à des interfaces haptiques intuitives, permet des missions autonomes complexes nécessitant intervention humaine ponctuelle.

Introduction

Robotics has emerged as a transformative field, revolutionizing industries, changing how humans interact with technology and opening new societal possibilities, with applications spanning industry, healthcare, agriculture, and exploration. Robots are progressively accepted as an integral part of our society and are being employed both in civil and military applications for decades. They are increasingly utilized to perform tasks that are dangerous, repetitive, or otherwise infeasible for humans. While early robotic systems were confined to structured environments, advances in sensing, computation, and actuation have enabled the deployment of robots in dynamic and unpredictable environments.

Within this broad field, multi-robot systems represent a critical area of research, offering the promise of enhanced efficiency, scalability, and robustness in task execution. Unlike single-robot systems, multi-robot systems can collaborate to achieve objectives that would be impractical or impossible for a single agent. Applications of such systems include disaster response, environmental monitoring, and cooperative transportation. The ability of multiple robots to operate autonomously while maintaining coordination poses significant challenges in control, communication, and perception, particularly in dynamic environments.

In scenarios where human expertise remains indispensable, shared control paradigms have gained traction as a means to integrate human intuition with robotic precision. Shared control frameworks distribute the decision-making responsibilities between human operators and robotic systems, allowing robots to assist or augment the operator’s actions. These frameworks are especially valuable in multi-robot systems, where the complexity of managing multiple agents can exceed the cognitive capacity of a single operator.

To further enhance the operator’s ability to interact with multi-robot systems, haptic feedback has been introduced as a critical component of shared control interfaces. By providing kinesthetic or tactile cues, haptic feedback enables operators to perceive the state of the robots and their environment intuitively. This sensory channel not only improves situational awareness but also facilitates precise and responsive control, particularly in complex and dynamic environments.

Thesis contributions

In light of these considerations, this thesis explores the intersection of multi-robot systems, shared control, and haptic feedback, with the goal of enabling efficient and intuitive remote operation of multi-robot teams. It seeks to advance the state of the art in shared control for multi-robot systems by addressing critical challenges and proposing innovative solutions. Specifically, we extend a fully decentralized connectivity maintenance scheme to address the limitations posed by body-frame measurements and the restricted field of view (FOV) of onboard sensors. While some techniques for relative orientation measurement exist, we propose a novel method for estimating relative orientations between robots using only local measurements and communication, further enhancing the system’s capabilities.

Building on these theoretical contributions, we propose several haptic-enabled control methods with varying control scales over the robot group, designed to facilitate the remote operation of quadrotor UAV teams. These methods maintain connectivity while adapting to diverse task requirements, ranging from single-agent operation to coordinated group movement.

The proposed approaches are implemented in a simulation environment where each robot operates as an individual process, emphasizing the decentralized nature of the algorithms and their closeness to real-world implementation. These methods are validated through a user study conducted across three distinct scenarios and demonstrated in a comprehensive use case, showcasing seamless transitions between control modalities to meet varying task demands.

Structure of the Thesis

This thesis is structured into three main parts. The first part (Part I) provides an introduction to robotics with a focus on multi-robot systems, covering both the theoretical foundations and practical tools necessary for their control. The second part (Part II) presents the core contributions of this work, that led to the author’s publication Bernard et al., [20] and a submission to the IEEE Transactions on Human-Machine Systems journal. Finally, the third part (Part III) provides the conclusions of the thesis, outlines potential directions for future research, and includes an appendix that offer further technical details to aid the reader’s understanding.

Below is a detailed summary of the content of each part.

Outline of Part I

This part provides a comprehensive review of the state of the art in multi-agent systems, localization, and shared control, along with the theoretical foundations of algebraic graph theory and formation control.

• Chapt. 1 introduces the fundamental concepts of multi-agent systems. It provides an overview of algebraic graph theory and briefly addresses key challenges in localization and sensing within multi-robot systems.

• Chapt. 2 reviews the state of the art in shared control strategies for multi-robot systems, with a particular focus on UAVs. The chapter introduces haptic shared control and explores essential multi-agent control techniques, including consensus and formation control, emphasizing decentralized approaches and their application to systems operating under global constraints.

Outline of Part II

This part presents the core contributions of this thesis, which are organized into two chapters.

• Chapt. 3 details the models for robot dynamics, measurements, and communication, forming the foundation for the subsequent chapters.

• Chapt. 4 presents a solution to the problem of connectivity maintenance in spite of sensing limitations like (i) limited field of view of the onboard cameras, and (ii) limited range of the onboard cameras, (iii) measurements expressed in individual body frames. The chapter also introduces a novel method for estimating relative orientations between robots using only local measurements and communication.

• Chapt. 5 presents three haptic enabled control methods for the remote operation of quadrotor UAV teams while maintaining connectivity. These methods are validated through a user study involving three specific scenarios, and are demonstrated in a final use case, where the operator transitions seamlessly between control modalities to adapt to the task requirements.

Outline of Part III

• This part presents the conclusions of the thesis, offering a summary of its key contributions. It also addresses the limitations and unresolved challenges encountered during the course of this work, proposing potential directions for future investigation and research.

Outline of Part Appendix

• Appendix A provides detailed information on the software architecture used to carry out the experiments described in Chapts. 4 and 5.

I Preliminaries and state of the art

Multi-agent systems

This chapter introduces key concepts for understanding multi-agent systems. While we first present the foundations of multi-agent systems and their applications before focusing on robotics usages, we then use the terms robot and agent interchangeably in this work. We begin by outlining the motivations and reviewing the state of the art in multi-agent systems. Next, we discuss a central aspect that enables robust and scalable coordination: decentralization. To model and analyze multi-robot systems and their interactions, we introduce graph theory. Finally, we explore localization methods for multi-robot systems, examining how they address the unique challenges of such environments. The use of graph theory to model inter-agent interactions and design decentralized control strategies is a well-established approach in the literature.

Motivations

Multi-agent systems (MAS) consist of a group of autonomous agents working collaboratively or competitively to achieve complex goals. These systems are increasingly used in domains such as robotics, distributed sensor networks, autonomous vehicles or distributed computation, where individual agents interact to achieve outcomes beyond the capacity of a single unit. In MAS, agents can be homogeneous or heterogeneous, they can have different capabilities, functions and roles. The strength of MAS lies in distributing functionality across multiple simple agents, avoiding the complexity of implementing all capabilities within a single unit, rendering the overall systems scalable, robust (to single agent failure) and flexible [34].

The design of MAS is a challenging task, as it requires addressing issues such as coordination, communication, and cooperation among agents. Each agent must not only perform its local tasks but also respond to the overall system’s objectives, requiring coordination mechanisms to ensure the balance between individual autonomy and collective behavior. Such coordination often lies on communication among agents, which is especially challenging to ensure in dynamic and unstructured environments as agents may have energy constraints, limited communication ranges and bandwidth, as well as being subject to noise and interferences. Control strategies must account for these constraints and ensure that agents can effectively communicate, coordinate, and adapt to changing conditions, while maintaining system stability and performance.

Multi-agent systems can be applied to robotics at different scales, from micro-robots to large-scale systems, and can be used in various applications such as monitoring, search and rescue, infrastructure inspection, or industrial automation [82, 86, 94]. Ferro et al., [48] explores the use of multiple microrobots for various applications such as microsurgery, drug delivery, microscale manufacturing or environmental remediation, a simulation of a medicinal application is shown in Figure 1.1a. Industrial applications, such as car manufacturing, employ multiple robots that work together to perform complex tasks such as assembly, welding, or painting, as shown in Figure 1.1b.

Another key domain for the development and application of multi-robot systems (MRS) in industry is warehouses automation. Robots can efficiently perform repetitive tasks such as handling, sorting, and transporting goods, improving productivity and reducing human labor. As illustrated in Figure 1.2, Amazon’s warehouses employ wheeled robots to transport goods and stationary robotic arms to grasp and handle packages on conveyors and shelves. Other companies, such as Otto Motors (https://ottomotors.com/), offer solutions for material handling in the industry. In such applications, robots can overcome the physical limitations of human workers, including heavy lifting, repetitive motion, and exposure to hazardous environments. However, developing robots capable of handling boxes and bags without causing damage remains a significant challenge. Moreover, controlling and coordinating multiple robots to operate collaboratively within a dynamic warehouse environment introduces additional complexity, requiring advanced algorithms and real-time communication systems.

A key interest in MAS lies in the development of systems capable of operating without a centralized control unit overseeing the entire system. These systems rely on individual agents making autonomous decisions based on local information and interactions. This decentralized approach leverages the advantages of MAS, such as scalability, robustness, and flexibility. The following section will explore this concept of decentralization, discussing how it enables efficient coordination and cooperation within MAS.

Decentralization

As discussed in Antonelli, [13], decentralized systems stand in contrast to centralized systems, where a single controller is responsible for making decisions for all agents. Decentralization plays a fundamental role in various domains such as economics, finance, politics, and society, and it is also present in many daily activities. For example, modern computing systems rely on multiprocessing, where tasks run concurrently across multithreaded, multicore processors. This distribution of tasks improves efficiency and precision, ensuring a seamless user experience, as discussed in Akhter and Roberts, [10]. Real-time route optimization applications such as Google Maps or Waze coordinate with other agents to guide drivers, optimizing routes, managing congestion and reducing travel times, see Dresner and Stone, [43].

Beyond technological applications, natural systems provide compelling evidence of decentralized behavior. Social insects, such as ants, bees, and termites, demonstrate complex collective behaviors without centralized control. These behaviors, including nest building, foraging, and bridge formation, emerge from local interactions and adaptive responses within the colony, as depicted in Figure 1.3a and analyzed in Reid et al., [100]; Sumpter, [119].

Decentralization is also prevalent in biological systems. For instance, the immune system operates through the coordinated actions of distributed cells that detect and eliminate pathogens without a central controller. In social contexts, decentralization can be observed when groups of individuals collaboratively perform tasks requiring collective effort. A notable example is the lifting and transport of a Mikoshi (portable shrine) during Osaka’s Festival of Fire and Water, where participants work together without a central directive to achieve a shared goal, as illustrated in Figure 1.3b. In such cases, communication between individuals typically occurs only when necessary, such as when maneuvering through obstacles. Similarly, agents in decentralized systems activate their sensing capabilities only at critical moments, rather than maintaining continuous awareness throughout the entire task. For instance, during the collaborative transportation of a load, individuals do not continuously observe one another. Instead, they rely on intermittent cues to adjust their actions when required, ensuring efficient coordination with minimal communication overhead. This selective interaction strategy contributes to the strength and adaptability of decentralized systems.

Robotics can significantly benefit from decentralized approaches, as they enable the development of scalable, resilient, and flexible multi-robot systems. In decentralized systems, each agent operates independently, making decisions based on local information. In contrast, a centralized robotic system relies on a single controller to oversee the entire fleet, requiring communication with each individual agent [14, 19]. This approach significantly increase both computational and communication loads. Decentralization keeps algorithmic complexity manageable regardless of system size while enhancing robustness by eliminating single points of failure. Furthermore, it reduces the information burden on individual agents as the number of agents in the system grows by limiting inter-agent interactions to local neighborhoods, promoting greater flexibility. These qualities are essential for the effective operation of large multi-robot systems.

Decentralized multi-robot control strategies have been widely studied in the literature, with applications ranging from collision avoidance [16] and coverage [22] to search and rescue missions [68] and patrolling [8].

Graph theory

As discussed earlier, multi-agent systems operate under the assumption that each agent has access only to partial information, typically gathered through local interactions with neighboring agents. Graph theory offers a powerful framework to model these interactions, with agents represented as nodes and their communication or sensing capabilities as edges. This approach is fundamental for analyzing system connectivity, information flow, and coordination, making graph theory an essential tool in designing control algorithms for multi-robot systems.

Notation

Here, we introduce the notations commonly used in robotic multi-agent systems.

• Scalar quantities: we usually represent scalar quantities by lowercase symbols such as $\omega$, $d$ or $\lambda$.

• Vectors: Elements of $\mathbb{R}^{N}$ are expressed as bold lowercase letters such as $\mathbf{v}$, $\mathbf{p}$.

• All-ones and all-zeros: $\mathbf{1}_{N}$ and $\mathbf{0}_{N}$ denote vectors of all ones and all zeros of dimension $N$.

• Matrices: matrices are represented by bold uppercase letters such as $\mathbf{A} \in\mathbb{R}^{n \times m}$, and scalar elements of matrices are denoted as uppercase letters such that $A_{ij}$ with $0 < i \le n$, $0 < j \le m$ is the element of $\mathbf{A}$ on row $i$ and column $j$.

• Hadamard product: We represent the element-wise product between two matrices, the so-called Hadamard product, with $\circ$.

• Kronecker product: We represent the Kronecker product between two matrices with $\otimes$.

• Set cardinality: $|\mathcal{V}|$ represents the cardinality of set $\mathcal{V}$.

• Matrix transpose: $\mathbf{A}^{T}$ denotes the transpose of matrix $\mathbf{A}$.

• Matrix inverse: $\mathbf{A}^{-1}$ denotes the inverse of matrix $\mathbf{A}$.

• Matrix pseudo-inverse: $\mathbf{A}^{\dagger}$ denotes the pseudo-inverse of matrix $\mathbf{A}$.

• Identity matrix: $I_N$ denotes the identity matrix of dimension $N$.

We now introduce the notation for multi-robot systems in the context of graph-based modeling:

• Agents as Vertices: Each agent is represented as a vertex $i$ in a graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, where $\mathcal{V} = \{1 \ldots N\}$ is the set of vertices (agents) in the system and $\mathcal{E} \subseteq\mathcal{V} \times\mathcal{V}$ denotes the set of edges in a graph.

• Edges as Connections: An edge $e_k := (i, j) \in\mathcal{E}$ is an ordered pair representing a direct connection from vertex $i$ to vertex $j$. The presence of an edge $e_k := (i, j)$ implies that agent $i$ can sense and communicate with agent $j$. An edge is undirected if the connection is reciprocal, i.e. $(i, j) \in\mathcal{E} \Longleftrightarrow(j, i) \in\mathcal{E}$, and directed otherwise. Throughout this thesis, we primarily focus on undirected graphs.

• Neighbor Set: For a given node $i$, the set of neighbors $\mathcal{N}_i$ is defined as the collection of vertices connected to $i$ by an edge. Formally, $\mathcal{N}_i = \{j \in\mathcal{V} \mid(i, j) \in\mathcal{E}\}$.

• Edge Weights: Each edge $(i, j)$ can be associated with a weight $w_{ij}$. Here, we consider weights ranging between 0 and 1, and symmetric, i.e. $w_{ij} = w_{ji}$, in line with the assumption of undirected graphs.

Graph Properties

Here are key concepts in graph theory relevant to multi-robot systems:

• Completeness: A graph is considered complete if every pair of vertices is connected by an undirected edge, see Figure 1.4a for an illustration.

• Paths and Connectivity: — Directed Path: A directed path exists from vertex $i$ to vertex $j$ if there is a sequence of directed edges connecting $i$ to $j$ within a directed graph. For example, in Figure 1.4b, there is a directed path from vertex 5 to vertex 4, passing through nodes 3 and 1. — Undirected Path: An undirected path exists between two vertices $i$ and $j$ if there is a sequence of undirected edges connecting $i$ to $j$ within an undirected graph. For example, in Figure 1.4c, there is a path between vertices 5 and 2 passing through node 4.

— Strong Connectivity: A directed graph is strongly connected if there is a directed path between every pair of vertices, as shown in Figure 1.4b.

— Connectivity: An undirected graph is connected if there is an undirected path between every pair of vertices, as shown in Figure 1.4c.

Algebraic Graph Theory

Graphs are not only effective for visually representing multi-agent systems, but they also offer a robust suite of mathematical tools for analysis. These algebraic tools can be employed to assess various properties within a multi-agent system such as connectivity. In the following section, we introduce some of these key algebraic concepts.

To represent and manipulate graphs using linear algebra, we encode graphs in matrix form. For a graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with $N$ vertices and $M$ edges, we define several matrices that encapsulate the structure and relationships within the graphs:

• Adjacency matrix: This $N \times N$ matrix represents the connections between vertices. Each element $A_{ij}$ is nonzero if there is an edge between vertex $i$ and vertex $j$. The adjacency matrix can be binary or weighted:

— Binary adjacency matrix: $\mathbf{A} \in\{0,1\}^{N \times N}$ is defined as

Graph theory

• Weighted adjacency matrix: $A \in\mathbb{R}^{N \times N}$ is defined as

• Degree matrix: This diagonal $N \times N$ matrix holds the degree of each vertex along the diagonal. Each diagonal element $D_{ii}$ represents the number of edges connected to vertex $i$:

This definition applies to both binary and weighted adjacency matrices.

• Incidence matrix: This $N \times M$ matrix represent the relationship between vertices and edges in a graph. Each column $B_k$ corresponds to an edge $e_k$ and contains two nonzero entries, $-1$ and $1$, indicating the start and end vertices of the edge:

For undirected graphs, the incidence matrix can be constructed by assigning arbitrary directions to each edge, even though no true directionality exists in undirected edges.

• Laplacian matrix: This $N \times N$ matrix is defined as the difference between the degree matrix and the adjacency matrix:

Alternatively, the Laplacian can also be expressed as:

where $B$ is the incidence matrix. For an undirected graph, the Laplacian matrix is positive semi-definite, meaning all its eigenvalues $\lambda_i$ are real and non-negative. The smallest eigenvalue is zero. Ordering the eigenvalues in ascending order $0 = \lambda_1 \le $\lambda_{2} \le\cdots\le\lambda_{N}$, an undirected graph $\mathcal{G}$ is connected if and only if $\lambda_{2} > 0$ [50, 51]. The eigenvalue $\lambda_{2}$ is known as the Fiedler value or the connectivity eigenvalue. Refer to Figure 1.5 for examples of graphs with different values for $\lambda_{2}$.

Localization and sensing

Sensing mechanisms are broadly categorized into two types: global sensing and local sensing.

• Global Sensing: In global sensing, each agent has access to information about the entire system, regardless of its proximity to other agents. Global sensing is often used in centralized systems, where a single controller has access to all agents’ states and can make decisions based on the complete system information. Examples of global sensing systems include GPS or centralized motion capture systems such as Vicon, OptiTrack or Qualisys. Different global sensing applications offer distinct advantages and limitations. For instance, the Global Navigation Satellite System (GNSS) provides precise positioning information, with accuracies reaching the centimeter level [64]. However, it is prone to signal loss in urban canyons or indoor environments, limiting its reliability in such settings.

Motion capture systems are renowned for their exceptional accuracy in position and orientation measurements, often achieving millimeter-level precision, as highlighted in Merriaux et al., [83]. For example, Antonini et al., [15] present a dataset where ground-truth positions were recorded using a Vicon motion capture system to support autonomous aggressive and agile flight of unmanned aerial vehicles. Similarly, Delmerico et al., [40] provides a dataset for the same application, utilizing a laser tracker device based on static close-range reference points to capture ground-truth positions. The outcomes of training a racing drone using such datasets are demonstrated in Kaufmann et al., [71], where an autonomous quadrotor competes against human pilots in a drone race, see Figure 1.6. Despite their accuracy, these systems are constrained to controlled environments, require an unobstructed line-of-sight between cameras and agents, and necessitate specific markers or tags on the agents for effective tracking.

• Local Sensing: In local sensing, each agent can only perceive information from its immediate surroundings, typically within a limited sensing radius. Local sensing is often used in decentralized systems, where agents must rely on their neighbors to gather information about the global system. Examples of local sensing systems include onboard cameras, LiDARs, or ultrasonic sensors. Local sensing systems, when compared to global sensing methods, exhibit greater robustness against signal loss and environmental disturbances as they rely on local information. This makes them particularly well-suited for operation in dynamic and unstructured environments.

For example, Kaufmann et al., [71] describes a drone race between human pilots and an autonomous drone, as shown in Figure 1.6. The autonomous drone was trained using datasets obtained through global sensing techniques but adjusts its velocity during flight using the video stream from its onboard camera, demonstrating its reliance on local sensing capabilities.

The combination of global and local sensing is considered a best practice for achieving accurate and reliable localization. Autonomous vehicles, such as self-driving cars, commonly integrate LiDARs, cameras, GNSS, and inertial measurement units (IMUs) to achieve precise localization within their environment [23, 62].

Relative sensing

In algebraic graph theory, the weights $w_{ij}$ in the weighted adjacency matrix $\mathbf{A}$, discussed in Sect. 1.3.3, are often determined by the relative configuration between agent $i$ and $j$. This relative configuration refers to geometric and positional parameters such as relative positions, orientations, and distances between agents. Consequently, relative sensing becomes a pivotal component in multi-robot systems, enabling agents to ascertain their positions and orientations relative to their neighbors. We introduce the following relative sensing measurements commonly used in multi-robot systems:

Bearing

The bearing indicates the direction of a neighboring agent relative to the local reference frame of the observing agent. It is represented as a unit vector pointing from agent $i$ to agent $j$ in the reference frame of agent $i$:

where $\mathbf{p}_{i}$ and $\mathbf{p}_{j}$ denote the positions of agents $i$ and $j$, respectively.

Bearing measurements are typically obtained using onboard cameras or other vision-based sensors. They have been widely used in tasks such as localization and formation control. For example, Mariottini et al., [80] propose a vision-based localization system that employs bearing measurements to estimate the relative positions of neighboring robots, implementing a formation control strategy. Similarly, Schiano et al., [114] explores the use of bearing rigidity to maintain graph structures in formation control.

Distance. The distance measurement provides the range between two agents and is computed as the Euclidean distance between their positions:

where $\mathbf{p}_i$ and $\mathbf{p}_j$ denote the positions of agents $i$ and $j$, respectively.

Distance measurements can be obtained using various sensors, including stereo cameras, ultrasonic sensors, LiDARs, or Ultra-Wideband radios. These measurements are essential for maintaining formations in multi-robot systems. For instance, Anderson et al., [12] present a formation control algorithm based on graph rigidity principles that relies on distance measurements to ensure that agents maintain the desired relative positions.

Relative position. The relative position measurement is the vector from agent $i$ to agent $j$ in a common reference frame $\mathcal{W}$:

where $\mathbf{p}_i$ and $\mathbf{p}_j$ denote the positions of agents $i$ and $j$, respectively. In this work, we consider bodyframe relative positions, where the reference frame is the body frame of the measuring agent. For instance, the relative position measurement from agent $i$ to agent $j$ expressed in agent $i$’s body frame is expressed as:

where $R_i^{T}$ is the rotation matrix from the global reference frame $\mathcal{W}$ to agent $i$’s body frame.

Relative position measurements are more informative than distance or bearing alone. They are often used in systems that require precise positioning for tasks such as collaborelative manipulation or tight formation control. However, they are also harder to obtain, often requiring more complex sensors.

Relative orientation

The relative orientation measurement represents the angular difference between the orientations of two agents. Specifically, the relative yaw considers the angular difference along the $Z$-axis:

where $\psi_i$ and $\psi_j$ are the yaw angles (rotations around the $Z$-axis) of agents $i$ and $j$, respectively.

In this thesis, we consider relative orientation specifically along the $Z$-axis (yaw) for both simplicity and practical relevance. This focus on yaw is common in multi-robot systems, particularly in aerial robotics, where controlling roll and pitch can be challenging. For underactuated robots, such as quadrotors with fixed cameras, adjusting roll or pitch to aim the camera typically induces undesired linear motions due to the vehicle’s dynamics. Therefore, yaw-based orientation is often prioritized for multi-robot applications [38] (Franchi, Secchi, Ryll, et al. 2012)

It is essential to emphasize that for relative measurements to hold meaning, agents must not only detect their neighbors but also identify them. Often overlooked in the literature, this identification process ensures that agents can accurately associate the relative measurements they perceive with the correct neighbors. Identification can be achieved through various methods, including unique identifiers, visual markers, or communication protocols. Walter et al., [128] address this challenge by introducing an innovative relative localization system for drones. This system utilizes blinking ultraviolet (UV) light-emitting diodes (LEDs), acting as active markers, paired with UV cameras, detecting neighbors’ LEDs, see Figure 1.7. The use of the UV spectrum provides robustness against ambient light interference, making it effective in diverse environments, including indoor and outdoor settings. By employing blinking LEDs at distinct frequencies, the system not only identifies individual drones but also determines the observed side of a sensed neighbor, facilitating accurate relative orientation measurements. This approach has been further enhanced in Horyna et al., [69] to enable short-range inter-agent communication, expanding its functionality and applicability.

In summary, relative sensing is a critical component in multi-robot systems, allowing agents to measure their positions and orientations relative to their neighbors.

Collaborative localization

Collaborative localization, also known as cooperative localization, is another fundamental challenge in multi-robot systems. In the context of this thesis, the term refers to the process of estimating the configuration of multiple agents within a shared reference frame using a sparse set of relative measurements between robots. The primary objective is to address the localization problem without depending on centralized localization system, such as a GNSS or motion capture system.

First introduced by Kurazume et al., [73], the problem of cooperative localization has been extensively studied through various approaches to robot localization, including probabilistic methods [53, 54], Kalman filtering techniques [107, 108], and observer-based frameworks [37, 114]. A critical requirement for collaborative localization algorithms is the ability to associate relative measurements with the corresponding identities of neighboring agents. For effective operation, an agent must reliably link each measurement to a specific neighbor. The sensing system described in Walter et al., [128] and introduced in the previous section achieves this capability.

Throughout this thesis, all proposed algorithms assume that the identities of neighboring agents are known. Therefore, the issue of processing anonymous measurements is beyond the scope of this work.

Conclusions

This chapter introduced key concepts in multi-agent systems (MAS), covering decentralization and graph theory as foundations for agent coordination. These tools are essential for understanding the dynamics of multi-robot systems and designing effective control strategies. We also examined localization and sensing techniques which enables agents to estimate their positions relative to one another without centralized systems. Finally, we discussed collaborative localization and the importance of identifying neighboring agents to associate relative measurements accurately. This critical challenge grants agents the ability to estimate their positions within a shared reference frame, facilitating effective multi-robot coordination and supervision by a human operator.

Graph theory can be used to encode and enforce sensing and communication limitations in multi-agent formations, allowing us to design high-level control strategies directly related to these physical limitations. By leveraging the local nature of these interactions, we can achieve decentralized applications allowing for scalability, versatility, and resilience in multi-robot systems. The next chapter will build on this by introducing formation control strategies, shared control approaches combining human and autonomous inputs, and the role of haptic feedback in enhancing human-robot interaction.

Shared Control of UAVs

In this chapter, we examine a specific scenario for multi-robot applications: the remote operation of a fleet of Unmanned Aerial Vehicles (UAVs) by a human operator. This scenario presents unique challenges due to the physical distance from the robots and the complexity of simultaneously controlling multiple agents in dynamic environments. To address these challenges, researchers have explored various methods to provide the remote operator with effective information feedback, which we discuss in the first section of this chapter. In the second section, we explore different control strategies designed to enable a single operator to oversee multiple agents, leveraging the concepts introduced in the first chapter. Finally, we present the formation control method designed to preserve global constraint that forms the foundation of this thesis: decentralized global connectivity maintenance.

The use of Unmanned Aerial Vehicles (UAVs) has demonstrated potential in a variety of applications, ranging from mountain rescue operations [118] to surveillance and humanitarian missions [Custers, 2016]. Given that these scenarios are often dynamic and unstructured, research into intuitive human-robot interfaces for effective control has become increasingly critical. These interfaces must balance between ease of control and informative feedback to allow the user to effectively execute tasks while receiving clear, intuitive information from the robots. Typically, such feedback is provided through a visual interface displaying real-time data on robot positions, states, and environmental context. However, haptic feedback systems are increasingly recognized for their potential to guide users more dynamically, as examined in Ferro et al., [49] and Aggravi et al., [5]. By integrating multimodal feedback, user awareness and control can be significantly enhanced, particularly in complex or visually demanding situations.

Telerobotics

The term telerobotics, also known as teleoperation or telemanipulation, is derived from the Greek word tele, meaning “distant” [90]. Teleoperation systems were first introduced in the 1940s to meet the demands of nuclear research, see Figure 2.1a, introduced in Goertz, [63]. These early systems allowed human operators to manipulate radioactive materials from a safe distance while providing haptic feedback from the remote environment. Initially, teleoperation systems relied on mechanical linkages, which limited their range of operation. To overcome this constraint, electrically coupled manipulators were developed in subsequent decades to support nuclear applications and enable remote handling over greater distances [126].

As nuclear-related activities declined, interest in telerobotics shifted toward other domains, including space exploration, medical applications, and underwater robotics. This transition was facilitated by advancements in computing power and the development of haptic devices such as the PHANToM device shown in 2.1b and presented in Massie, Salisbury, et al., [81]. Similarly to nuclear applications, the adoption of telerobotics in these fields was driven by the need to perform tasks in environments where human operators could not be physically present, either due to safety concerns or the high costs associated with direct human involvement.

Teleoperation of UAVs

Commercial aerial robots typically rely on proprietary remote controllers or smartphone applications. However, researchers have been exploring more intuitive control methods to enhance user experience and effectiveness. Gesture-based control has captured significant attention, with several studies employing cameras to recognize natural gestures for drone navigation [96, 112] (Ng & Sharlin 2011). Gaze control, which leverages eye-tracking technology for UAV navigation, is another promising approach, particularly beneficial for users with physical impairments [66]. In addition, brain-computer interfaces (BCIs) have been investigated, enabling UAV control through the interpretation of EEG signals [Yu et al., 2012]. Wearable interfaces, including custom-made suits, exoskeletons, and gloves, have also been developed to provide immersive control experiences [111] (Rognon, Mintchev, et al. 2018). More recently, infrared hand trackers such as the Leap Motion device have gained popularity for gesture-based control, offering precise detection of hand movements for drone navigation [30, 47, 113].

Visual Feedback

Effective remote operation of robots typically requires providing the operator with visual feedback about the surrounding environment. For single-robot operations, the most common approach is streaming video from the robot onboard cameras [Fattah et al., 2016]. However, this approach becomes quickly impractical for multi-robot applications, where it is challenging for a single user to simultaneously monitor multiple video feeds. Streaming multiple videos is also costly in terms of communication bandwidth and energy consumption.

To address these challenges, visualization techniques often leverage data fusion, aggregation, and abstraction to reconstruct the environment and track the robots’ positions and states. A common approach is the creation of a "digital twin" of the remotely operated system in a virtual environment, offering a comprehensive view of the robot’s state [17, 121]. The environment may be pre-defined or dynamically reconstructed using Simultaneous Localization and Mapping (SLAM) techniques [45, 85]. For example, Figure 2.2a shows a dense point cloud reconstruction generated by ORB-SLAM2, a widely-used SLAM algorithm. Alternatively, other methods, such as the Next-Best-View path planning algorithm, are employed to model the environment. A cluster-based approach for this algorithm is proposed in Hardouin et al., [67]. The approach discussed here aims to achieve accurate 3D reconstruction of large structures with minimal runtime by utilizing a fleet of UAVs.

For multi-robot systems, visualizing the entire group requires knowledge of each robot’s position and state within a shared reference frame. In controlled environments or simulations with motion capture systems [8] (Aggravi, Elsherif, et al. 2021), this information is readily available. However, in real-world applications, it must be estimated through sensor data and collaborative localization algorithms, as discussed in § 12.2.1.2.

Collaborative SLAM (C-SLAM) extends the SLAM framework to multi-robot systems, allowing robots to share sensor data, construct a unified map, and localize themselves within it. Both centralized [9] and decentralized [44, 75] C-SLAM approaches have been explored using various sensors, such as LiDAR, stereo, and RGB-D. Figure 2.2b presents a map and pose graph generated by a Swarm-SLAM experiment with three robots, as described in Lajoie and Beltrame, [75].

Haptic Feedback

Traditional teleoperation systems primarily relied on visual feedback to guide the operator. However, visual feedback alone can impose a high cognitive load on the operator [1] and may degrade performance when the view is partially occluded.

To address these limitations, researchers have investigated the integration of haptic feedback as a means to enhance the operator’s situational awareness and improve control accuracy [95, 129].

The broad definition of haptics refers to the human sense of touch, which relies on both kinesthetic (force and position) and cutaneous (tactile) receptors. These sensory mechanisms enable humans to perceive, interact with, and manipulate objects in their environment. Everyday tasks highlight the importance of this sense, for instance, writing, reaching for an object in the dark, or buttoning a shirt all require haptic feedback

More fundamentally, even basic motor functions would be impossible without haptic information. Humans rely on cutaneous sensations in their feet and fingertips to walk, move their limbs, or grasp objects, even in well-lit environments. Without these tactile inputs, precise movement control would be significantly impaired.

In robotics and virtual reality literature, haptics refers to the various combinations of real and simulated touch interactions between humans, robots, and their environments, whether real, remote, or virtual. These interactions are facilitated by haptic interfaces, which allow human operators to experience the sense of touch in teleoperated or simulated environments [65].

Haptic feedback systems can be broadly categorized into two types: grounded and wearable interfaces. Wearable devices, designed to be worn by the user, provide tactile feedback through mechanisms embedded in gloves, suits, or vests. Given the high density of nerve receptors in the fingers, research has focused on designing haptic devices specifically dedicated to fingertip use. Through contact simulation [87], vibration [32], pressure [31], or electric stimulation [127], these devices aim to provide precise tactile feedback, capitalizing on the sensitivity of the fingertips to enhance the user’s ability to perceive texture, pressure, and other environmental cues during interactions with remote or virtual environments. Figure 2.3 illustrates two examples of wearable haptic devices. Tactile haptic feedback is less commonly employed in drone teleoperation compared to other haptic modalities [7, 6] (Aggravi, Sirignano, et al. 2021).

To address the need for effective haptic feedback in drone teleoperation, researchers have explored various wearable devices, including cable-driven systems [106], vibrotactile armbands [28], and wearable tactile gloves [122]. These devices aim to convey information about drone trajectories, formation states, and environmental interactions, enhancing the operator’s ability to control and perceive the system’s behavior intuitively.

Compared to wearable devices, grounded haptic interfaces can deliver kinesthetic feedback, providing the human operator with information about position, velocity, forces, and torques. These interfaces have been employed in various applications, including force-feedback devices used to control fleet leaders and convey team status updates [76, 103] (Franchi, Secchi, Ryll, et al. 2012). Additionally, some researchers have explored hybrid approaches that combine haptic feedback with visual or auditory cues to further enhance the operator’s situational awareness [Aggravi et al., 2020]. This thesis specifically utilizes the Virtuose 6D device, shown in Figure 2.4, a high-precision haptic interface developed by Haption. The Virtuose 6D delivers 6 degrees of freedom (DoF) force feedback, enabling intuitive 6 DoF motion control by the operator.

As the complexity of multi-robot systems continues to grow, providing rich and intuitive feedback to human operators becomes increasingly important. These feedback mechanisms are designed to improve the operator’s understanding of the robots’ state and the surrounding environment, thereby enhancing control accuracy and mission success rates. The nature of the haptic feedback information is often determined by the type of control method employed for operating the drones and typically is linked to aspects that the user can directly influence.

Multi-agent control

Beyond addressing the challenge of a user’s physical absence from the remote environment, telerobotics is also motivated by the complementary capabilities of humans and robots. Humans contribute cognitive flexibility, high-level decision-making, and adaptability to complex tasks. In contrast, robots offer precision, strength, and motion scaling, which are critical for scenarios requiring fine manipulation. Additionally, robots provide autonomy, data aggregation, and low-level control, enabling the management of tasks involving multiple robots that would be beyond the capabilities of a single human operator. This synergy between human intelligence and robotic capabilities enhances the effectiveness and versatility of telerobotic systems.

In this context, consensus algorithms enable agents to align their states by exchanging local information, forming the basis of distributed decision-making. Formation control focuses on maintaining desired spatial configurations among agents, which is essential for tasks like exploration, transportation, or surveillance. Lastly, shared control integrates human input with autonomous decision-making, allowing an operator to guide the system while leveraging the collective intelligence of the agents while providing the user with information feedback and control assistance.

Consensus

Consensus-based algorithms, often referred to as agreement algorithms in the literature [24], originated in the field of computer science [77] and have since become foundational in multi-robot research [74]. These algorithms address the consensus problem, which aims to ensure all agents within a system converge to a common value for a specific variable, relying solely on information exchanged with a local subset of neighboring agents.

Continuous-time consensus algorithms are inherently tied to the Laplacian matrix of the communication graph. Consider a system of $N$ agents, where each agent has an internal state $\boldsymbol{p}_i \in\mathbb{R}^{m}$ and simple integrator dynamics defined as $\dot{\boldsymbol{p}}_i = \boldsymbol{v}_i$, with $\boldsymbol{v}_i$ representing the control input for agent $i$. With $\mathcal{G}$ representing the communication topology, the continuous-time consensus algorithm is expressed as:

where $L$ is the Laplacian matrix of graph $\mathcal{G}$. If $\mathcal{G}$ is connected, the system will converge to a common state, i.e. $\lim_{t\to\infty}\boldsymbol{p}_i=\lim_{t\to\infty}\boldsymbol{p}_j$ for all $i,j\in\mathcal{V}$.

For individual agents, convergence to a common state is guaranteed by applying the following control input:

with $\mathcal{N}_i$ the set of neighbors of agent $i$ and $w_{ij}$ the weight of the edge connecting agents $i$ and $j$ (i.e. the $(i,j)$-th element of the adjacency matrix $\boldsymbol{A}$).

Let us assess the stability of the consensus algorithm. We recall that for a connected undirected graph $\mathcal{G}$, the Laplacian matrix $L$ is symmetric, positive semi-definite (i.e. all its eigenvalues are non-negative), and its smallest eigenvalue is zero and corresponds to the eigenvector $\boldsymbol{1}_N$ (a vector of all ones) [25]. Let us define a Lyapunov candidate function based on the disagreement among agents:

Since $L$ is positive semi-definite, $V(\boldsymbol{p})\geq0$ for all $\boldsymbol{p}$ and $V(\boldsymbol{p})=0$ if and only if $\boldsymbol{p}$ lies in the null space of $L$, i.e. $\boldsymbol{p}=\alpha\boldsymbol{1}_N$ for some scalar $\alpha$. From this definition, we can derive the time derivative of $V(\mathbf{p})$:

Using the product rule:

Since $L$ is symmetric ($L = L^{T}$):

From the consensus algorithm, $\dot{\mathbf{p}}=-L\mathbf{p}$, so:

Since $L$ is positive semi-definite, $L^{2}$ is also positive semi-definite, so $\dot{V}(\mathbf{p}) \leq0$ for all $\mathbf{p}$, implying that $V(\mathbf{p})$ is non-increasing over time. Therefore, the disagreement among agents’ states decreases over time. Furthermore, $\dot{V}(\mathbf{p}) = 0$ if and only if $\mathbf{p}$ lies in the null space of $L$, i.e. $\mathbf{p} = \alpha\mathbf{1}_{N}$ for some scalar $\alpha$. This implies that the agents’ states converge to a common value.

From Bullo et al., [25], $\dot{V}(\mathbf{p}) < -\lambda_{2}V$, where $\lambda_{2}$ is the second smallest eigenvalue of $L$. Thus, the consensus algorithm is globally exponentially stable if $\lambda_{2} > 0$ (i.e. if the graph is connected). This proof holds for undirected graphs; proofs can be found in the literature for directed graphs, see Moreau, [84].

The consensus protocol can be applied at various levels and applied to multi-agent systems control, serving different objectives:

• Rendezvous: Agents aim to converge to a common position in space.

• Alignment: Agents adjust their orientations to achieve uniform alignment.

• Synchronization: Agents synchronize their internal clocks to operate cohesively.

• Distributed Estimation: Agents collaboratively estimate and agree upon the value of a shared parameter.

Many decentralized control and estimation laws are based on the consensus protocol. Examples include the distributed average consensus algorithm introduced in Olfati-Saber and Murray, [93], the distributed Kalman filter detailed in Olfati-Saber, [92], and the distributed optimization algorithms discussed in Nedić and Liu, [88].

Formation Control

Controlling multiple robots simultaneously presents unique challenges, and various strategies have been proposed to address these issues. Formation control typically involves imposing geometric constraints on the relative state of agents. Depending on the robots’ states and their sensing capabilities, specific constraints can be applied to achieve different behaviors. These constraints may include inter-agent distances [12, 76] or angles between agents [18, 21, 56]. This method is particularly valuable in scenarios that require precise positioning of multiple drones. The example provided in Dang et al., [36] illustrates a formation control algorithm that enables a group of autonomous robots to track a moving target while navigating through environments with obstacles. The algorithm ensures that the robots maintain various predefined formations, defined by a set of inter-agent distances and relative angles, adjusting their positions dynamically to avoid collisions, as depicted in Figure 2.5.

In contrast to rigid formation control, leader-follower strategies enable dynamic formation control by having follower agents maintain relative positions to a leader using virtual spring-like couplings. This approach supports both autonomous navigation and operator guidance, ensuring flexible and cohesive multi-agent coordination [33, 101] (Franchi, Secchi, Son, et al. 2012). Dynamic leader selection has been explored by some researchers, optimizing tracking performance by changing the leader identity in real-time [55, 57]. This approach adapts to changing environmental conditions or mission requirements.

Formation control methods has been further developed to maintain global properties such as connectivity or rigidity across the entire formation [104, 115] (Zelazo, Franchi, et al. 2015).

Shared Control

Unlike applications where robotic systems are fully teleoperated [136] or fully autonomous [130], shared control systems combine human and autonomous control to leverage the strengths of both approaches [97, 116]. In sush systems, the human operator and the autonomous control system work together to achieve a shared goal [48] (Abi-Farraj et al. 2018).

For instance, in Ferro et al., [49], a human operator controls a robotic arm to insert a needle into soft tissues. While the operator guides the arm toward the target, an autonomous system assists by ensuring a safe trajectory and providing haptic feedback to indicate the needle’s penetration depth. Similarly, Rahal et al., [98] investigates haptic feedback to reduce manipulator torque during remote manipulation tasks. The system also provides visual and haptic guidance to help the operator select an appropriate grasping pose for displacing an object, as shown in Figure 2.6.

Shared control systems are particularly advantageous in multi-robot systems, where the complexities of coordinating multiple robots can quickly overwhelm a single human operator [99, 117] (Aggravi, Elsherif, et al. 2021) In these systems, the human operator can focus on high-level decision-making, while the autonomous system assists with coordination, obstacle avoidance, or maintaining formation or connectivity [58, 123, 133].

Formation Control under Global Constraints: Connectivity Maintenance

This section addresses the connectivity maintenance challenge in multi-robot systems. Properly modeling each robot’s ability to sense and communicate with surrounding robots is fundamental to achieving autonomy in such systems. As introduced in § 12.1.3.3, algebraic graph theory offers a powerful framework for representing these interactions. In this context, an edge between two agents signifies their capability to exchange information, either through communication or sensing. Among the key properties of a graph is its connectivity, defined as the ability to traverse between any two nodes via direct or indirect paths and quantified by the connectivity eigenvalue $\lambda_2$.

In multi-robot systems, ensuring global connectivity within the robotic group is essential for enabling decentralized tasks such as consensus [92], rendezvous [41], flocking [91], leader-follower strategies [80], and other collaborative operations. Consequently, maintaining connectivity throughout the system’s operation has become a central research area, with two predominant approaches emerging:

• Conservative methods: These methods aim to preserve the initial connected graph topology by maintaining existing edges. For example, Ji and Egerstedt, 2007 considers an inter-robot sensing model based on maximum range with a fixed graph topology. in Dimarogonas and Kyriakopoulos, [42], the possibility to permanently add edges over time is included.

• Flexible methods: In contrast, flexible approaches allow the graph topology to evolve dynamically, permitting the addition and removal of edges as long as

global connectivity is maintained. For instance, Kim and Mesbahi, [72] proposed a centralized approach to achieve a specific $\lambda_2$ value by placing a set of robots with maximum-range constraints in an obstacle-free environment. De Gennaro and Jadbabaie, [39] pursued a similar objective with a decentralized solution.

This thesis focuses on flexible and decentralized strategies for solving the connectivity maintenance problem in multi-robot systems. In particular, Robuffo Giordano et al., [104] introduced a fully decentralized approach capable of handling complex sensing models, dynamically forming or dissolving inter-agent links while preserving global connectivity, and executing additional tasks beyond sole connectivity maintenance.

Notations

We now recall the definitions introduced in Sect. 1.3. Let $\mathcal{G} = (\mathcal{V},\mathcal{E})$ represent the graph associated with the multi-robot system, where $\mathcal{V}$ is the set of vertices corresponding the robots, and $\mathcal{E}$ is the set of edges representing the inter-robot communication links.

Let $\mathbf{A} \in\mathbb{R}^{N \times N}$ denote the adjacency matrix of $\mathcal{G}$, where $N = |\mathcal{V}|$ is the number of robots. The matrix $\mathbf{A}$ is symmetric and has elements $A_{ij} \ge0$, with $A_{ij} = 0$ if $(i,j) \notin\mathcal{E}$, and $A_{ij} > 0$ otherwise.

The Laplacian matrix, $\mathbf{L} \in\mathbb{R}^{N \times N}$, is defined as $\mathbf{L} = \mathbf{D} - \mathbf{A}$, where $\mathbf{D}$ is the degree matrix of the graph, a diagonal matrix with elements $D_{ii} = \sum_{j=1}^{N} A_{ij}$. Because of its symmetry and positive semi-definiteness, all $N$ eigenvalues of $\mathbf{L}$ are real and non-negative.

When the eigenvalues are ordered in ascending order, $0 \le\lambda_1 \le\lambda_2 \le\ldots\le\lambda_N$, we have $\lambda_1 = 0$, and the second smallest eigenvalue, $\lambda_2$, known as the connectivity eigenvalue or Fiedler eigenvalue, is strictly positive if and only if the graph is globally connected.

Finally, let $\mathbf{v}_i \in\mathbb{R}^{N}$ represent the normalized eigenvector corresponding to the eigenvalue $\lambda_i$ of the Laplacian matrix, i.e., a vector satisfying $\mathbf{v}_i^{T}\mathbf{v}_i = 1$ and $\lambda_i = \mathbf{v}_i^{T}\mathbf{L}\mathbf{v}_i$. The eigenvector $\mathbf{v}_2$, associated with the second smallest eigenvalue $\lambda_2$, is referred to as the “connectivity eigenvector”.

Gradient descent

The connectivity maintenance presented in Robuffo Giordano et al., [104] involves a decentralized gradient descent method applied to a cost function $V^{\lambda}(\lambda_2) \ge0$ of $\lambda_2$, which is designed to maintain $\lambda_2$ above a positive threshold. The cost function $V^{\lambda}(\lambda_2)$ is designed to have a vertical asymptote for a minimum connectivity value $\lambda_2^{\min} > 0$ and to smoothly decrease towards 0 for $\lambda_2 > \lambda_2^{\min}$ (see Figure 2.7).

From a conceptual perspective, minimizing the cost function $V^\lambda(\lambda_2(\boldsymbol{p}))$, where $\boldsymbol{p} = [\boldsymbol{p}_1^T,\boldsymbol{p}_2^T,\ldots,\boldsymbol{p}_N^T]^T$ represents the vector of positions of all agents, with $\boldsymbol{p}_i \in\mathbb{R}^3$ being the position of the $i$-th agent, can be achieved by having each agent $i$ implement a gradient-based controller:

This force will be referred to as the Generalized Connectivity Force. It is important to note that, in general, $\boldsymbol{F}_i^\lambda(\boldsymbol{p})$ depends on the state of all agents in the system. This would typically require some form of centralization for its evaluation by agent $i$. However, as we will demonstrate in the following sections, the design of $\boldsymbol{F}_i^\lambda(\boldsymbol{p})$ exhibits a decentralized structure. Specifically, its evaluation by agent $i$ can be carried out using only local and 1-hop neighbor information. This crucial property forms the foundation for the fully decentralized implementation of this approach.

It is also important to note that, while following the gradient force $\boldsymbol{F}_i^\lambda(\boldsymbol{p})$, the agents are not required to maintain a fixed topology, meaning that the edge set $\mathcal{E}$ of the interaction graph $\mathcal{G}$ does not need to remain constant. The creation and deletion of individual or multiple links, even concurrently, are fully permissible as long as the value of $\lambda_2$ is kept above the minimum threshold, $\lambda_2^{\min}$.

The stability issues arising when controlling the agents’ motion using the proposed force are analyzed and discussed in Robuffo Giordano et al., [104].

Control action

The connectivity control action for a single integrator model $\dot{p}_i = v_i$ is given by:

where $v_{2i}$ is the $i$-th component of the connectivity eigenvector $\mathbf{v}_2$, and $\mathcal{N}_i$ is the set of neighbors of agent $i$. If the weights $A_{ij}$ are properly designed so that they (and their gradients) can be evaluated from locally available quantities, then the connectivity maintenance law (15) can be implemented by robot $i$ in a decentralized way. Evaluation of (15) requires knowledge of $\lambda_2$ and of the local components of $\mathbf{v}_2$ for robot $i$ and its neighbors.

Decentralized estimation of $\mathbf{v}_2$ and $\lambda_2$

For the sake of decentralization, the estimation of $\lambda_2$ and $\mathbf{v}_2$ is performed as proposed by Yang et al., [131] and refined in Sabattini et al., [109]; Sabattini et al. [110]. This strategy relies on local estimation of $\lambda_2$, i.e. $\tilde{\lambda}_2$, and of the $i$-th component of $\mathbf{v}_2$, i.e. $\tilde{v}_{2i}$, for each robot $i$. From Sabattini et al., [109], the estimation of $\mathbf{v}_2$ is obtained through a power iteration procedure with the following update law:

where $k_1,k_2,k_3>0$ are the control gains, and $\operatorname{Ave}(\cdot)$ is the averaging operation. To implement the update law (16) in a decentralized way, the averaging operation is performed by each robot $i$ using a proportional integral average consensus estimator (PI-ACE), as presented in Freeman et al., [60].

PI-ACE

Consensus estimators allow $N$ agents, each of which measures some time-varying scalar $\alpha_i(t)$, to compute an approximation of $\tilde{\alpha}(t) = \frac{1}{N}\sum_{i=1}^{N}\alpha_i(t)$ using only local communication with connected neighbors $j \in\mathcal{N}_i$. Let $\tilde{y}_i$ be the average estimate of agent $i$, the PI-ACE estimator has the form:

where $\gamma> 0$ is the rate new information replaces old information, $\mathcal{N}_i$ is the set of neighbors of agent $i$, and $k_P, k_I$ are estimator gains.

With inputs $\alpha_{i1} = \tilde{v}_{2i}$ and $\alpha_{i2} = \tilde{v}_{2i}^{2}$, the PI average consensus estimator provides $(\tilde{y}_{i1},\tilde{y}_{i2})$ as the $i$-th agent’s estimate of $(\operatorname{Ave}\{\tilde{v}_{2i}\}, \operatorname{Ave}\{\tilde{v}_{2i}^{2}\})$. Each agent receives from communication its neighbors estimates $(\tilde{v}_{2j},\tilde{y}_{i1},\tilde{z}_{i1},\tilde{y}_{i2},\tilde{z}_{i2})$ and implements, alongside the PI-ACE, the following decentralized update law:

As detailed in Yang et al., [131], when the gain condition $k_3 > k_2\lambda_N$ is satisfied, for almost all initial condition the system 2.10 converges to an eigenvector $\tilde{\mathbf{v}}_2$ of $\mathbf{L}$ corresponding to the eigenvalue $-\lambda_2$ satisfying $\|\tilde{\mathbf{v}}_2\| = \sqrt{N\left(\frac{k_3-k_2\lambda_2}{k_3}\right)}$. Thus, each agent can compute its own estimation of $\lambda_2$ as

Conclusions

This chapter discussed key aspects of telerobotics and multi-robot systems control. Shared control systems leverage the complementary strengths of humans and robots, with humans providing cognitive flexibility and robots offering precision. Haptic feedback enhances teleoperation, guiding operators with real-time force and positional cues.

Conclusions

Decentralized connectivity maintenance in multi-robot systems was addressed through algebraic graph theory.

Together, these concepts form the foundation for effective collaboration in remote multi-robot environments, integrating human oversight, autonomous compliance with global constraints, and efficient, intuitive feedback mechanisms

II Haptic Shared Control for Multi-Robot Systems

Main modeling assumptions

In this chapter, we introduce the core modeling assumptions and definitions that form the foundation for the analysis presented in the following chapters of this thesis. The multi-robot system under consideration consists of quadrotors operating in $\mathbb{R}^{3} \times\mathbb{S}^{1}$, with their dynamics expressed in the body frame of each individual robot. The chapter first outlines the dynamic model of the quadrotors, followed by a detailed description of the measurement model used to obtain relative position information. Finally, we define the sensing and communication model that governs the interaction between agents in the system.

Agent model

We consider a group of $N$ robots, focusing specifically on quadrotor UAVs due to their extensive use and popularity within the robotics community. Nonetheless, the proposed methodology remains general and can be adapted to other types of mobile robots with minimal adjustments to account for differences in locomotion and sensing capabilities.

No global common frame is assumed directly available to the robots since they can only obtain relative measurements w.r.t. each other and their velocity commands are in the robot body frames. A common frame must, however, be implicitly or explicitly selected for expressing the simulated robot poses. Further details on the simulation can be found in the appendix.

Robots are not assumed to have direct access to a global common reference frame; instead, they rely solely on relative measurements with respect to one another. Their velocity commands are expressed in their respective body frames. However, for simulation purposes, it is necessary to define a global fixed frame to express the robots’ poses consistently across the system.

In this thesis, we consider a world frame $\mathcal{W} : \{O_{\mathcal{W}}, X_{\mathcal{W}}, Y_{\mathcal{W}}, Z_{\mathcal{W}}\}$ where $Z_{\mathcal{W}}$ is aligned with the vertical (gravity) direction. The specific details of how the simulation framework handles this reference frame are provided in the appendix.

Similarly to Franchi, Masone, et al., 2012; Schiano and Robuffo Giordano, [115]; Schiano et al., [114] and other related works, we consider a group of $N$ “velocity-controlled” quadrotor UAVs with the following kinematic model

where $\mathbf{p}_i \in\mathbb{R}^{3}$ is the robot 3D position in $\mathcal{W}$, $\psi_i \in S^{1}$ the yaw angle about $Z_{\mathcal{W}}$ with $R_i = R_z(\psi_i) \in SO(3)$ the associated rotation matrix, and $\mathbf{v}_i \in\mathbb{R}^{3}$ and $\omega_i \in\mathbb{R}$ represents the body-frame linear velocity and yaw rate, which are assumed to be known and controllable.

Measurement model

The robots are assumed to be equipped with two kinds of sensors: a camera-like sensor with limited FOV and an omnidirectional distance sensor (Figure 3.1). The camera is used for detecting other UAVs in the group. In particular, when robot $j$ is in visibility of robot $i$, we assume that robot $i$ can obtain from its onboard camera a measurement of

that is, the relative position of $j$ w.r.t. $i$ expressed in the body-frame of $i$. The distance sensor is instead used for detecting any “obstacle point” $\mathbf{p}_{\mathrm{obs},i}$ within a given sensing range, including other robots as well.

Contrarily to most of the existing literature on connectivity-maintenance algorithms (as a small selection, see Aggravi et al., 2020; Capelli et al., 2021; Gasparri et al., 2017; Robuffo Giordano et al., 2013), we note the following facts: we do not assume the presence of a common frame shared by all robots (as in all previous work) where all the measurements and commands are expressed. This is evident from model (21) (body-frame input commands) and the measurement (22) (body-frame relative position instead of a world-frame relative position). Furthermore, in this work, the onboard camera is modeled with a limited FOV: among others, this prevents the use of the camera for obstacle avoidance whenever an obstacle (or another UAV) is outside the FOV. This is why a second (typically much simpler) omnidirectional distance sensor is also assumed onboard the robots. This distance sensor is assumed able to only detect “non-free space” within a sensing range, but with no ability to discern actual obstacles from other robots (that are then just treated as obstacles for collision avoidance purposes, see Sect. 4.1.4). The limited camera FOV also differentiates this work from the existing literature on connectivity maintenance, which instead simply assumes presence of an omnidirectional sensor able to measure relative positions w.r.t. robots and obstacles in all directions.

Communication model

Let $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ be an undirected graph with $\mathcal{V} = \{1 \ldots N\}$ being the vertex set and $\mathcal{E} \subseteq\mathcal{V} \times\mathcal{V}$ the edge set. We also let $\mathcal{N}_i = \{j \in\mathcal{V} \mid(i,j) \in\mathcal{E}\}$ be the set of neighbors of robot $i$. Presence of an edge $e_k = (i,j)$ in $\mathcal{E}$ models the possibility of robots $i$ and $j$ to sense and communicate among themselves. As usual, this is obtained by associating each edge with a state-dependent weight $A_{ij} \ge0$ such that $A_{ij} > 0$ if the robots can sense/communicate and $A_{ij} = 0$ otherwise. The adjacency matrix $\mathbf{A} \in\mathbb{R}^{N \times N}$ associated to $\mathcal{G}$ is then built out of the weights $A_{ij}$.

In the context of connectivity-maintenance algorithms, in general, we consider an undirected graph with $A_{ij} = A_{ji}$ and, thus, a symmetric adjacency matrix $\mathbf{A} = \mathbf{A}^{T}$. This is straightforward when assuming omnidirectional (and equal) sensing capabilities for all robots as in, e.g., Aggravi et al., [5]; Capelli et al., [29]; Gasparri et al., [61]; Robuffo Giordano et al., [104] and similar works. In our context, however, presence of individual (and not aligned) body-frames and of the limited camera FOV would in general result in non-symmetric weights: if $A_{ij} > 0$ (robot $i$ measures robot $j$), $A_{ji}$ could be 0 (robot $j$ does not measure $i$ because of the limited FOV). In the next Chapter we discuss how we address this issue for recovering in all cases the symmetry of the weights $A_{ij}$.

We are now prepared to present the core contributions of this thesis, organized into two chapters:

• Chapt. 4 presents a solution to the problem of connectivity maintenance in spite of sensing limitations like (i) limited field of view of the onboard cameras, and (ii) limited range of the onboard cameras, (iii) measurements expressed in individual body frames. The chapter also introduces a novel method for estimating relative orientations between robots using only local measurements and communication.

• Chapt. 5 presents three haptic enabled control methods for the remote operation of quadrotor UAV teams while maintaining connectivity. These methods are validated through a user study involving three specific scenarios, and are demonstrated in a final use case, where the operator transitions seamlessly between control modalities to adapt to the task requirements.

Decentralized Connectivity Maintenance with Field of View Constraints

In this chapter we consider the problem of controlling a formation of quadrotor UAVs equipped with onboard cameras able to measure relative position in their local body frames with respect to neighboring UAVs. Distributed connectivity-maintenance algorithms such as Robuffo Giordano et al., [104] provide a flexible yet intuitive solution for enabling a human user the control of a multi-robot systems while guaranteeing inter-agent communication throughout the considered mission. Connectivity maintenance has been successfully employed in applications of cooperative surveillance [8], search-and-rescue [4], and guidance [6] using teams of multiple drones controlled by a human operator. The maintenance of the team connectivity is enforced in presence of a series of sensing and interaction constraints, aiming at keeping the agents safe and able to communicate with the rest of the team. However, despite their success and broad applicability, the vast majority of previous works on connectivity maintenance makes several strong simplifying assumptions. about the robot sensing capabilities and shared knowledge. For instance, they typically assume presence of an omnidirectional sensor for retrieving relative positions of neighboring robots, which are also assumed expressed in a single common frame shared by the group. These assumptions simplify the control design but are not very realistic, since most onboard sensors have a limited field of view (cameras being a prominent example) and, in addition, every robot collects measurements and expresses command actions in its own body-frame, which in general does not need to be aligned with the body frames of the other robots.

The goal of this chapter is to then propose a decentralized connectivity-maintenance algorithm able to overcome the two aforementioned issues and explicitly take into account presence of a limited FOV and lack of a common shared frame among the robots. For implementing the connectivity-maintenance action under these new constraints, we also develop a novel decentralized estimator of the relative orientation among neighboring robots, which is needed by the proposed scheme. Finally, we validate the proposed approach in a realistic simulation scenario employing a team of 6 quadrotors UAVs.

This work has been published in Bernard et al., [20].

The rest of the chapter is structured as follows. § 14.2.1 details the proposed decentralized connectivity maintenance strategy by taking into account the FOV constraints. Section 4.2 illustrates the decentralized estimator of the relative orientation among the robot body-frames. Section 4.3 then validates the approach in a realistic simulation involving 6 quadrotor UAVS, and § 14.4.1 finally concludes the chapter.

Decentralized connectivity maintenance

We briefly recall the connectivity-maintenance algorithm presented in Robuffo Giordano et al., [104] and introduced in Sect. 2.3 and then discuss its extension to the setting considered in this work. The connectivity maintenance presented in Robuffo Giordano et al., [104] consists of a decentralized gradient descent of a cost function $V^{\lambda}(\lambda_{2}) \ge0$ of the so-called connectivity eigenvalue $\lambda_{2}$, the second smallest eigenvalue of the Laplacian matrix $\mathbf{L} = \operatorname{diag}(\delta_{i}) - \mathbf{A}$ associated to the graph, where $\delta_{i} = \sum_{j=1}^{N} A_{ij}$. The cost function $V^{\lambda}(\lambda_{2})$ is designed to have a vertical asymptote for a minimum connectivity value $\lambda_{2}^{\min}$ and to smoothly decrease towards 0 for $\lambda_{2} > \lambda_{2}^{\min}$ (see Figure 2.7). Assume that the edge weights $A_{ij}$ are smooth enough functions of the state with some (mild) locality properties (in short, a weight $A_{ij}$ may only depend on the state of robot $i$ and of its neighbors).

The connectivity control action for a single integrator model $\dot{x}_i = u_i$ is the gradient descent

where $v_{2i}$ is the $i$-th component of the eigenvector $\mathbf{v}_2$ associated to the eigenvalue $\lambda_2$. Evaluation of (23) requires knowledge of $\lambda_2$ and of the local components of $\mathbf{v}_2$ (of $i$ and its neighbors). These can be estimated online in a decentralized way as explained in Robuffo Giordano et al., [104]. If the weights $A_{ij}$ are properly designed so that they (and their gradients) can be evaluated from locally available quantities, then the law (23) can be implemented by robot $i$ in a decentralized way.

The robot model (21) considered in this work is, however, not a simple single integrator. Therefore, the gradient descent (23) needs to be adjusted accordingly. To this end, let us consider the time derivative of the cost $V^\lambda$:

where we used (21). By inspection, the $i$-th robot body-frame velocity commands for implementing a gradient descent of $V^\lambda$ are then

The upcoming sections detail the design of the interaction weights $A_{ij}$ based on the chosen sensing model and assess the feasibility of implementing the gradient control law (25) using only the quantities assumed available to robot $i$.

Design of the weights

As discussed in Sect. 3.1, we consider the presence of an onboard camera with limited FOV and of a (simpler) omnidirectional distance sensor. The camera has a minimum and maximum detection range, while the distance sensor has only a maximum range value to consider. For simplicity, we assume that the maximum range of the camera equals the maximum communication range of the robots. The weight $A_{ij}$ is then designed as the product of two sub-weights

where $\gamma_{ij}$ accounts for the min/max detection range and $f_{ij}$ for the camera FOV. Similarly to Aggravi et al., [5], let us introduce two (classical) rising/falling activation functions

and

with $a = \pi/(x^D-x^d)$ and $b=-ax^d$, and lower/upper thresholds set at $x^d$ and $x^D$ (see Figure 4.1 for an example).

Weight $\gamma_{ij}$ is then defined as

where $d_{ij} = \lVert\mathbf{p}_{ij} \rVert$ is the distance among the robots and $d_{min} < D_1 < D_2 < d_{max}$ two suitable thresholds representing the min/max detection ranges.

As for the FOV weight, we define $\boldsymbol{\beta}_{ij} = \frac{\mathbf{p}_{ij}}{d_{ij}}$ as the (unit-norm) bearing vector from $i$ to $j$ in the body-frame of robot $i$ (note that $\boldsymbol{\beta}_{ij}$ can be directly computed from the measured $\mathbf{p}_{ij}$). Let also $\mathbf{o}_C$ be the fixed body-frame direction of the camera optical axis. The quantity $a_{ij} = \mathbf{o}_C^T\boldsymbol{\beta}_{ij}$ (the cosine of the angle between $\boldsymbol{\beta}_{ij}$ and $\mathbf{o}_C$) can therefore be leveraged for defining the FOV weight. Towards this end, we start by defining the function

where $a_{max}$ is the cosine of the FOV limit (expressed as a maximum angular displacement from $\mathbf{o}_C$). Although tempting, function $f_{ij}^{*}$ cannot be directly used as the sought FOV weight, as it is not symmetric, i.e., in general, $f_{ij}^{*} \ne f_{ji}^{*}$. We then propose to define the FOV weight as

This definition has two advantages: first of all, $f_{ij}$ is now symmetric, as it can be easily verified. Furthermore, $f_{ij} = 1$ whenever $f_{ij}^{*} = 1$ or $f_{ji}^{*} = 1$ (or both), and $f_{ij} \to0$ only when both $f_{ij}^{*} \to0$ and $f_{ji}^{*} \to0$. In other words, the FOV weight $f_{ij}$ will not force robots $i$ and $j$ to simultaneously look at each other; instead, it will only enforce that at least one of the robots keeps the other one in sight. This is, in our opinion, an important feature for not overconstraining the robot group during motion.

Weight gradient

We detail here the computation of the weight gradients. From (26), one clearly has

and

since $\gamma_{ij}$ is only a function of the distance $d_{ij}$, it follows that

and $\partial\gamma_{ij}/\partial\psi_i = 0$.

As for $\partial f_{ij}/\partial p_i$, from (31) one has

and analogously for $\partial f_{ij}/\partial\psi_i$.

Recalling the definition (30) and that $a_{ij} = {}^{o}_{C}\boldsymbol{\beta}_{ij}$, one obtains

and

where $\boldsymbol{P}_{ij} = \boldsymbol{I}_3 - \boldsymbol{\beta}_{ij}\boldsymbol{\beta}_{ij}^{T}$. Further details about this derivation can be found in Schiano et al., [114]; Zelazo, Robuffo Giordano, and Franchi (2015). Similar considerations also yield

and $\partial f_{ji}^{\ast}/\partial\psi_i = 0$ with $\boldsymbol{S} = [[0\ 0\ 1]^{T}]_{\times}$ and $[\cdot]_{\times}$ being the usual skew-symmetric matrix operator. Therefore, we can conclude that the structure of the gradient of weights $A_{ij}$ (26) needed for implementing (25) is as follows

and

where ${}^{i}\boldsymbol{R}_{j} = \boldsymbol{R}_{i}^{T}\boldsymbol{R}_{j} = \boldsymbol{R}_{z}(\psi_j-\psi_i)$ depends on the relative rotation among the body frames of robot $j$ and $i$.

Discussion

We now discuss the maintenance controller (25)–(4.17–4.18) in terms of the quantities actually needed by robot $i$ for implementing it. From inspection of the various terms, one can conclude that evaluation of (39) requires knowledge of $d_{ij}$, $\beta_{ij}$, the relative angle $\psi_{ij} = \psi_j - \psi_i$ (for evaluating ${}^{i}R_j$), and $\beta_{ji} = -{}^{i}R_j\beta_{ij}$. Evaluation of (40) requires again knowledge of $d_{ij}$, $\beta_{ij}$, and $\beta_{ji}$. Sect. 4.2 will discuss how an estimation of the relative angle $\psi_{ij}$ can be maintained by robot $i$ during motion.

We then distinguish the following cases:

1. robot $i$ can measure robot $j$: in this case, robot $i$ has access to all the needed quantities, since $d_{ij}$, $\beta_{ij}$ and $\beta_{ji}$ can be obtained from the measured $\mathbf{p}_{ij}$ and the estimated angle $\psi_{ij}$;

2. robot $i$ cannot measure robot $j$ but it is measured by robot $j$: in this case, robot $j$ needs to communicate to robot $i$ the measured $\mathbf{p}_{ji}$ and estimated $\psi_{ji}$ from which robot $i$ can again compute all the needed quantities;

3. robot $i$ and robot $j$ do not measure themselves: in this case the terms $\partial A_{ij}/\partial\mathbf{p}_i = 0$ and $\partial A_{ij}/\partial\psi_i = 0$ (and alike for robot $j$).

Concerning the symmetry of weights $A_{ij}$ in (26), we note that the weight $f_{ij} = f_{ji}$ is symmetric by construction, thanks to the definition (31), and that $\gamma_{ij} = \gamma_{ji}$ since $d_{ij} = d_{ji}$. The only remark is that, in the case 2. listed above, robot $i$ still needs to compute the weight $\gamma_{ij}$, even if robot $j$ is not in visibility, so as to ensure symmetry of the total weight $A_{ij}$. This can of course be done using the quantity $\mathbf{p}_{ji}$ received from robot $j$.

In summary, the weights $A_{ij}$ can be made symmetric and the maintenance controller (25)–(4.17–4.18) can be implemented by the robots in the group with the requirement that, if a robot $i$ measures another robot $j$, then it needs to communicate its measured relative position $\mathbf{p}_{ij}$ and current estimation of $\psi_{ij}$ to robot $j$. Since this amount of information is constant per neighbor, we can conclude that the overall maintenance scheme is decentralized and scalable.

Collision Avoidance

Collision avoidance with obstacles or other robots is dealt with in a simple yet practical way. Let $\mathbf{o}_i^k$ be the position of a $k$-th obstacle point in the frame of robot $i$, detected by the onboard sensor. For each obstacle point inside the sensing range $d_{max}$, we define a repulsive velocity command $\mathbf{v}_{i,obs}^k = -k_{obs}(\lVert\mathbf{o}_i^k\rVert)\mathbf{o}_i^k$ where the gain $k_{obs}(\lVert\mathbf{o}_i^k\rVert) \ge0$ is any monotonic function with a vertical asymptote at $\lVert\mathbf{o}_i^k\rVert= d_{min}^{obs}$ and vanishing (with vanishing slope) for $\lVert\mathbf{o}_i^k\rVert\to d_{max}^{obs}$, where $0 < d_{min}^{obs} < d_{max}^{obs}$ is a safety distance. The $i$-th robot velocity command $\mathbf{v}_i$ from (25) is then augmented with the additional term $\mathbf{v}_{i,obs} =$ $\sum_k v_{i,\mathrm{obs}}^k$ for all detected obstacle points $o_i^k$. This simple reactive strategy is sufficient to handle collision avoidance in most practical situations. Stability of the coupling between connectivity maintenance and obstacle repulsion can be shown by following the same arguments as in Lee et al., [76]: in short, for any bounded external action (in our case, a bounded external command from the human operator), the closed-loop system is stable besides practically improbable (zero-measure) situations of perfect (and unlikely) alignments among the robots.

Estimation of the relative orientation

As explained in the previous section, for implementing the maintenance controller, each robot $i$ needs to know the relative orientation $\psi_{ij} = \psi_j - \psi_i$ w.r.t. the other robots it is interacting with. This is a typical requirement common to previous works sharing the same robot model (21), see, e.g., Franchi, Masone, et al., 2012; Schiano and Robuffo Giordano, [115]; Schiano et al., [114] and references therein. In general, in these prior works the problem of obtaining $\psi_{ij}$ is addressed by assuming special structures of the interaction graph $\mathcal{G}$. in Franchi, Masone, et al., 2012, any interacting pair of robots is always assumed in mutual visibility so that both $\beta_{ij}$ and $\beta_{ji}$ are available. This, as well-known, allows for an algebraic computation of $\psi_{ij}$. in Schiano and Robuffo Giordano, [115]; Schiano et al., [114], constant mutual visibility is not required, but the graph is assumed to be (bearing) rigid. The rigidity condition allows implementing a simple decentralized localization algorithm that allows estimating $\psi_{ij}$ from the measured bearings. These solutions however cannot be used in the setting considered in this work as we do not require mutual visibility nor maintenance of graph rigidity (which is a much stronger requirement than just graph connectivity). Therefore, we detail here an alternative (and, to our knowledge, novel) scheme for retrieving an estimation of $\psi_{ij}$.

Estimation of the relative orientation

Note that we do not (purposely) assume the relative yaw angle $\psi_{ij} = \psi_j - \psi_i$ to be directly measurable from the onboard camera: indeed, while extracting a relative position is (relatively) feasible, obtaining a relative orientation information from vision is in practice much harder because of the typical rotational symmetries of the UAVs. Therefore we consider $\boldsymbol{p}_{ij}$ as the only information that can be reliably extracted from the images. Also, while from the onboard IMUs one could obtain a (noisy) measurement of $\dot{\psi}_{ij}$, a plain forward integration of $\dot{\psi}_{ij}$ would clearly quickly drift over time. These considerations then motivate the estimation algorithm described hereafter.

Our initial approach involves observing the velocity of a neighboring robot $j$ in the body-frame of robot $i$ and comparing it to the velocity of robot $j$ in its own body frame. As illustrated in Figure 4.2, the relative orientation is determined as the angle between these two velocity vectors when projected onto the $XY$-plane.

We start by considering the dynamics of the measurement (22)

We note that in (41) the only unknown term is ${}^{i}R_jv_j$, since $v_i$, $\omega_i$, and $p_{ij}$ are assumed available to robot $i$. This motivates to consider the use of a “disturbance observer” for estimating the unknown ${}^{i}\mathbf{R}_{j}\mathbf{v}_{j}$, seen as a disturbance acting on the dynamics of $\mathbf{p}_{ij}$. In particular, let $\hat{\mathbf{p}}_{ij}$ be an estimation of $\mathbf{p}_{ij}$ maintained by robot $i$ and updated with the following law

with $k_{1}>0$. Let us also define the quantity

which, using (4.19–4.20), has dynamics

In Laplace domain, this is equivalent to

Therefore, the quantity $\mathbf{e}(t)$ (which can be evaluated by robot $i$) acts as a first-order low-pass filtered version of the unknown ${}^{i}\mathbf{R}_{j}\mathbf{v}_{j}$ with a cut-off frequency given by gain $k_{1}$. Robot $i$ can then (algebraically) obtain an estimation $\hat{\psi}_{ij}$ by computing the angle around the vertical axis among the vectors $\mathbf{e}$ (approximating ${}^{i}\mathbf{R}_{j}\mathbf{v}_{j}$) and $\mathbf{v}_{j}$ (communicated by robot $j$).

This estimation strategy works reasonably well, but it has some shortcomings. First of all, the estimated $\hat{\psi}_{ij}$ is reliable only if $\mathbf{v}_{j}\ne0$ and if $\mathbf{v}_{j}$ is not aligned with the vertical direction (since, in this case, ${}^{i}\mathbf{R}_{j}\mathbf{v}_{j}=\mathbf{v}_{j}$). This latter situation is, in practice, very rare, but the former situation ($\mathbf{v}_{j}\approx0$) can frequently occur during the group motion. Second, the estimation dynamics (42) does not exploit knowledge of the angular velocity $\omega_{j}$, which is instead an available quantity. Third, when robots $i$ and $j$ are in mutual visibility, the relative angle $\psi_{ij}$ can be algebraically computed from the two measured relative positions $\mathbf{p}_{ij}$ and $\mathbf{p}_{ji}$, see, e.g., Franchi, Masone, et al., 2012. The possible availability of a direct measurement of $\psi_{ij}$, although possibly noisy, is also not exploited by (42).

To address these points, we propose an improved estimation strategy based on the previous one. Let $\tilde{\psi}_{ij}$ be the state of the second estimator maintained by robot $i$. Since $\dot{\psi}_{ij} = \omega_j - \omega_i$, we choose the following estimation dynamics

where $k_2(\lVert v_j\rVert) = \bar{k}_2 w^0_1(\lVert v_j\rVert, v_{\min}, v_{\max})$, $0 < v_{\min} < v_{\max}$ are lower/upper threshold for the norm of $v_j$ and $\bar{k}_2 > 0$ and $k_3 > 0$ are estimation gains. The rationale behind (46) is as follows: if robots $i$ and $j$ are in mutual visibility, then $f^*_{ij}f^*_{ji} = 1$ and (46) reduces to

i.e., a forward odometry propagation with a correction term on the “measured” (but possibly noisy) relative angle $\psi_{ij}$ (the use of the $\sin(\cdot)$ allows dealing with possible jumps of $\pm2\pi$ in $\psi_{ij}$). Note that, in this case, both robots $i$ and $j$ will implement their own instance of (47): they can then exchange their estimated $\tilde{\psi}_{ij}$ and $\tilde{\psi}_{ji}$ for obtaining an average of the two values (exploiting the fact that $\psi_{ij} = -\psi_{ji}$).

If, instead, the two robots are not in mutual visibility but robot $i$ can measure robot $j$ (the other case being symmetrical), then $f^*_{ij}f^*_{ji} = 0$ and (46) reduces to

which is again a forward odometry propagation but now using the estimated $\hat{\psi}_{ij}$ in the correction term. Since, as explained above, the reliability of the estimated $\hat{\psi}_{ij}$ decreases with the norm of $v_j$, the gain $k_2$ is designed so as to vanish for a $\lVert v_j\rVert\to v_{\min}$ so that, when $\lVert v_j\rVert$ is too small, (46) further reduces to $\dot{\tilde{\psi}}_{ij} = \omega_j - \omega_i$.

Robot $i$ can then communicate $\tilde{\psi}_{ji} = -\tilde{\psi}_{ij}$ to robot $j$, who needs this information for implementing the maintenance controller. Finally, in any intermediate case, i.e., when $0 < f^*_{ij}f^*_{ji} < 1$, the estimator (46) will blend the two cases in a continuous way.

Simulation results

This section presents the results of a simulation in which $N = 6$ UAVs were controlled by the connectivity-maintenance algorithm and obstacle avoidance actions discussed in the previous sections.¹ We implemented the proposed control and estimation algorithms using decentralized ROS 2 instances, a custom Unity scene to visualize the UAV behavior, and robotpkg from LAAS’ Openrobots project [78] to simulate their dynamics². The parameters used in the algorithm are reported in Table 4.1.

$d_{min}=1\ m$	$a_{min}=0.4$	$d^{obs}_{min}=1.0\ m$	$k_1=10$
$D_1=3\ m$	$a_{max}=0.95$	$d^{obs}_{max}=3.5\ m$	$\bar{k}_2=10$
$D_2=7\ m$	$\mathbf{o}_C=\frac{[1,1,0]}{\sqrt{2}}$	$\lambda^{min}_2=0.01$	$k_3=10$
$d_{max}=9\ m$		$\lambda^{max}_2=3.01$

A human operator was asked to control the multi-robot team in a 3-dimensional space to reach a predefined target, employing the “leader-follower” paradigm presented in Robuffo Giordano et al., [104]. Using an analog joystick, the human operator imparts an additional input velocity to one robot (the leader) for controlling its motion, see Figure 4.3. As all the robots are subject to the connectivity-maintenance action of (23), the team follows the movement of the leader so as to keep the team connected. Since the external human input is bounded, the connectivity maintenance action will always be able to overcome it and prevent disconnection from the group. A discussion on the effectiveness of this approach in controlling multi-robot teams is presented in Aggravi et al., [5].

The task begins with the team in a sparse random configuration. At the very beginning, the connectivity-maintenance algorithm makes the robots reaching a (local) maximum for the group connectivity by creating new links whenever two robots start sensing each other. Then, as the human user imparts external commands to the robot leader $i=1$ to reach the target, the algorithm keeps adjusting the robot formation so as to satisfy the constraints described in Sect. 4.1, including the newly-introduced FOV weight.

The video available at https://youtu.be/CdnAmIrdBQU shows various details of our approach. Trajectories of the experiment present at 43 seconds in the online video (https://youtu.be/CdnAmIrdBQU?t=43) are shown in Figure 4.4a and Figure 4.4b.

Figure 4.5 (top) reports the behavior of the connectivity eigenvalue $\lambda_2(t)$ (solid red

Simulation results

line) and the $N$ estimations $\tilde{\lambda}^{i}_{2}$ maintained by the $N$ robots during motion. One can note how the estimation error remains quite small despite the (sometimes) rapid changes in $\lambda_{2}(t)$ also due to the action of the human operator (green phases). Figure 4.5 (bottom) reports the number of edges during motion which is not constant, as expected.

Figure 4.6 reports the error in estimating the relative orientation among robot 1 and robot 2 and 3. For the reader’s convenience, this estimation error is quantified by the quantity $\sin((\psi_{1j}-\tilde{\psi}_{1j})/2)$, which vanishes for $\psi_{1j}-\tilde{\psi}_{1j}=\pm2k\pi$. The plot also shows, with colored bands, the four possible visibility combinations among robot 1 and the other robots (mutual visibility, one robot measuring the other but not being measured by it, the two robots not in visibility of each other). We can note that the estimation error is always limited. Moreover, when the two robots are in mutual visibility (green phase in Figure 4.6) the error is close to zero as, in this case, a direct measurement of $\psi_{1j}$ is available. When the two robots are instead completely disconnected (white phases), no measurement is available. In this case, the estimations of $\psi_{1j}$ are not updated anymore, thus leading to an (unavoidable) increased estimation error, which is however quickly recovered when one of the robots (or both) become visible again (red dashed line). Estimation errors during blue and yellow phases can also temporarily increase when $\omega_i \ne0$ or $\omega_j \ne0$ but $v_j=0$ (blue phase) or $v_i=0$ (yellow phases) (see Sect. 4.2).

Simulation results

Conclusions

We presented a decentralized connectivity-maintenance control algorithm for a team of quadrotor UAVs having limited FOV and not sharing a common reference frame for collectively expressing measurements and commands. To the best of our knowledge, these contributions advanced the existing literature on this subject which, instead, has been based on the (simplifying) assumptions of omnidirectional sensing and presence of a common frame for all the robots in the group. The proposed maintenance control action is therefore able to address a more realistic setting, closer to real-world conditions. The approach has been successfully validated in realistic simulations involving 6 quadrotor UAVs.

A limitation of the proposed algorithm lies in the model (21), which assumes a simplified motion model that does not account for the tilting (roll/pitch) movements of quadrotors required to accelerate in space. In scenarios where these tilting motions are significant, particularly outside of near-hovering regimes, the accuracy of the FOV weight may degrade, as the sensing capabilities of an onboard camera are influenced by the quadrotor’s tilt angle. Addressing this limitation would require extending model (21) to incorporate the effects of tilting on the field of view.

Another key challenge in vision-based relative localization is the identification of observed neighbors, which is not inherently resolved by the algorithm. Several works from the literature, as reviewed in Sect. 1.4.1, address this issue through different techniques.

Haptic shared control techniques for aerial multi-robot systems

The rapid advancement of robotic technologies has opened up new possibilities for efficient and effective task execution across various domains. Aerial robots, in particular, have emerged as versatile tools capable of addressing complex challenges in a broad range of environments. From urban planning and environmental monitoring to infrastructure inspection and entertainment, the applications of aerial robotics continue to expand, revolutionizing industries and enhancing human capabilities [52, 79]. The increasing sophistication of aerial robotic systems, however, brings forth the challenge of intuitive and efficient control mechanisms. As the number of robots in a team grows, traditional one-to-one control paradigms become impractical, necessitating innovative approaches to human-robot interaction [5, 120].

To meet the need for effective feedback in drone teleoperation, researchers have explored a variety of modalities. Haptic feedback has been a primary focus, with studies employing cable-driven systems [106], vibrotactile armbands [28], and wearable tactile gloves [122] to convey information about drone trajectory and formation states. Grounded haptic interfaces have also been utilized, using force-feedback devices to control fleet leaders and provide team status updates [76, 103] (Franchi, Secchi, Ryll, et al. 2012). Some researchers have adopted hybrid approaches, integrating haptic feedback with visual or auditory cues to enhance situational awareness [5].

This chapter introduces a series of haptic-enabled shared control techniques for the teleoperation of multiple aerial robots. We extend the foundational decentralized connectivity-maintenance algorithm for aerial robot teleoperation presented in the previous chapter by implementing versatile control architectures: (i) a single leader-follower model, (ii) a multiple leader-follower control approach, and (iii) a formation centroid control. With these extensions, we aim to enable diverse applications in complex environments, offering greater robustness and flexibility in maintaining formation and connectivity within the robotic team. Our goal is to not only enhance the operator’s situational awareness but also enable more intuitive and efficient management of multi-robot systems.

The key contributions of this work are:

1. Design and development of new shared control techniques for human-swarm interaction, utilizing innovative haptic and visual interfaces for both control and feedback.

2. Experimental evaluation via three human-subject studies conducted in a simulated environment, comparing the proposed feedback and control techniques in tasks of search and rescue, exploration, and surveillance.

We address key limitations in current teleoperation approaches for multi-robot systems, notably the challenges of keeping desired team formations, enabling intuitive and responsive human control, and grounding the framework in practical sensing and communication assumptions suitable for real-world deployment. To enhance human-robot interaction, we introduce advanced haptic feedback mechanisms, supported by user studies, which improve control precision and operator immersion. Additionally, we develop an improved visual interface to optimize operator situational awareness, facilitating the efficient management of multiple robots in complex environments while minimizing ground-to-remote communication requirements.

This work is under review for publication in the IEEE Transactions on Human-Machine Systems journal.

Automatic Leader Selection

This section introduces a method for automatically selecting the leading robot(s) in a group to minimize the constraints imposed by the connectivity maintenance task. Yaw control in multi-agent systems is particularly challenging when agents must balance field-of-view (FOV) constraints with the need to maintain connectivity. These requirements can restrict the operator’s ability to freely adjust an agent’s orientation without compromising visibility constraints. We propose a method to help the operator select the agent with the greatest rotational freedom. The approach is based on assessing each agent’s ability to look around, which reflects how unconstrained the agent is in maintaining its neighbors within its FOV.

Recalling the definition of $f_{ij}^{*}$, from § 14.2.1.1, as the function that quantifies the visibility of robot $j$ from robot $i$:

where $a_{ij} = \mathbf{o}_{C}^{T}\boldsymbol{\beta}_{ij}$ represents the cosine of the angle between $\boldsymbol{\beta}_{ij}$, the unit vector pointing from robot $i$ to robot $j$ (bearing vector), and $\mathbf{o}_{C}$, the fixed body-frame direction of the camera’s optical axis.

We can quantify the yaw maneuverability of robot $i$ based on the number of neighbors that can see it. Specifically, if a robot is seen by many neighbors, we can infer that it faces fewer constraints on its yaw orientation, as it does not need to keep those neighbors within its FOV to maintain connectivity. To quantify this, the quantity $\sum_{j \in\mathcal{N}_{i}} f_{ji}^{*}$ is a good metric as it represents the visibility of robot $i$ from its neighbors, with $f_{ji}^{*}$ defined as in (49):

Depending on the task, this metric can be used to select as a leader the agent $L$ with the greatest yaw maneuverability in a set $S$ as

where $S$ represents a suitable subset of the robots in the group, depending on the need of the task (see also Secs. 5.2 and § 14.11.1).

The next section will introduce the haptic shared control techniques and illustrate how the automatic leader selection is used to assist the operator.

Haptic shared control for Connectivity Maintenance

Haptic shared control enables a human user to control a complex robotic system, such as a group of aerial robots, while receiving force feedback on the status of the team and task. This Section outlines how we enable users to control the robotic team via a haptic interface and how they are provided with suitable sensory feedback.

Haptic interface

We use a 6 active degrees-of-freedom haptic device (Virtuose, Haption), shown in Figure 5.1a. It is able to provide force and torque feedback along all translational and rotational axes, allowing users to experience realistic interactions with virtual or remote objects across a wide workspace. The device handle is equipped with a laser sensor that detects when the user grasps it. This sensor serves two purposes:

1. It acts as a security trigger, i.e., a dead man’s switch, which disables both the control inputs and the applied force when the user is not holding the handle.

2. It sets the origin of the user’s workspace $p^0_V$ to the current position of the handle when the user grasp the handle for the first time, so as to enable clutching.

Following Aggravi, Elsherif, et al., 2021; Aggravi, Sirignano, et al., [8], we design three force contributions to achieve such a joystick-like behavior:

Haptic shared control for Connectivity Maintenance

• $F_D$ (deadzone force): This force helps users stay at the center of the workspace, within a sphere with radius $p_V^{min}$ around $p_V^0$, whenever they do not want to provide any control input; within this deadzone, control inputs are disabled.

• $F_C$ (centering force): This force brings users towards the center of the workspace $p_V^0$, making the deadzone easy and intuitive to find.

• $F_L$ (workspace limit force): This force keeps the user within the device workspace, in a sphere with radius $p_V^{max}$ centered on $p_V^0$.

For translations, let $p_V$ represent the position of the handle, with $\tilde{p}_V = p_V - p_V^0$. We define

with $k_d$, $k_c$, and $k_l$ being the deadzone, the centering, and the workspace limit gains, respectively.

From (5.4–5.6), we define $\mathbf{F}_V$ as this base force applied to the handle:

Figure 5.2 illustrates the shape of $\mathbf{F}_V$ on the $x$-axis with $p_V^0 = 0$, $p_V^{min} = 0.03$, $p_V^{max} = 1$, $k_d = 100$, $k_c = 20$, and $k_l = 500$.

For rotations, let $q_V^0$ represent the idle orientation of the handle and $q_V$ its current orientation. The difference in orientation, $\tilde{\eta}_V$, is defined as the angular difference between $q_V$ and $q_V^0$ in Euler angles. Similarly as above, we can define $T_D$ as the deadzone torque,

$T_C$ as the centering torque, and $T_L$ as the workspace limit torque, expressed as follows:

Variables $k_{dr}$, $k_{cr}$, and $k_{lr}$ represent the deadzone gain, the centering gain, and the workspace limit gain for the rotational motion, respectively. We define $T_V$ as this base torque applied to the handle:

Forces $F_V$ and $T_V$ are always active and are of course designed to be weaker than the other force contributions related to the status of the team and task, as described in § 14.11.1. This mapping approach is quite common and enables precise yet intuitive use of the haptic interface [99, 98].

Control and feedback conditions

For a set of $N$ robots, let $C_u(t)$ denote the time-varying set of agents controlled by the user, where $0 \leq|C_u(t)| \leq N$. To facilitate user-agent interaction, we assume that a ground station has access to the position of each controlled agent $j \in C_u(t)$ within a common reference frame, as well as their connectivity maintenance inputs. The ground station hosts the user interfaces and communicates directly with the agents in the group. While the robots are implementing the connectivity maintenance control presented in Sect. 4.1, we consider three control methods that allow a single user to steer the whole group:

• Single Agent Control (S): The user controls one agent at a time, with $|C_u| = 1$.

• Pinning Control (P): The user controls a virtual point that exerts a spring-like force to pull selected agents, with $|C_u| \in\{1, \ldots, N\}$.

• Centroid Control (C): The user controls the entire robot group simultaneously by translating and rotating it around the group’s centroid, as well as scaling/expanding the formation, with $|C_u| = N$.

For each of these control methods, we consider two haptic feedback conditions:

• No haptic feedback (NH): the user receives no haptic feedback about the connectivity force applied on the robot(s) they are controlling. Only the joystick-like forces $F_V$ and torques $T_V$ are applied to the Virtuose device, as introduced in Sect. 5.2.1.

• Haptic feedback (H): the user receives haptic feedback about the connectivity force applied on the robot(s) they are controlling. The forces and torques applied to the Virtuose device are the sums of the joystick-like and feedback contributions, i.e., $F_V + F_F$ and $T_V + T_F$.

According to the design of the connectivity maintenance control described in § 14.2.1, we define $v_j^\lambda$ and $\omega_j^\lambda$ as the linear and angular velocities of robot $j$ obtained from the connectivity maintenance algorithm. Stability of the coupling between connectivity maintenance and external inputs can be shown by following the same arguments as in Lee et al., [76], i.e., for any bounded external action (in our case, a bounded external command from the human operator), the closed-loop system is stable besides improbable (zero-measure) situations of perfect (and unlikely) alignments among the robots.

In the case NH (no haptic feedback), the user-controlled agents $j \in C_u$ follow the velocity control input $(v_j^\lambda, \omega^\lambda)$ from the connectivity maintenance algorithm, in addition to the user’s inputs, i.e.,

In the case $\mathbf{H}$ (haptic feedback), the controlled agents $j \in\mathcal{C}_u$ follow the user’s inputs, supplemented by a residual connectivity maintenance input $(\tilde{v}_j^\lambda,\tilde{\omega}_j^\lambda)$. This input ensures that connectivity is maintained if the haptic feedback is unable to provide enough force to guide the user (e.g., saturation of the haptic device force). Details of this mechanism will depend on the control method at hand, as presented in subsections 5.2.3, 5.2.4, and 5.2.5. The velocity control inputs are then expressed as:

In this case, the connectivity maintenance control input $(v^\lambda,\omega^\lambda)$ is not directly used as a control input, but it is provided to the users as haptic guidance, informing them about the connectivity force acting on the robot(s) they are controlling. With $\mathbf{F}_F$ representing the force feedback and $\mathbf{T}_F$ representing the torque feedback applied by the Virtuose haptic device, we have:

As $(v^\lambda,\omega^\lambda)$ increase when $\lambda_2$ is low, the force and torque feedback $(\mathbf{F}_F,\mathbf{T}_F)$ consequently become more intense. To prevent overwhelming the user with excessive feedback, we implement a saturation mechanism that limits the feedback to maximum values of $\mathbf{F}_F^{max}$ and $\mathbf{T}_F^{max}$, to be adjusted according to the needs of the task and haptic system.

In the following subsections, we describe the three control methods and the associated haptic force feedback. A video, available at https://youtu.be/gDR6Lky8_OI, shows how the different control methods work.

Single Agent Control (S)

This control mode enables the user to control one (leader) agent of the team. The rest of the agents follow the connectivity maintenance control presented in § 14.2.1. This control modality is similar to that used in the previous chapter and in Aggravi, Elsherif, et al., 2021; Aggravi et al., [5].

Control Inputs

The user controls a single agent $j$ with four degrees of freedom: $(v_j,\omega_j)$ where $v_j$ represents the linear velocities in the robot body frame and $\omega_j$ its yaw rate. As showed in (60), the user’s inputs are applied to the selected agent as an external contribution $(v_j^{E},\omega_j^{E})$.

With $\tilde{\mathbf{p}}_{V}$ and $\tilde{\boldsymbol{\eta}}_{V}=(\tilde{\eta}^{\phi},\tilde{\eta}^{\theta},\tilde{\eta}^{\psi})$ representing the control inputs from the Virtuose haptic device as described in Sec. 5.2.1, the control inputs are given by:

with $k_{E v}$ and $k_{E\omega}$ being the linear and angular velocity gains, respectively.

Haptic Feedback (H)

With $\mathbf{F}_{F}$ representing the force feedback and $\mathbf{T}_{F}$ the torque feedback guiding the user, we have

with $k_{F}$ and $k_{T}$ being the force and torque feedback gains, respectively.

By pressing the “select” button, as shown in Figure 5.1b, the user can switch the control to a different agent (only one agent at a time). The new agent to be controlled is selected using the algorithm presented in Sec. 5.1. When the controlled agent is changed, the previously controlled robot resumes following the connectivity maintenance control $(v^{\lambda},\omega^{\lambda})$, while the newly selected agent receives only the user’s inputs as discussed above.

No Haptic Feedback (NH)

In this case, no additional haptic feedback is provided to the operator; only the joystick-like force $\mathbf{F}_{V}$ (eq. (55)) and torque $\mathbf{T}_{V}$ (eq. (59)) are provided. The user-controlled agent $j$ follows the velocity control input $(v_j^\lambda,\omega_j^\lambda)$ from the connectivity maintenance algorithm, in addition to the user’s inputs, as presented in eq. (60).

Pinning Control (P)

Control Inputs

The user controls a virtual point that pulls selected agents using a spring-like force: $\lvert\mathcal{C}_u\rvert\in\{1...N\}$. Let $(p_p,\psi_p)$ represent the position and yaw of the virtual point and $(v_p,\omega_p)$ its linear and angular velocities in the body frame. The dynamics of the virtual pinning point are expressed as:

The user can impart two types of motions to the controlled UAVs: translational motion by moving the virtual point to draw the agents towards it and rotational motion by orbiting around the virtual point. For each controlled robot $j \in\mathcal{C}_u$, the spring-like force applied is given by:

where $p_{jp}=(p_p-p_j)$ and $d_{jp}^{\min}$ is the fixed minimum distance between the virtual point and the agent $j$. Figure 5.3a illustrates the pinning force that pulls agent $j$ towards the orbit of the virtual point, while Figure 5.3b shows the rotational pinning force, enabling the user to control the agent’s orbit around the virtual point.

To provide visual feedback to the user about the pinning location, at least one agent should face the direction of motion of the group. To achieve this, we can select an agent $l$ from the set of controlled UAVs $\mathcal{C}_u$ using the algorithm outlined in § 14.5.1. This agent is then controlled to minimize the angle between its onboard camera direction and the vector $p_{jp}$. The yaw control input for the selected robot $l$ is given by:

where $\psi_l$ is the yaw of robot $l$.

In both feedback cases NH and H, the selected agent $l$ follows the pinning control yaw rate input in addition to its implemented connectivity maintenance yaw rate. This is expressed as

To ensure that connectivity is maintained, the residual connectivity inputs $(\tilde{v}_j^\lambda,\tilde{\omega}_j^\lambda)$ are applied to the controlled agents $j \in\mathcal{C}_u$ as follows:

with $k_r$ being a scaling gain (set to 0.1 in our experiments). As the connectivity maintenance force becomes increasingly strong for low values of $\lambda_2$, these residual inputs ensure that connectivity is maintained if the haptic feedback proves insufficient or unable to fully guide the user, while otherwise allowing the user with maximum control freedom.

Haptic Feedback (H)

In the set of controlled robots $\mathcal{C}_u$, the user controls the virtual point rather than the individual agents directly. Consequently, the force feedback guiding the user is the average of the projection of the connectivity maintenance forces $\mathbf{F}_j^\lambda$, $j \in\mathcal{C}_u$ onto the vector $\mathbf{p}_{jp} and the tangent plane containing $\mathbf{p}_{jp} \times\tilde{\boldsymbol{\eta}}_{V}$, as illustrated in Figure 5.4.

Let $I^{3}$ denote the $3 \times3$ identity matrix, $J = \lvert C_{u} \rvert$ be the number of controlled agents, and $\mathbf{u}_{jp} = \mathbf{p}_{jp}/\lVert\mathbf{p}_{jp}\rVert$ represent the unit direction of the pinning force. The force feedback is expressed as:

with $k_{F}$ and $k_{T}$ the force and torque feedback gains, respectively.

No Haptic Feedback (NH)

In this case, no additional haptic feedback is provided to the operator; only the joystick-like force $\mathbf{F}_{V}$ (eq. (55)) and torque $\mathbf{T}_{V}$ (eq. (59)) are provided. The user-controlled agent $j$ follows the velocity control input $(\mathbf{v}_{j}^{\lambda}, \boldsymbol{\omega}_{j}^{\lambda})$ from the connectivity maintenance algorithm, in addition to the user’s inputs, as presented in eq. (60).

Centroid Control (C)

Control Inputs

In this configuration, the user controls the entire robot group with seven degrees of freedom: translations, orbiting around the centroid, and scaling of the group. The centroid position of the group, $\mathbf{p}_{c}$, is not directly available to the individual agents and is calculated centrally by the ground station. This value is used to determine the velocity inputs sent to the drones. The centroid position is computed as the average of the positions of the controlled agents $j \in\mathcal{C}_u$, given by:

with $N$ denoting the total number of robots in the group. The orientation of the centroid $\psi_c$ is controlled by the user with the dynamics:

Let $\mathbf{\Gamma} = [\Gamma_x, \Gamma_y, \Gamma_z, \Gamma_\phi, \Gamma_\theta, \Gamma_\psi, \Gamma_\epsilon]$ represent the user’s inputs for the centroid control. We define the control mapping matrix $\mathbf{H}_c$ as follows:

with $\mathbf{K}_{\Gamma} = [K_{\Gamma_x}, K_{\Gamma_y}, K_{\Gamma_z}, K_{\Gamma_\phi}, K_{\Gamma_\theta}, K_{\Gamma_\psi}, K_{\Gamma_\epsilon}]$ representing a vector of gains. The 6 DoF inputs from the Virtuose device are then mapped to the centroid control inputs as follows:

with $\circ$ representing the Hadamard product.

Let $\mathbf{p}_{jc} = (p_{jc}^x, p_{jc}^y, p_{jc}^z) = \mathbf{R}_j^T(\mathbf{p}_j - \mathbf{p}_c)$ denote the relative position of each robot $j$ with respect to the centroid of the group $\mathbf{p}_c$ in the body frame of the agent $j$. We define $\mathbf{u}_{jc} = \mathbf{p}_{jc}/\lVert\mathbf{p}_{jc}\rVert$ as the normalized vector of $\mathbf{p}_{jc}$. Using $b_p \in\{0,1\}$ and $\bar{b}_p = (1-b_p)$ to switch between pan-tilt control ($b_p = 1$) and scaling control ($\bar{b}_p = 1$), we define the

Centroid Control matrix $\mathbf{Q}_j$ as follows:

From (75), we construct the global matrix $\mathbf{Q}$ as follows:

The value of $b_p$ can be changed by pressing a button on the Virtuose device, allowing the user to switch between pan-tilt and scaling control.

From (76), the control inputs for an individual agent $j \in\mathcal{C}_u$ can be expressed as:

and the overall control inputs for the entire robot group is then given by:

Haptic Feedback (H)

As the user does not control all the degrees of motion of the agents but rather their position around the centroid, we decompose the connectivity force $\mathbf{F}_j^{\lambda}$ implemented by each agent $j \in\mathcal{C}_u$ into two components:

• $\pi_{\parallel}$: The part of the connectivity force the user can control through the inputs

presented in Sect. 5.2.5;

— $\tau_{\perp}$: The part of the connectivity force the user cannot directly control, e.g., the yaw of individual agents.

We propose to guide the user with the component $\tau_{\parallel}$ through haptic feedback, while the remaining component $\tau_{\perp}$ will be added to the robot inputs as part of the group autonomy through $\tilde{v}_{j}^{\lambda}$ and $\tilde{\omega}_{j}^{\lambda}$.

From (76) we define $\tau_{\parallel}$ as:

where $v_{j}^{E}$ and $\omega_{j}^{E}$ represent the aggregated linear and angular velocities for the entire group, respectively, and $Q^{\dagger}$ represents the pseudo-inverse of $Q$. Similarly, $\tau_{\perp}$ is defined as:

Using the control mapping matrix $H_{c}$ from (73), the force feedback guiding the user is then:

and the overall residual connectivity inputs $(\tilde{v}^{\lambda},\tilde{\omega}^{\lambda})$ are given by:

such as

No Haptic Feedback (NH)

In this case, no additional haptic feedback is provided to the operator; only the joystick-like force $F_V$ (eq. (55)) and torque $T_V$ (eq. (59)) are provided. The user-controlled agent $j$ follows the velocity control input $(v_j^x,\omega^\lambda)$ from the connectivity maintenance algorithm, in addition to the user’s inputs, as presented in eq. (60).

Software Implementation and User Interface

We use ROS 2 to interconnect two main components: the ground station and the robot group. The functions and integration of the different components are summarized in Figure 5.5 and detailed in the following subsections. An even more detailed description of the software architecture can be found in Appendix A. We use ROS 2 to interconnect the ground station and the robot group, facilitating communication and coordination between the system’s components. The overall structure is summarized in Figure 5.5 and further explained in the following subsections. For additional implementation details, please refer to Appendix A.

Ground station

Given the control methods presented in Sect. 5.2, a centralized ROS 2 node functions as the ground station and serves as a bridge between the robots and the user interface. On one hand, it interfaces with the Virtuose device to process user inputs and provide connectivity-based haptic feedback, if applicable. On the other hand, it forwards external control commands to the agents and receives their states for display in the Unity 3D visualization.

The ground station runs several other ROS 2 nodes responsible for operating other functions such as takeoff, emergency stop or landing, logging data for debugging purposes or emulating sensors, such as a limited FOV sensor providing relative positions w.r.t. their neighbors to the robots.

Robots control and simulation

Each robot operates as two individual ROS 2 nodes, as shown in Figure 5.5:

Ground Station

• Visualization - TCP link with Unity;

• Haptic interface - wrapper for Raptor API, handling the Virtuose device;

• Process user commands for Single Agent Control; Pinning Control or Centroid Control.

Information feedback

• Robots’ state

• Controllers related values (weights, forces...)

• Other debugging data...

Command inputs

• External control ($v_j^E, \omega_j^E$);

• ROS 2 services calls (takeoff, landing, onboard controller toggling...)

drone $i$, $i \in N$

Control node(s)

• Processing of command inputs ($v_j^E, \omega_j^E$);

• Estimations (eg. relative yaw, $\lambda_2$);

• Decentralized connectivity maintenance instance computing ($v_i^\lambda, \omega_i^\lambda$);

• Communication with neighbors.

Position and Velocity

Velocity commands

Simulation node

• Simulates second order dynamics with limited acceleration and velocity;

• Handles workspace limitations.

1. Simulation Node: It implements the dynamics of a second-order system, with a maximum acceleration of $5\,\mathrm{m}\cdot\mathrm{s}^{-2}$. It receives control commands as input and outputs the robot’s state, including position, velocity and orientation. This node runs at 100 Hz.

2. Control Node: It implements the decentralized connectivity maintenance algorithm and handles the necessary estimations and communication protocols. It also integrates external control inputs from the user with the connectivity-maintenance control. This node runs at 30 Hz.

Visualization

We use the ROS TCP Connector Unity plugin Unity Technologies, [124] and the ROS TCP Endpoint ROS 2 package [125] to connect the ROS 2 control and simulation nodes with the Unity visualization through a TCP bridge.

In a virtual Unity 3D environment, each robot is represented by a UAV 3D model, as shown in Figure 5.6, Figure 5.7 and Figure 5.8.

In addition to providing visual feedback, the Unity interface also allow for user interaction with the robots. For example, during Pinning Control, the user can select or deselect pinned agents by clicking on them within the Unity interface. Depending on the

control method, the third-person camera follows either a single agent, the virtual pinning point, or the centroid of the group. At all times, the user can rotate the camera around the tracked object and zoom in or out using the mouse.

Experimental Scenarios

This Section describes how we leverage the visualized simulated environment to test the different control methods. We have implemented three distinct scenarios, each presented in the video available at https://youtu.be/gDR6Lky8_OI:

1. Single Agent Control ($S$, see Sect. 5.2.3): Each robot is equipped with one of three types of sensors, distinguished by different colors. There are six target locations, and reaching one reveals the next in sequence. Each target must be visited by a robot equipped with a specific sensor type. The user controls the appropriate robot to reach the designated target location, viewing the scene through a third-person camera attached to the controlled robot. Video: https://youtu.be/gDR6Lky8_OI?t=34.

2. Pinning Control ($P$, see Sect. 5.2.4): Similarly as before, each robot is equipped with one of three types of sensors, distinguished by different colors. Here, each objective requires two robots, each with a specific and distinct sensor type, to reach the target. The user controls the pinning point to make the two designated robots reach the current target location, viewing the scene through a third-person camera attached to the virtual pinning point. Video: https://youtu.be/gDR6Lky8_OI?t=177.

3. Centroid Control ($C$, see Sect. 5.2.5): At the start of the task, the environment is hidden by a dark fog that gradually clears as the robotic team moves across the map. Certain areas are marked in red and must be avoided. The user controls the robotic team to uncover as much of the environment as possible, viewing the scene from a top-down perspective with an orthographic projection. Video: https://youtu.be/gDR6Lky8_OI?t=334.

To investigate the effect of haptic feedback on user performance depending on the control strategy, users repeat the given mission on each scenario twice: once with ($H$) and once without ($NH$) haptic feedback. To mitigate learning effects from repeating the same mission in identical visual settings, we have developed two visually distinct environments, which we called “Plains” (see Figure 5.6) and “Desert” (see Figure 5.7). The two environments are visually different, but the target objectives to visit are placed in the same locations.

User study

To evaluate the effectiveness and usability of the proposed control and feedback methods, we carried out a user study enrolling 20 participants.

Methods

Setup and Task

		Control conditions
Feedback conditions		Single Agent	Pinning	Centroid
Feedback conditions	without Haptic Feedback	SNH	PNH	CNH
Feedback conditions	with Haptic Feedback	SH	PH	PNH

Participants are asked to seat in front of a large TV screen and hold the end-effector of the Virtuose 6D device to control the multi-robot team (see Figure 5.9).

The experiment is divided into three blocks, one for each of the experimental scenarios described in § 14.11.1.4, presented in the same sequence for all users: Single Agent Control (S), Pinning Control (P), and Centroid Control (C). Each scenario is then repeated once with haptic feedback ($H$) and once without haptic feedback ($NH$). The order of presentation of the haptic feedback ($H$ or $NH$) and visual environment (Desert and Plains) was counterbalanced among participants.

At the beginning of each block, we carried out a training session in an empty environment, so as to make users familiar with the considered control method and haptic feedback.

User study

Collected Data

We recorded the user inputs and the force feedback provided. In scenarios $S$ and $P$, we measured the task completion time; shorter times were interpreted as indicative of a more effective control method. In scenario $C$, we measured the area explored by the user; a larger discovered area was interpreted as indicative of a more effective control method. Prior to the experiment, participants completed a demographic questionnaire.

At the end of each block, participants filled out a subjective questionnaire to evaluate their experience with the control method and the haptic feedback system. This evaluation included two aspects: perceived effectiveness in achieving the task and the intuitiveness of the control method.

Population

The study involved 20 participants (16 M, 4 F) aged 18 to 35 years ($m$: 25.5y, $\mathrm{sd}$: 4.5y). Of these participants, 18 were right-handed and 2 were left-handed. Half of the participants were experienced with haptics, while the other half were not. Similarly, half had experience with multi-robot systems, while the other half did not. Additionally, two participants had very limited experience with virtual 3D environments (such as video games), whereas the remaining 18 participants had extensive experience.

Results

Single Agent Control

Figure 5.10a presents the results of user-rated effectiveness and intuitiveness for this scenario (higher scores indicate better user experience). A paired-samples t-test determined that the registered effectiveness differed statistically significantly between feedback conditions ($NH$ vs. $H$, $t(19) = -2.342$, $p = 0.030$, $H$ is better, confidence interval of 95%). No statistically significantly difference was registered for the intuitiveness metrics. Figure 5.10b shows the measured completion time (lower is better). A paired-samples t-test determined that the completion time differed statistically significantly between feedback conditions ($NH$ vs. $H$, $t(19) = -4.296$, $p < 0.001$, $H$ is better, confidence interval of 95%).

As discussed in Sec. 5.2, in the $NH$ condition, the controlled agent follows the connectivity maintenance control command with a reduced gain compared to the other agents, providing the user with greater control flexibility. In contrast, in the $H$ condition, the controlled agent exclusively follows the user’s input, which likely accounts for the improved completion time.

Pinning Control

Figure 5.11a presents the results of user-rated effectiveness and intuitiveness for this scenario (higher scores indicate better user experience). A paired-samples t-test determined that the registered effectiveness differed statistically significantly between feedback conditions ($NH$ vs. $H$, $t(19)=-2.179$, $p=0.042$, $H$ is better, confidence interval of 95%). No statistically significantly difference was registered for the intuitiveness metrics. Figure 5.11b shows the measured completion time (lower is better). No statistically significantly difference was registered for this metrics.

Centroid Control

Figure 5.12a presents the results of user-rated effectiveness and intuitiveness for this scenario (higher scores indicate better user experience). A paired-samples t-test determined that the registered intuitiveness differed statistically significantly between feedback conditions ($NH$ vs. $H$, $t(19)=-2.773$, $p=0.012$, $H$ is better, confidence interval of 95%). No statistically significantly difference was registered for the effectiveness metrics. Figure 5.12b shows the measured completion time (lower is better). A paired-samples t-test determined that the discovered environment differed statistically significantly between feedback conditions ($NH$ vs. $H$, $t(19)=-2.255$, $p=0.036$, $H$ is better, confidence interval of 95%).

Use case

We show the robotic team carrying out a more complex recognition task, seamlessly switching from one control modality to the other. The video of this use case is available at https://youtu.be/gDR6Lky8_OI?t=463.

A human operator is tasked with controlling the multi-robot team to reach multiple objectives in a large map using different control methods. The mission begins with the operator using the Centroid Control method to quickly guide the entire robot group to the target area. Upon arrival, the operator switches to Single Agent Control to identify the areas to visit. Once the objectives are located, the operator employs Pinning Control to direct multiple agents to the target locations and execute the tasks. After visiting all the objectives, the operator returns to Centroid Control to quickly guide the group back to the home base.

Figures 5.13 and 5.14 shows the trajectories of the eight drones throughout the task. Star markers indicate the final position of each agent, while the fading lines represent their trajectory, with older positions appearing more transparent. Each dot along the trajectory corresponds to the agent’s position at 5-second intervals.

Figure 5.15 (top) illustrates the evolution of the second eigenvalue of the Laplacian matrix $\lambda_{2}(t)$ throughout the experiment, colored patches indicate changes in the used control method. Figure 5.15 (bottom) presents the evolution of the number of connected

agents. As depicted in the figures, the connectivity maintenance algorithm successfully maintains the connectivity of the multi-robot team throughout the mission, even when the control methods change. Additionally, we observe the dynamic removal and creation of edges, demonstrating the flexibility and robustness of the connectivity maintenance algorithm.

Conclusions

In this chapter, we proposed and evaluated a series of haptic shared control techniques for the teleoperation of multi-robot aerial systems. Specifically, we implemented and assessed three control paradigms: Single Agent Control, Pinning Control, and Centroid Control, each designed to address different aspects of multi-robot teleoperation. These approaches featured innovative haptic guidance mechanisms to inform the operator about the team’s connectivity state and task dynamics.

The experimental results from the user study involving 20 participants demonstrated the overall benefit of employing haptic feedback. For instance, in the Single Agent Control scenario, the use of haptic guidance significantly reduced task completion times,

Conclusions

highlighting the efficacy of haptic cues in providing situational updates. Subjective feedback indicated an improvement in perceived effectiveness and intuitiveness when haptic feedback was utilized. This was particularly evident in the Centroid Control mode, where participants rated the haptic-enabled interface as significantly more intuitive compared to the non-haptic variant. Overall, the designed control methods effectively maintained global swarm connectivity under varying conditions, including transitions between control modes. The use case demonstrated the seamless transition between control modalities, showcasing the framework’s adaptability to mission requirements. This flexibility supports a wide range of applications, from search and rescue to environmental exploration.

The presented framework addresses some important challenges in human-swarm interaction, particularly the need for intuitive and responsive control mechanisms for large-scale robotic teams. By leveraging haptic feedback, operators can maintain high levels of situational awareness while managing complex tasks. The findings validate the hypothesis that haptic feedback not only enhances task efficiency but also reduces cognitive load by providing interpretable cues about system states.

Conclusions and Future Work

In this final part, we aim to summarize the main theoretical and experimental contributions of this thesis, highlighting the key results achieved. Additionally, we will discuss the limitations and unresolved challenges encountered during the course of this work. For these open issues, we propose potential directions for future investigation and research.

Summary and contributions

The primary objective of this thesis was to explore haptic shared control for multirobot systems, emphasizing the development of human-robot interfaces that harness the complementary strengths of humans and robots. We sought to address challenges arising from sensing constraints and incomplete relative measurements, while relaxing the strict requirement of rigidity of the formation, thereby granting the robots greater freedom of movement.

Building on the concept of formation control under global constraints (§ 12.3.3), this thesis introduced an extension to connectivity maintenance that addresses field-of-view limitations and body-frame measurements (§ 14.2.1). Additionally, a novel method for estimating relative orientations among sensed agents was proposed (Section 4.2). This approach enables robots to autonomously adjust pair-wise distances and orientations to maintain a globally connected communication and sensing graph, as demonstrated through simulated experiments (Section 4.3).

After establishing the feasibility of the connectivity maintenance approach, it was integrated into a haptic shared control framework, allowing human operators to guide robots via three distinct control architectures: single-leader control (Section 5.2.3), multiple-leader control (Section 5.2.4), and formation centroid control (§ 14.10.0.1). The effectiveness of this framework was validated through user studies comparing the shared control interface with haptic feedback to a teleoperation interface without haptic feedback (Section 5.4). These studies highlighted increased efficiency when controlling a single leader and greater intuitiveness when managing the formation centroid (§ 15.1.2.1). Furthermore, a use case demonstrated seamless transitions between control modalities, enabling operators to adapt dynamically to mission requirements (§ 15.3.1).

Limitations and future perspectives

The results of this work highlight the potential of haptic shared control in enhancing the operation of multi-robot systems in dynamic environments with realistic assumptions on sensor capabilities and communication constraints. However, several limitations and challenges remain.

The current implementation of haptic feedback is limited to force feedback, overlooking the potential advantages of incorporating tactile feedback. Enhancing the interface with additional sensory modalities could significantly improve the operator’s situational awareness and task performance. For instance, the DORNELL project (https://www.inria.fr/fr/dornell) aims to design a multi-actuator vibrotactile handle intended to guide visually impaired individuals. Such handle could be integrated with the Virtuose device, greatly improving the system’s capability to convey information to the operator [26, 27].

Furthermore, the user study was conducted in a controlled environment with a relatively small number of participants, which may not adequately capture the variability in user preferences and behaviors. Conducting a broader study with a larger and more diverse participant pool, as well as investigating long-term learning effects, would offer a more comprehensive evaluation of the proposed haptic shared control framework. Additionally, exploring collaborative control by multiple operators using haptic interfaces could expand the framework’s applicability to larger and more complex missions.

Several aspects of the algorithms developed in this thesis could be further investigated to enhance their performance and robustness. The connectivity maintenance algorithm presented in Sect. 4.1 relies on a simplified motion model that does not account for the roll and pitch motions of quadrotors required for acceleration. In scenarios involving agile or aggressive maneuvers, such tilting can cause onboard cameras to lose sight of neighboring robots, preventing relative position measurements. Overcoming this limitation would require extending the model to account for the impact of tilting on the field of view.

Moreover, the connectivity maintenance algorithm cannot recover from a temporary loss of global connectivity. If the second smallest eigenvalue of the Laplacian matrix becomes zero due to external perturbations, latencies or other factors, the formation splits into disconnected components. In such cases, the algorithm cannot autonomously restore a globally connected graph without external intervention. A recent work by Restrepo and Giordano, [102] proposes a solution to address this limitation, enabling the addition or removal of agents within the connected formation.

To account for field-of-view constraints, the connectivity algorithm presented in this thesis does not include line-of-sight occlusion and obstacle avoidance capabilities, which were included in the algorithm originally proposed by Robuffo Giordano et al., [104]. Future work could focus on integrating these features into the connectivity maintenance algorithm to enhance the robots’ safety and improve the system’s robustness in cluttered environments.

However, due to its design, the algorithm can account for heterogeneous field-of-view among agents. This feature was tested, although not formally documented, in a scenario where one agent of the team had a null field-of-view, rendering it incapable of measuring neighbors’ relative positions. Nevertheless, by leveraging communication and the visual perception of other agents, the blind agent maintained connectivity with the team. This capability is particularly relevant for heterogeneous teams of robots, where UAVs equipped with advanced sensors can supervise ground agents while maintaining connectivity to ensure communication and coordination in areas where direct communication with a remote operator is not possible. Such fully autonomous systems offer significant potential for applications in search and rescue missions, where heterogeneous robot teams can maintain communication and sensing capabilities while efficiently exploring unknown environments.

In scenarios where remote operator intervention is required, the connectivity maintenance algorithm can still play a crucial role in ensuring local communication and sensing capabilities among the robots. The following remarks offer insights into the practical feasibility of remotely controlling a group of robots in realistic scenarios using the proposed framework.

These considerations pave the way for sophisticated real-world experiments, where the robotic team operates with full autonomy. The human operator oversees the mission through a user-friendly interface, gaining access to the robots’ state and environmental data without requiring direct communication with each individual robot.

Connectivity maintenance is, in our opinion, a key feature for decentralized multirobot systems, as, with appropriately designed weights modeling inter-agent constraints, it ensures that robots maintain physical requirements. The algorithm proposed in this thesis guarantees that both the communication and sensing graphs remain globally connected, enabling the integration of other decentralized, task-specific algorithms that rely on global connectivity such as coverage, team state estimation or exploration and mapping. In advanced scenarios requiring human intervention to resolve deadlocks, make decisions, or adapt to unforeseen events, the haptic shared control framework can assist the user in comprehending the situation and guiding robots efficiently and intuitively.

Appendix

Decentralized Software Implementation

In this appendix, we present the software architecture and implementation details of the decentralized multi-agent human interaction system developed for this thesis. The system is designed to operate in a simulated environment, with minimal modifications required for deployment in real-world experiments.

Built on the Robot Operating System 2 (ROS 2) middleware, the software provides a flexible and modular framework for robotic applications. It consists of three main components: the onboard software running on each agent, the ground station, and a Unity-based graphical interface. The onboard software handles the autonomous control of the agents, while the ground station acts as the central hub for human-multi-robot interaction. The Unity environment offers a user-friendly interface for monitoring and controlling the agents.

Designed with modularity in mind, the system allows for seamless integration of new controllers, sensors, and communication protocols, making it adaptable to various applications. However, in this appendix, we focus solely on the specific use case showcased in this thesis.

Onboard software

The onboard software is responsible for controlling the drone. While all nodes run on the same machine in a simulated experiment, the onboard nodes are isolated under a dedicated ROS 2 namespace for each agent, effectively simulating their physical separation from the rest of the system. The main nodes composing the onboard software of a given agent $i$, as illustrated in Figure A.1, are:

• Integrator node: In a simulated environment, drone dynamics are modeled using a double integrator with limited acceleration and velocity. This node receives velocity commands in the body frame of drone $i$ and outputs the corresponding velocity and pose of the agent.

• Main controller node: This node loads and executes high-level controllers, such as the connectivity maintenance controller and the reactive controller. It combines their outputs with external inputs to generate the final velocity command for the integrator node. — Connectivity maintenance controller: Implements the decentralized connectivity maintenance algorithm presented in this thesis. It processes the relative positions of sensed neighbors along with communicated data (PI-ACE values and relative position measurements) to compute the connectivity maintenance force and generate the desired velocity command. Additionally, it transmits relevant data to sensed neighbors using peer-to-peer dedicated topics. This communication design presents a scalability limitation, as the number of topics increases exponentially with the number of agents. In a real-world experiment, inter-agent communication would be managed by a dedicated communication module, mitigating this limitation. — Reactive controller: Implements a simple reactive avoidance algorithm. It receives the relative positions of neighboring agents and computes a repulsive force to prevent collisions.

Groundstation

The ground station serves as the core of the human-multi-robot interaction system. Its primary component is the user control interface, which manages initialization, monitoring, and external control of the agents. Additionally, the ground station simulates onboard sensors, such as a relative position sensor, and provides force feedback to the user through a haptic device. Finally, the ground station handles scenario-specific tasks, including managing objectives, logging data, and debugging.

User inputs

Leveraging standardized ROS 2 topics, users can interact with the system through various input devices. Commands can be sent to the agents using a standard gamepad running the default joy node provided by ROS 2 or via the Virtuose from Haption, a 6-DOF haptic device.

A custom ROS 2 node (test_cm_haptic), implemented in C++ for high execution speed, outputs a sensor_msgs/joy message, mimicking a regular gamepad. Unlike a gamepad, however, the haptic device requires force feedback from the ground station, which is transmitted via a geometry_msgs/WrenchStamped message to the test_cm_haptic node on the /groundstation/force_feedback topic. This topic is published by a dedicated node (force_feedback_node) that computes the force feedback based on current control modality, while the test_cm_haptic node handles the joystick-like forces. Both forces are detailed in Chapter 5.

To interface with the Virtuose device, we utilize the Raptor API from Haption, embedded within a dedicated wrapper node provided by Hap tion and slightly modified to fit our requirements. This wrapper node initializes the haptic device and serves as an interface for a client node (test_cm_haptic in our case). It exposes several readable topics, such as the device’s position, velocity, and status, as well as a writable topic for desired force feedback.

Figure A.2 provides a graphical representation of the haptic device implementation within our system.

User control interface

The centralized_controller_node serves as the primary interface between the user and the system. It enables the user to send commands to the agents, such as takeoff, landing, or switching high-level controllers. Additionally, it allows dynamic selection of the controlled agent and adjustment of the control modality (single-agent, pinning, or centroid).

The node operates based on a state machine, illustrated in Figure A.3, which transitions through various initialization states before entering the main execution loop.

To support the various control modalities presented in this thesis, the centralized_controller_node either directly interacts with a single agent, forwarding velocity commands from the user input system, or delegates control to a dedicated node responsible for implementing pinning or centroid control strategies. This design enables seamless switching between control modalities, as demonstrated in the video available at https://youtu.be/gDR6Lky8_OI.

Pinning control The pinning_control_node emulates a virtual agent that is directly controlled by the user. Its state is sent to the Unity environment for visualization and is used to compute external commands for the pinned agents, as detailed in Chapter 5. Additionally, the node computes force feedback and publishes it to the /pinning_control/feedback_force topic, which is accessed by the force_feedback_node.

Centroid control The centroid_control_node calculates the centroid of the agents group and transmits its position to the Unity environment for visualization. It also determines the external commands for the agents, as described in Chapter 5. The position of the centroid is not directly controller by the user but rather by the movement induced by the user onto the drones. The node is also responsible for computing the force feedback, publishing it onto a dedicated /centroid_control/feedback_force topic, accessed by the force_feedback_node.

Sensor simulation

A central node (relat_pose_node) simulates an onboard sensor that provides relative position measurements of identified neighbors within a limited field of view (FOV). It computes the relative position of all neighboring agents in the body frame of the observing agent, considering FOV constraints, and publishes it to the /dronei/relat_poses topic, which serves as input for the connectivity maintenance controller. Additionally, it supplies the reactive controller with relative position data of neighboring agents without FOV restrictions, effectively simulating an omnidirectional close-range distance sensor.

Miscellaneous nodes

Several optional nodes operate within the system, handling scenario-specific tasks, data logging, or debugging. These nodes include:

• Objective node: This node manages objectives in the single-leader and pinning scenarios. It retrieves the list of objectives from the Unity environment and verifies whether the designated drone has reached the currently highlighted objective. If so, it either progresses to the next objective or concludes the mission.

• Task timer node: This node provides two services to start and stop a timer, enabling the measurement of mission completion time. It continuously publishes time data on a dedicated topic throughout the mission in order to save it in a ROS bag.

• Debug node: Initially designed to compare estimated $\lambda_{2}$ values with actual values, this node can also supply agents with the real centrally computed $\lambda_{2}$ value for improved stability and efficiency. Additionally, it transmits inter-agent weight values to the Unity environment for visualization.

• Leader suggestion node: Implementing the leader suggestion strategy described in Chapter 5, this node provides a service to recommend the most suitable leader from a given subset of agents.

• Data grabber node: Previously used for logging experimental data into CSV files for offline analysis, this node has been deprecated. Data logging is now managed via a ROS bag recorded from a separate computer connected to the same network. This physical separation ensures that the logging process does not interfere with the system’s real-time performance.

Graphical interface

This section provides a brief overview of the functions implemented in the Unity environment, which serves as the graphical interface for the user. For a more comprehensive understanding of the system, we strongly encourage the reader to refer to the full video available at https://youtu.be/gDR6Lky8_OI, as presented in Chapter 5.

The Unity-based visual interface offers a user-friendly menu for selecting various display options, including:

• Visualization of topological links between agents, color-coded according to their connectivity weights.

• Selection of the observer camera type (onboard the current leader or external 3rd person view).

• Display or concealment of onboard camera views.

• Choice of environment (desert or plains)...

The Unity environment interfaces with the ROS 2 system via TCP, utilizing the default_server_endpoint node from the ROS TCP Endpoint package [125]. This TCP connection enables the Unity visualization to run on a separate device from the ROS 2 system, reducing the load on a single machine and allowing for the use of a Windows-based platform, which facilitates integration with a Virtual Reality headset. The Unity environment subscribes to multiple topics such as robot positions, inter-agent connectivity weights, and the positions of the virtual pinning agent or centroid (if applicable), along with other scenario-specific data. Additionally, it implements and calls ROS 2 services while monitoring environment-specific metrics depending on the scenario.

For instance:

• Single-leader or pinning scenarios: The Unity environment provides a service called by the objective_node, which supplies the list and types of environment-specific objectives. It is also used to set which objective is currently highlighted for the user, using a colored cylinder and a moving arrow, as shown in the videos.

• Centroid control scenario: The Unity environment detects when an agent enters a restricted (red) area and reports it to the ROS 2 system. It also manages the progressive discovery of new areas by dynamically removing the fog of war. At the mission’s conclusion, the Unity environment saves the discovered regions as a bitmap image.

Finally, the Unity environment provides the ROS system with the altitude of the drones in the simulated environment through a service call, allowing the system to adjust the drones’ starting altitude and avoiding setting a virtual limit to the drones’ vertical movement.

Launch files

This project uses the simple launch package available on Github at https://github.com/oKermorgant/simple_launch. This package provides a Python class to help writing ROS 2 launch files. The way ROS 2 launch files are handled allows for an easily configurable system, allowing to change parameters and arguments without modifying the code nor compiling the project. This feature is particularly useful for the system presented in this thesis, as it allows for quick changes in the system’s configuration, such as the number of agents, the control modality, or the simulation environment.

Here is an example of a launch file for the system presented in this thesis:

def generate_launch_description():
    sl = SimpleLauncher()

    drone_ids = ['drone1', 'drone2', 'drone3', 'drone4','drone5', 'drone6',
                 'drone7', 'drone8']
    leader_id = 'drone1'

    sl.declare_arg('simulation', True, description='Switch simulation/real
        experiment')
    sl.declare_arg('debug', True, description="Use debugging features")
    sl.declare_arg('save_data', False, description="Save experiment data")
    sl.declare_arg('rate', 30, description='Control rate')
    sl.declare_arg('min_dist', 1.5, description='Minimum distance for the
        reactive controller')
    sl.declare_arg('use_pinning', False, description='Use pinning control')
    sl.declare_arg('use_objectives', True, description='Use objectives')
    sl.declare_arg('double_type_objectives', False, description='Use
        objectives with two types')
    sl.declare_arg('ros_ip', '127.0.0.1', description='ROS tcp connector IP')
    sl.declare_arg('use_leader_suggestion', True, description='Use leader
        suggestion service')
    sl.declare_arg('use_force_feedback', True, description='Use force
        feedback')
    sl.declare_arg('centroid_control', False, description='Use centroid
        control')
    sl.declare_arg('drones_ids', drone_ids, description='List of drones ids')
    sl.declare_arg('use_virtuose', False, description='Switch between joystick
        control and Virtuose control')
    sl.declare_arg('takeoff_height', 4.0, description='Altitude objective for
        takeoff service')
    sl.declare_arg('max_velocity', 2.0, description='Maximum velocity for
        drone simulator')

24 weights_file_name = "drones_weights_params.yaml" % Weight parameters file
   # for the connectivity maintenance controller
25 weights_config_file =
   sl.path_join(get_package_share_directory("olive_controller"),
   "config", weights_file_name)
26
27 args = {'drones_ids': sl.arg('drones_ids'),
28         'leader_id': leader_id,
29         'debug': sl.arg('debug'),
30         'rate': sl.arg('rate'),
31         'save_data': sl.arg('save_data'),
32         'min_dist': sl.arg('min_dist'),
33         'use_pinning': sl.arg('use_pinning'),
34         'centroid_control': sl.arg('centroid_control'),
35         'use_objectives': sl.arg('use_objectives'),
36         'double_type_objectives': sl.arg('double_type_objectives'),
37         'use_leader_suggestion': sl.arg('use_leader_suggestion'),
38         'use_virtuose': sl.arg('use_virtuose'),
39         'use_force_feedback': sl.arg('use_force_feedback'),
40         'weights_config_file': weights_config_file
41         }
42 # All the arguments are loaded and passed on to the launch file
   # responsible for launching all the groundstation components
43 sl.include('olive_groundstation', 'gs_launch.py', launch_arguments=args)
44
45 for drone in drone_ids:
46     args = {'simulation': sl.arg('simulation'),
47             'use_genomix': sl.arg('use_genomix'),
48             'debug': sl.arg('debug'),
49             'rate': sl.arg('rate'),
50             'drone_id': drone,
51             'min_dist': sl.arg('min_dist'),
52             'drones_ids': sl.arg('drones_ids'),
53             'use_pinning': sl.arg('use_pinning'),
54             'centroid_control': sl.arg('centroid_control'),
55             'takeoff_height': sl.arg('takeoff_height'),

'max_velocity': sl.arg('max_velocity'),
'weights_config_file': weights_config_file
}
# For each agent of the group, the arguments are loaded and passed on
    to the launch file responsible for launching all the onboard
    components
sl.include('olive_controller', 'cm_control_launch.py',
    launch_arguments=args)

return sl.launch_description()

This launch file is used to start the system with a given number of agents, a specific leader, and various configuration options. It launches the ground station and all the agents, passing the necessary arguments to each launch file. The launch files for the ground station and the agents are written in a similar fashion, allowing for easy configuration of the system.

This software is part of a project named Olive, which aims to facilitate the deployment of multi-agent systems in both simulated and real environments while remaining agnostic to the underlying hardware. Various high-level controllers, such as formation control and flocking, have been explored, and integration with our motion capture system has been achieved, though not presented in this thesis.

Bibliography

Contrôle Partagé avec Assistance Haptique pour Systèmes Multi-Robots / Haptic Shared Control for Multi-Robot Systems

Les systèmes multi-agents offrent des avantages en termes de flexibilité et de redondance, permettant la répartition efficace des tâches complexes entre plusieurs robots. Cependant, leur déploiement exige des stratégies avancées de contrôle afin d’assurer une coopération optimale entre agents autonomes et opérateurs humains. Nous proposons une modification de l’algorithme de maintenance de connectivité globale afin d’intégrer des mesures réalisées dans le repère propre de chaque agent, avec un capteur à champ de vision limité. Cet algorithme, assurant le contrôle autonome du système multi-robots, est combiné à différentes stratégies de supervision adaptative, offrant à l’utilisateur un contrôle à échelle variable sur l’ensemble du groupe : (i) Contrôle d’agent unique (S) : l’opérateur pilote un robot spécifique, (ii) Contrôle par ancrage (P) : l’opérateur impose des contraintes à plusieurs robots pour influencer le groupe, (iii) Contrôle par barycentre (C) : l’opérateur agit sur le centre de masse du système multi-robots pour guider la flotte. Associées à l’algorithme de maintenance de connectivité, ces méthodes garantissent le respect des contraintes inter-agents tout en fournissant un retour haptique à l’utilisateur sur l’état de connectivité du groupe. L’efficacité de ces approches a été évaluée dans une étude utilisateur comparant les performances avec et sans retour haptique à travers plusieurs scénarios adaptés aux différents modes de contrôle. Multi-agent systems offer significant advantages in terms of flexibility and redundancy, enabling the efficient distribution of complex tasks among multiple robots. However, their deployment requires advanced control strategies to ensure optimal cooperation between autonomous agents and human operators. We propose a modification of the global connectivity maintenance algorithm to incorporate measurements taken in agent’s local frame, using sensors with limited fields of view. This algorithm, responsible for the autonomous behavior of the multi-robot system, is combined with different adaptive supervision strategies, allowing the user to exert control at varying scales over the entire group. (i) Single Agent Control (S): The operator directly controls a single robot, (ii) Pinning Control (P): The operator imposes constraints on multiple robots to influence the group, (iii) Centroid Control (C): The operator acts on the system’s center of mass to steer the entire fleet. In conjunction with the connectivity maintenance algorithm, these methods ensure compliance with inter-agent constraints while providing haptic feedback to the user regarding the system’s connectivity state. The effectiveness of these approaches was evaluated through a user study, comparing performance with and without haptic feedback across multiple scenarios tailored to the different control modes.