WorksComputer Science
Technical Report of Mobile Manipulator Robot for Industrial Environments
Abstract
This paper presents the design and implementation of Auriga Robotics' mobile manipulator robot, developed at the Robotics and Intelligent Automation Laboratory, Department of Electrical Engineering, Shahid Beheshti University. Hereafter, this platform is referred to as the @Work Robot. The robot targets industrial tasks, with an emphasis on improving efficiency in repetitive or hazardous work environments. It uses a 4-wheel Mecanum drive for omnidirectional movement and a 5-degree-of-freedom manipulator arm with a 3D-printed gripper for object handling and navigation. The electronics consist of custom PCB boards built around ESP32 microcontrollers, together with an NVIDIA Jetson Orin NX that handles high-level processing and real-time system control. On the software side, Hector SLAM provides mapping and localization, A* handles path planning, and YOLO performs real-time object detection. These modules draw on fused sensor data to deliver robust navigation and reliable collision avoidance. Overall, the system shows effective hardware-software integration, with consistent performance in autonomous navigation, object handling, and task execution in industrial-like settings.
Introduction
Robotics has become a core part of modern industrial settings, offering practical ways to optimize tasks and boost productivity. Within this context, @Work Robots work alongside human operators, helping create safer and more efficient workspaces. These robots can be deployed for a range of tasks, including production line support, transport in constrained spaces, object handling, and jobs that are difficult or unpleasant for humans to perform. This growing range of applications reflects the increasing importance of autonomous mobile manipulation in the future of industrial automation.
The goal of this project is to build a modular, highly adaptable platform that addresses these operational challenges — a versatile testbed that bridges academic research in mobile manipulation with practical, scalable industrial deployment.
Auriga Robotics at Shahid Beheshti University developed an autonomous mobile manipulation platform for the RoboCup @Work league, shown in Figure 1. The robot is built to handle perception, navigation, manipulation, and real-time decision-making. The project is part of the team’s ongoing research in intelligent robotics and autonomous systems, aimed at producing a modular, extensible platform for academic research and experimental work in industrial automation. Among the main contributions of this work are custom electronics for optimized power and sensor integration, and a vision-based dynamic obstacle mapping framework that improves navigation in complex environments.

Figure 1. Auriga’s @Work Robot.
This paper covers the design process and system integration of the Auriga Robotics @Work platform, with emphasis on its modular hardware and intelligent control architecture. It is organized as follows: Section 2 presents the hardware design, including the mechanical structure, electronics, and sensors. Section 3 describes the embedded firmware architecture — sensor integration and communication between the onboard processing units — and details the software infrastructure and ROS-based communication architecture. Section 4 discusses perception and computer vision algorithms. Section 5 covers robot control, navigation, and decision-making.
Hardware Description
Mechanical Design
Base Structure Design
The chassis and base structure are built mainly from aluminum profiles, giving a frame that is both lightweight and sturdy. The base measures 65 cm in length, 42 cm in width, and 56 cm in height, with a total weight of roughly 17 kg. Several 3D-printed components and plates were also added to the structure for a more integrated, customized design. Locomotion relies on four Mecanum wheels, each driven independently by a DC motor, giving the robot omnidirectional movement and high maneuverability. To cut down on task execution time, the object-handling area sits at the front of the robot, so the manipulator never has to rotate backward during pick-and-place operations — a choice that meaningfully improves efficiency and shortens task duration.
Manipator Design
The manipulator arm has 5 joints and 3 links, ending in a purpose-built gripper, shown in Figure 2. It reaches up to 65 cm and can carry a payload of roughly 450 g. The first three joints provide rotational motion about the robot’s principal axes, driven by three worm gearboxes powered by DC motors that supply enough torque and positioning precision. The fourth and fifth joints control gripper orientation using servo motors, letting the manipulator hold objects at different angles. Two more servo motors actuate the gripper fingers for grasping and releasing.
Figure 2. Design and implementation of the robotic gripper: (a) Fabricated prototype; (b) 3D model.
Electrical Design
Auriga’s @Work platform relies on two main custom electronic boards: one controls the manipulator arm, the other manages base movement. Keeping them separate allows the arm and base systems to be tested and debugged independently, which simplifies development and maintenance. The system also includes custom ESP32 [1] boards and a support board with op-amps and other ICs to improve the efficiency of the main boards, covered in detail below. An emergency button is wired into the main board for operational safety. Both main boards draw power from a four-cell 16000 mAh lithium-polymer battery, with voltage regulators supplying the correct operating voltage to each circuit.
Body Main Control Board
On this board, shown in Figure 4, LM2596 switching regulators supply several voltage levels: 11 V for the encoders, 12 V for the NVIDIA Jetson Orin NX, and 5 V for the ICs, microcontrollers, range-measurement sensors, gyroscopes, and CAN module. The gyroscopes and range-measurement sensors talk to the microcontroller over , while data between the microcontroller and the Jetson Orin NX moves over a UART serial connection. The DC motors are driven by the HW-039 motor driver, rated for 6–27 V DC and a continuous current of 43 A, which suits high-power applications well. We also implemented a dedicated method for motor direction control that uses a single ESP32 pin to manage the direction of all four wheel motors; its functional simulation appears in Figure 3a, and the fabricated PCB (top and bottom layers) is shown in Figure 3b and Figure 3c. The PWM signal from the ESP32 is first converted into an analog voltage, then digitized using op-amps and a resistor-divider network to set up a stable reference. Two 74LS148 encoder ICs extract a 4-bit value from the resulting 16-bit range, giving up to 16 distinct directional states for the motors. These bits then pass through a 7404 NOT IC and a MUX/DeMUX IC to set the motor directions. The same approach is used to read data from the VL53L1X sensors through a MUX.
Figure 3. PWM-to-binary conversion system: (a) Simulation environment demonstrating the proposed method, (b) Top layer of the fabricated PCB, (c) Bottom layer of the fabricated PCB.

Figure 4. Main control board PCB.
The emergency button on the body board can safely disable and re-enable the motors. Its safety circuit uses an IRF840 MOSFET and relays to reliably cut power when needed. Circuit design, PCB layout in Altium Designer, and hardware assembly were all carried out in-house by the electronics team.
Arm’s Board
On this board, shown in Figure 5, three LM2596 switching regulators supply different voltage levels: 8 V for one servo motor and three encoders, 6 V for three servo motors, and 5 V for the microcontroller and logic ICs. An AMS1117 regulator supplies an additional 3.3 V logic voltage for the DC motor driver. The board is built around a custom ESP32-based control unit, shown in Figure 6. Motor actuation on the arm is handled by two HW-039 drivers and one L298N driver. The board also includes three relays for the emergency button system, four BJT transistors for switching, and a CAN module for communication with the main body board. Four servo motors are used here as well, driven using the same PWM-to-binary conversion method described above. To save ESP32 GPIO pins, an 8:1 analog multiplexer reads the microswitch signals used as reference inputs for motion initialization.

Figure 5. Arm’s board PCB.

Figure 6. Custom-designed ESP32 PCB.
Sensors
Intel RealSense D435i Depth Camera. The Intel RealSense D435i [2] is an RGB-D camera that handles depth perception and visual data acquisition, feeding synchronized RGB and depth data to the robot’s main processor for perception and scene understanding.
RPLIDAR A1. The RPLIDAR A1 is a time-of-flight LiDAR that performs a 360-degree laser scan to detect surrounding obstacles. Mounted at the front of the robot, it is used mainly for mapping, navigation, and physical obstacle detection.
Rotary Encoder. We use the E38S6G5-100B-G24N rotary encoder, which measures shaft rotation to estimate angular displacement and velocity. It is used on both the arm and the mobile base to track joint positions, wheel velocities, and overall system state.
Range Sensor. The VL53L1X [3] is a time-of-flight distance sensor that covers LiDAR blind spots, improving collision avoidance during lateral and backward movement thanks to its speed and measurement accuracy.
Gyroscope. Robot orientation is measured using both CMPS14 and MPU6050 sensors. Their outputs are fused for a more reliable angle estimate, which improves localization accuracy.
Load Cell. A load cell in the gripper measures grip force, with a resolution of 0.5 grams and a threshold of 10 grams. A sudden drop in force signals possible slippage or loss of the object. It also enables controlled force application, which keeps stress on the servo motors low and extends their lifespan.
Embedded Firmware, Communication, and ROS Architecture
Embedded Firmware and Design
The platform relies on three main processing units: one NVIDIA Jetson Orin NX module and two ESP32 microcontrollers. This split is meant to improve processing efficiency, keep subsystems in sync, and simplify overall data management.
The NVIDIA Jetson Orin NX acts as the high-level computing platform, running the perception and control algorithms; its computational headroom lets it comfortably keep up with the system’s processing demands.
Of the two embedded controllers, one sits in the lower chassis, where it manages communication between subsystems, handles data exchange, and gathers measurements from onboard sensors; in the system architecture this is the lower-level module referred to as the Navigation subsystem. The second ESP32 controller is dedicated to the robotic manipulator and is called the Arm module. It stays in contact with the Navigation subsystem, drives the arm actuators, and reads joint position feedback from the rotary encoders to keep motion under precise control.
Embedded Firmware and Communication
The embedded subsystem handles sensor data acquisition, deterministic communication between units, and real-time actuator control, with continuous, low-latency data exchange keeping the whole system in sync.
The Navigation controller acts as the central communication hub in the embedded architecture. It gathers sensor measurements and receives status updates from the ARM unit, then sends this data to the Jetson module over UART at 115200 bps, as shown in Figure 7. In the other direction, high-level commands from the Jetson go to the Navigation unit, which either executes them directly or passes the relevant commands on to the ARM controller. Communication between the Navigation and Arm units runs over CAN, chosen for its resistance to electrical noise and suitability for reliable real-time embedded communication.

Figure 7. Hardware architecture and communication block diagram of the robotic system, illustrating the interconnection between the Nvidia Jetson, ESP32 controllers, sensors, and actuators.
Both embedded controllers run FreeRTOS-based firmware to make efficient use of resources, avoid task blocking, and guarantee deterministic scheduling. On the Arm controller, one task handles PID-based motor control while a separate task manages communication with the Navigation unit and the data it exchanges. The Navigation controller follows a similar pattern: one task acquires and preprocesses sensor data before sending it to the Jetson, another manages communication with the Arm unit, and a third runs PID-based velocity control for the navigation motors.
Sensor and Actuator Architecture
A range of sensors provides situational awareness and monitors the robot’s own state. For distance to obstacles, the LiDAR is supplemented with optical time-of-flight sensors (VL53L1X). Since these communicate over C, a hardware multiplexer managed by the Navigation controller allows them to run simultaneously. To keep pin usage on the controller low, a PWM-to-binary counter module is used as well.
Inertial measurements come from the MPU6050 and CMPS14 sensors, which track the robot’s angular acceleration and orientation and communicate directly with the Navigation unit.
Several microswitches handle manipulator calibration, with their states sent to the Arm controller. Rotary encoders measure the angular position and velocity of the robot’s actuators, feeding the control loops.
All control computations run on the respective controllers, which then send the resulting commands to the motor drivers for execution.
Robot Communication, Workflow Infrastructure and ROS Architecture
This section covers the software infrastructure behind the @Work Robot’s operation, from receiving a task order from the referee box down to executing low-level actuator commands on the embedded hardware. The system runs on the Robot Operating System (ROS) Noetic [4] under Ubuntu 20.04 LTS, on an NVIDIA Jetson Orin NX onboard computer. The description below follows the system’s data flow in the order it occurs during a competition run: task reception, decision-making, cross-machine communication, task execution and navigation, perception integration, and finally low-level actuation over UART. The internal logic of each subsystem — how a path is planned, how an object is detected, how a manipulation is performed — is covered in its own dedicated section elsewhere in this report; here we focus purely on the architectural glue connecting these subsystems.
Task Reception and Multi-Master Communication
Task Reception via atwork_commander. During a competition run, task orders come from the referee box via the atwork_commander [5] framework, which publishes a structured task description containing an ordered set of object manipulation subtasks, each with a source location, destination location, and object identifier. A dedicated decision-making node (decision_making) receives this description by subscribing to /atwork_commander/object_task. Once the task message arrives, the node unsubscribes from the topic, since only one task order is issued and processed per run.
Multi-Master Communication Bridge and atwork_commander Integration. Task instructions for the @Work Robot come from the RoboCup @Work referee box (refbox), implemented as the atwork-commander ROS package. In the competition setup, the refbox runs its own roscore on the referee’s machine (Machine 1), while each competing robot runs an independent roscore on its onboard computer (Machine 2). Getting task messages from the refbox to the robot, and robot state back in the other direction, therefore means bridging two independent ROS masters.
The atwork-commander package originally used multimaster_fkie [6] as its multi-master communication backend. That library has since been deprecated and no longer works under ROS Noetic. Since no maintained drop-in replacement existed that preserved compatibility with atwork_commander_msgs and the expected topic interface, we designed and built a custom, lightweight bridge from scratch to restore full interoperability between the refbox and the robot.
System Architecture. All components of the bridge run on Machine 1 (the referee’s machine). The system consists of two C++ ROS nodes—sc and orphan—and a Python supervisor script (env_script.py); their relationships are shown in Figure 8.
Figure 8. Multi-master ROS bridge architecture. All components reside on Machine 1. sc connects to the local atwork-commander roscore; orphan connects to the remote robot roscore on Machine 2. Topic data is relayed between them over Machine 1’s loopback network. The env_script.py supervisor manages the lifecycle of orphan, relaunching it against a new robot roscore on demand.
The sc node. The sc node registers as a ROS node under the atwork-commander’s roscore on Machine 1. It subscribes to the refbox’s output topics—/atwork_commander/task and /atwork_commander/object_task—while also listening for incoming data on a dedicated TCP port. When a message arrives on either subscribed topic, sc serializes it with the standard ros::serialization API and forwards the binary payload to the orphan node over Machine 1’s loopback network. Going the other way, sc receives serialized RobotState data from orphan on its listening port, deserializes it, and republishes it to /atwork_commander/robot_state on the atwork-commander roscore.
The orphan node. The orphan node also runs on Machine 1, but registers under the robot’s roscore on Machine 2 by launching with ROS_MASTER_URI pointed at the robot’s network address. This is the key design decision behind the bridge: since a running ROS node cannot switch its hosting roscore after initialization, orphan is started fresh for each robot connection with the right master URI, so it looks like a native participant in the robot’s ROS graph. Once running, orphan receives the serialized payloads that sc forwards over the loopback network, deserializes them back into their original message types, and publishes them to the topics the robot expects: /atwork_commander/task and /atwork_commander/object_task on Machine 2’s roscore. At the same time, it subscribes to /atwork_commander/robot_state on the robot’s roscore and forwards any incoming RobotState messages back to sc, closing the loop on the bidirectional bridge. Table 1 summarizes the three topic streams and their corresponding TCP ports.
| Topic | Message Type | Direction | TCP Port |
/atwork_commander/task | Task | Refbox → Robot | 9673 |
/atwork_commander/object_task | ObjectTask | Refbox → Robot | 9672 |
/atwork_commander/robot_state | RobotState | Robot → Refbox | 9671 |
Table 1. ROS topics bridged between the refbox roscore (Machine 1) and the robot roscore (Machine 2)
The env_script.py supervisor. During competition, the referee has to send the same generated task to each competing robot in turn. Since a node can’t migrate between roscores at runtime, a fresh instance of orphan has to be spawned for each robot connection. The env_script.py script automates this: it asks the referee for the IP address of the target robot’s roscore, polls for reachability with rosgraph.is_master_online(), and once confirmed, launches orphan as a subprocess with ROS_MASTER_URI set accordingly. It keeps monitoring the subprocess, and if the robot roscore drops offline, the connection is lost, or the referee disconnects manually, the script kills the current orphan instance and asks for the next robot’s IP, ready to repeat the process for the next competitor.
Wire Protocol and Reliability. All three TCP channels use the same length-prefixed framing scheme. Each transmitted unit consists of a 4-byte big-endian payload length followed immediately by the ROS-serialized message bytes:
Receivers implement a two-state parser (READ_HEADER READ_PAYLOAD) that accumulates bytes across successive read() calls before committing to deserialization. This handles TCP stream fragmentation correctly regardless of MTU or scheduling jitter, so no message is lost or corrupted in transit.
Each outbound TCP channel is backed by an AsyncPortSender object, which owns a dedicated std::thread that drains a mutex-protected message queue over a persistent TCP connection. This keeps ROS subscription callbacks decoupled from network I/O, so transient link latency doesn’t stall the ROS spin loop. Connections are checked with periodic keepalive probes; on failure, the sender tears down and re-establishes the connection automatically, recovering from transient network interruptions without a node restart.
Task Execution Pipeline
Decision-Making Node. The decision_making node takes the parsed task order and works out an optimized sequence of pick-and-deliver operations, factoring in station locations, object types, and estimated travel and manipulation times. We don’t detail the internal optimization logic here, since it’s not part of the ROS communication architecture. From an infrastructure standpoint, this node acts as a single-shot translator: it converts the atwork_commander task representation into an internal, robot-specific one, and republishes the result once the full task order has been resolved.
The resulting task sequence is published on a custom topic, decision, using a nested custom message type (estrings, itself made up of a repeated string-array message estring) that encodes each subtask as a row of string-serialized fields (object identifier, task status, workstation type, and target pose). This message is consumed exactly once, by the navigation and task-execution node described in § 3.2.2, marking the handoff between the decision layer and the execution layer.
Navigation and Task-Execution Node. A single node, main, orchestrates the robot centrally. It subscribes to the decision topic and keeps an internal task queue built from the received message. This node runs a finite-state machine that governs the robot’s high-level behavior — navigation, alignment, manipulation triggering, inter-task transitions — and it’s the point where all other subsystems, including localization, perception, and low-level actuation, come together. Because this node touches every other subsystem in the architecture, its outward-facing interfaces are described individually in the following subsections, while its internal state-machine logic and control algorithms are covered elsewhere in this report.
Sensor and Perception Interfacing
Localization and Low-Level Sensor Interfacing. Low-level actuation and sensor acquisition are delegated to a dedicated microcontroller, reached through a Python bridge node, esp_comms, which talks to the microcontroller over a UART serial link (/dev/ttyTHS0) using a fixed-format ASCII packet protocol. This node handles two jobs at once: serializing outgoing actuator commands into the packet format the microcontroller expects, and deserializing incoming sensor telemetry back into ROS messages.
On the outgoing side, esp_comms subscribes to motor4 (wheel actuation commands) and arm_datas (manipulator commands), both published by main, and merges them into a single serial packet sent to the microcontroller each cycle. On the incoming side, it parses the microcontroller’s serial reply and republishes the extracted data as three separate topics: encoder4 (wheel encoder readings), vl (time-of-flight range array), and cmps (compass heading), all consumed by main for localization. The microcontroller’s arm-response status isn’t published as a topic at all; instead, it’s written directly to the ROS parameter server (arm_response) and polled by main each control cycle. We use this parameter-server channel deliberately, as a lightweight, low-latency status flag rather than a queued topic, since only the most recent value ever matters to the state machine.
The main node fuses the encoder, compass, and range data internally to maintain the robot’s global pose estimate; this fusion logic is covered in the localization section and isn’t repeated here.
Perception Integration. Visual perception is handled by two cooperating nodes, camera_node and vision_node, both part of the arm package. camera_node initializes and continuously polls an Intel RealSense D455 camera, and instead of publishing raw color and depth frames over ROS topics, it writes them directly into two POSIX shared-memory segments (image_frame and depth_frame). vision_node maps the same shared-memory segments and reads the latest available frames directly, skipping ROS message serialization entirely for this high-bandwidth path. This mirrors the shared-memory pattern used elsewhere in the team’s software stack for large image payloads.
Beyond camera-based perception, vision_node also manages a 2D LiDAR unit (RPLidar) on its own, without a separate ROS driver node; the LiDAR is opened directly as a serial device from within vision_node on a dedicated background thread, and its scan data is consumed internally by the same node rather than published on a topic. This keeps all mapping-relevant sensing — camera and LiDAR — inside a single perception process.
vision_node and main coordinate through one topic plus the ROS parameter server. vision_node publishes its detection output — object, hole, or station-relative measurements, depending on the active mode — on /vision_out as a two-dimensional array packed into a single Int32MultiArray message with explicit layout dimensions. main, on the other hand, doesn’t command the perception mode through a topic; instead it writes the desired operating mode and target object identifier, along with the robot’s current global pose, to the parameter server (mode, obj, x, y, theta), which vision_node polls every cycle. This keeps the perception node a passive follower of the task-execution state machine, with no dedicated command topic or service needed.
System Integration and Summary
System Bring-Up. The system comes up through two independent ROS launch files rather than one unified configuration. The first, esp.launch, starts the esp_comms bridge and the main navigation/task-execution node together, reflecting how tightly they’re coupled through the motor, encoder, and arm-command topics described in § 3.2.3. The second, vision.launch, starts camera_node and vision_node together, reflecting their shared-memory coupling from § 3.2.3. This split lets the perception subsystem and the navigation/actuation subsystem be started, restarted, and debugged independently during development and testing.
Summary of the Communication Architecture. Table 2 summarizes the nodes described in this section, and Table 3 summarizes the distinct communication mechanisms used across the system. Figure 9 shows the resulting node and topic graph, and Figure 10 illustrates the end-to-end data flow from task reception to actuation described throughout this section.
| Node | Package | Primary Function |
| decision_making | decision_making | Receives the atwork_commander task order and publishes an optimized, robot-specific task sequence (single-shot). |
| main | navigationing | Central finite-state-machine orchestrator: navigation, localization, and coordination of perception and actuation. |
| esp_comms | navigationing | Serial bridge to the drivetrain/sensor microcontroller over UART; translates between ROS topics and the serial packet protocol. |
| camera_node | arm | Acquires synchronized RGB-D frames from the RealSense D455 and exposes them via shared memory. |
| vision_node | arm | Performs detection and LiDAR-based mapping; publishes results on /vision_out and follows commanded state via the parameter server. |
Table 2. Summary of ROS nodes in the @Work Robot’s software architecture
| Mechanism | Used Between | Purpose |
| ROS topics (custom msgs) | decision_making → main | One-shot transfer of the resolved task sequence (decision). |
| ROS topics (std_msgs) | main ↔ esp_comms | Continuous actuation commands and sensor telemetry (motor4, arm_datas, encoder4, VL, cmps). |
| ROS topics (std_msgs) | vision_node → main | Continuous perception output (/vision_out). |
| ROS parameter server | main ↔ vision_node | Low-latency perception mode/target/pose commanding and state polling. |
| ROS parameter server | esp_comms → main | Arm-response status flag. |
| POSIX shared memory | camera_node → vision_node | High-bandwidth RGB and depth frame transfer, bypassing ROS serialization. |
| Direct serial (non-ROS) | vision_node ↔ RPLidar | Direct sensor acquisition without an intermediate driver node. |
| Direct serial/UART (non-ROS) | esp_comms ↔ microcontroller | Fixed-format ASCII packet protocol carrying actuation commands and sensor telemetry. |
Table 3. Communication mechanisms used across the @Work Robot’s software stack
Figure 9. Nodes and communication graph of the @Work Robot’s ROS architecture, showing publish–subscribe relationships between decision_making, main, esp_comms, camera_node, and vision_node.
Figure 10. End-to-end data flow from task reception via atwork_commander through decision-making, task execution, perception, and low-level actuation.
Computer Vision and Perception
Mapping Subsystem
The mapping subsystem pairs a LiDAR sensor with the Hector SLAM [7] algorithm to simultaneously estimate the robot’s pose and build a two-dimensional map of the environment.
A key feature of Hector SLAM is that it doesn’t depend on wheel odometry or IMU measurements, relying solely on laser scan data for localization and map construction. This makes the system robust to odometry drift and sensor noise.
The occupancy grid map is updated continuously across mission phases, including when it encounters static and dynamic obstacles. This keeps localization consistent and navigation reliable throughout task execution.
We chose Hector SLAM for its sensor independence, its accuracy in small- to medium-scale indoor environments, and its computational efficiency, which suits real-time operation on embedded hardware.
Image Processing and Computer Vision
The computer vision system is built around six primary operating modes, triggered by control signals from the robot’s main module. It processes visual data and returns outputs for decision-making and task execution.
Neural Network Development
For real-time perception, we built a custom object detection and segmentation framework based on the YOLO [8] architecture, optimized for the target hardware.
We built a dedicated dataset to ensure robustness and generalization across varied environmental conditions. Images were captured directly from the robot’s operational environment for realistic training scenarios, and synthetic images with varied backgrounds and lighting were generated to broaden generalization further. Pixel-level manual annotation was also carried out to support segmentation and improve detection accuracy.
Combining real, synthetic, and carefully annotated data gives the framework high detection precision and reliable segmentation, enabling real-time perception in dynamic environments. Figure 11 shows the system’s real-time performance, with simultaneous detection, tracking, and physical dimension estimation of objects in the robot’s surroundings. The values shown in the figure are the estimated real-world width (RW) and distance (D) from the sensor.

Figure 11. Real-time object detection and classification.
Final Dataset Statistics. The final dataset has 61,285 training images and 3,879 validation images, for 65,164 annotated samples in total. Roughly 6,000 of these (about 9%) are original real-world images, with the remaining 59,164 generated through data augmentation for diversity. The dataset spans varied lighting conditions, object orientations, and backgrounds to improve robustness and generalization. All samples were manually checked for annotation consistency and segmentation accuracy.
Model Architecture and Training.
Base Model. YOLOv11s-Seg (small segmentation variant), chosen to balance accuracy against computational cost for real-time deployment.
Task. Simultaneous object detection and instance segmentation, giving both bounding-box localization and pixel-level object mask prediction.
Loss Functions. Training combines bounding box regression loss, classification loss, objectness loss, and segmentation loss to jointly optimize localization accuracy and mask quality.
Training Strategy. The model was trained to convergence using performance-based early stopping, while monitoring mean Average Precision (mAP), precision, and recall to keep generalization stable.
Experimental Results. The model trained for 50 epochs at a batch size of 32. The training and validation curves show stable convergence across all loss components — box, classification, segmentation, and distribution focal loss — as shown in Figure 12. Both training and validation losses drop consistently with no sign of overfitting, which points to good generalization.

Figure 12. Training and validation loss curves, illustrating convergence across all loss components.
The model reaches a precision and recall of roughly 0.99 for both bounding box and segmentation outputs. The evaluation results also show an mAP50 near 0.995 and an mAP50-95 of about 0.98 for bounding boxes and 0.93 for segmentation masks.
The final deployed model runs at 17 FPS on the target hardware, confirming it’s fast enough for real-time autonomous operation.
Operational Modes
Mode 1: Main Detection. This mode detects primary task-related objects using the YOLOv11s-Seg network, identifying both basic industrial components (e.g., bolts, nuts, rods) and more complex tools (e.g., drills, motors, bearings).
Output Parameters. By combining depth information with camera intrinsic parameters, the system computes, for each detected object: real-world dimensions (width/height), object orientation, the three-dimensional distance between the gripper and object center (), and the gripper opening distance needed for a successful grasp.
Post-Processing Classification. Because some objects look visually similar, an additional dimension-based classification step assigns subcategories based on estimated physical measurements.
For example, a detected white rod is classified as a Large Rod if its estimated width exceeds 35 mm, and as a Small Rod otherwise. Spatial filtering constraints — such as a 1 meter maximum detection range — also help filter out irrelevant background detections.
Figure 13 illustrates this post-processing step, where objects are sorted into subcategories (e.g., small vs. large) based on their and as defined above.

Figure 13. Subcategory classification based on estimated physical width (RW) and distance filtering (D).
Some objects start out under broader parent classes and get refined into specific child classes during post-processing. This hierarchical approach is needed because certain objects (e.g., large and small rods or nuts) look too similar for the network to reliably tell apart by appearance alone. Final categorization instead relies on estimated physical properties such as real-world dimensions and aspect ratio.
Mode 2: Precision Placement. This mode detects insertion points or empty target spaces using a lightweight YOLOv8n segmentation model, chosen for faster inference while keeping accuracy adequate.
Geometric Analysis. Since insertion points (e.g., rod slots and nut recesses) can look similar from a top-down view, appearance-based classification alone isn’t enough. Detected shapes are first sorted into four geometric classes — Screw, Hexagon, Polygon, and Rectangle — and then refined based on aspect ratio and real-world dimensional thresholds.
Classification Rules. Objects are classified by their geometric characteristics. Rectangular shapes longer than 70 mm are identified as rod slots, and shorter ones as nut sockets. Nut sockets are further split by width: those under 35 mm are small nuts, and those above are large nuts. Figure 14 shows this classification process, with objects labeled small or large based on these real-world dimensional thresholds ( and ).

Figure 14. Geometric classification of objects based on real-world length (RL) and width (RW) thresholds for precision placement.
Mode 3: Navigation. This mode continuously updates the environment map with both virtual and physical obstacles, enabling dynamic path planning and safe navigation.
A. Virtual Obstacles (Tape Detection). Red and yellow tapes are detected with the YOLOv11s segmentation model. To account for the offset between the camera and LiDAR coordinate frames, a trigonometric transformation is applied: the horizontal distance is calculated with the Pythagorean theorem from the camera height and measured depth, then corrected for the camera–LiDAR spatial offset.
Each detected obstacle is then projected onto the global map with its estimated size and orientation, and a corresponding flag is sent to the control unit. As shown in Figure 15, the system segments the tape obstacles and computes their real-world dimensions and distance, which are needed for accurate spatial projection and map updating.

Figure 15. Segmentation and dimension estimation of virtual tape obstacles for global map projection.
B. Physical Obstacles (LiDAR Detection). Physical obstacles are found by comparing real-time LiDAR point cloud data against the original static map. The DBSCAN [9] clustering algorithm groups nearby points and filters out noise, with the maximum distance between neighboring points (epsilon, ) and the minimum cluster size (MinSamples) set accordingly.
Newly found clusters are flagged as dynamic obstacles and added to the updated map, and the control module is notified to trigger replanning. Figure 16 shows the resulting local map, with LiDAR-detected clusters processed and represented as distinct physical obstacles for real-time navigation.

Figure 16. Real-time obstacle representation in the occupancy grid map based on LiDAR data and DBSCAN clustering.
Mode 4: Zero Station and Physical Station. These modes align the robot with the center of a target station. Detection starts by extracting the front edge of the station with Canny edge detection, followed by line fitting using RANSAC (Random Sample Consensus) [10].
We chose RANSAC because experiments showed it outperforms the traditional Hough Line Transform, with better robustness to noise and outliers while still reliably estimating the dominant line.
From this, the system computes both the translational displacement needed for lateral alignment and the angular offset needed for orientation correction. Figure 17 shows the edge detection and line fitting process on different stations, where the red line marks the estimated dominant edge and the blue point marks the calculated center for robot alignment.

(a) (a)

(b) (b)
Figure 17. Robot alignment process using Canny edge detection and RANSAC line fitting: (a) detection of the station front edge, and (b) center point estimation for translational and angular correction.
Mode 5: Emptiness Check. This mode checks whether a station has room for object placement. The system scans the station regions with a virtual circle of 5 cm radius, scaled by distance. When it finds an empty spot, the coordinates go to the control unit for the placement action that follows. Figure 18 shows this check, with a green circle marking a detected empty spot and a red circle marking an occupied region, preventing placement errors.

Figure 18. Emptiness check using a virtual scanning circle: (a) identification of an empty placement spot, and (b) detection of an occupied area.
Mode 6: Container Detection. This mode detects containers and reports their center coordinates to the control unit. The red and blue container classes are part of the main YOLO Segmentation V11s model and are only detected when the control system explicitly requests it. Once a container is found, the system computes the real-world distance from the gripper to the container center (X, Y, Z) and publishes it.
To keep measurements reliable, data is only published once the entire container is visible in the camera’s field of view, verified by width and height checks that rule out erroneous center calculations from partial views. Figure 19 shows the detection of red and blue containers, where the system checks real-world dimensions (e.g., 146 × 174 mm) to confirm the container is fully in view (labeled ‘Size:OK’) before publishing coordinates.

Figure 19. Real-time container detection and visibility verification for accurate center coordinate estimation.
Control and Planning Architecture
Localization and Mapping
The map produced by the Computer Vision module is exported as a text file representing a discrete two-dimensional occupancy grid, stored for later use in navigation and decision-making. Using this map together with the robot’s kinematic model and onboard sensor measurements — gyroscope and wheel encoder data — the robot’s global pose (position and orientation) is estimated in the world coordinate frame. This pose estimate and the occupancy grid then feed into path planning toward the target location.
The robot’s kinematic model is described by the following equations [11]:
This kinematic model, shown in Figure 20, relates the individual wheel velocities to the robot’s overall motion. For the four-wheel omnidirectional drive used here, global velocity breaks down into longitudinal, transverse, and angular components, letting the robot estimate its relative displacement (odometry) by integrating encoder data over time.

Figure 20. Kinematic representation of the mecanum-wheeled robot, showing the velocity vectors () for each wheel and the resulting motion vectors.
Decision-Making and Path Planning
The robot follows a structured decision-making and planning pipeline for task execution. Given a list of tasks, it first works out the optimal execution order with a dedicated decision-making algorithm, then a path planning algorithm computes an efficient route to each target location.
Decision-Making
The decision-making module evaluates all feasible orderings of task execution. Each candidate sequence is scored with a cost function that factors in path length and complexity, object type, target station type, number of empty slots, and the priority assigned to each task.
Sequences that cost more than previously evaluated candidates are pruned along the way, so the system converges on the most cost-efficient task sequence, as shown in the optimization workflow in Figure 21.

Figure 21. Flowchart of the task sequence optimization process, illustrating the iterative cost evaluation and neighbor sequence generation to achieve the most efficient execution plan.
The weights and coefficients for each cost parameter were tuned empirically through systematic experimentation to get robust decision-making performance.
Path Planning
The navigation system generates a collision-free trajectory from the robot’s current pose to the target location given by the Decision Making module. Since the competition environment is a known occupancy grid, global path planning uses the A* search algorithm [12], giving an efficient, near-optimal path around static obstacles. To speed things up [13], A*’s open set is implemented as a Min-Heap priority queue, cutting the time complexity of selecting the next node from to — a real gain for longer paths in particular. The node evaluation function is defined as follows.
where the heuristic term is the Euclidean distance to the goal,
and the safety penalty is given by where is the distance to the nearest obstacle and is a tunable coefficient. This safety term biases the planner toward paths with more obstacle clearance, keeping the robot closer to the center of corridors and reducing collision risk from localization or tracking errors. Once the optimal path is generated, it’s passed to the motion controller as the reference trajectory. A PID-based path tracking controller continuously minimizes position and heading errors as the robot follows the planned route. During execution, LiDAR measurements are monitored in real time for unexpected obstacles; if a potential collision is detected, the robot stops immediately, updates the navigation map, replans the path, and resumes. The whole navigation framework is implemented in C++ and runs in real time on the NVIDIA Jetson Orin NX, giving reliable, responsive autonomous navigation throughout the competition environment. The overall system architecture follows the design described in [14].
Manipulator Structure and Object Handling Algorithm
The robot’s 5-degree-of-freedom (5-DOF) manipulator is controlled by an ESP32 microcontroller. High-level target poses and Cartesian coordinates are computed on the Jetson Orin NX and sent to the ESP32 for low-level execution.
Once it receives the target pose, the manipulator computes the corresponding joint angles from its inverse kinematic model. These joint commands are then applied to reach the end-effector position and orientation needed for object manipulation.
The forward kinematic model used to find the end-effector’s position from the manipulator’s geometry and joint angles is given by:
Here and are the lengths of the manipulator links, and , , are the respective joint angles, as shown in Figure 22. These equations let the high-level controller map the manipulator’s internal state to the operational workspace coordinates.

Figure 22. Geometric model of the manipulator highlighting the joint angles and link structure used for kinematic calculations.
Object Placement Slots
As mentioned earlier, the robot has a set of object placement slots at its front. These slots move linearly along the horizontal axis (left and right). During pickup and placement, a predefined control algorithm positions the slots so the target slot sits directly in front of the manipulator.
Since the system always knows the exact position of each slot through this precise control, no extra scanning, perception, or processing is needed to find slot positions during pickup or placement. This cuts computational overhead and makes object-handling more efficient and reliable.
Object Picking
At the start of the picking procedure, the robot first checks it has enough storage capacity for a new object. If there’s room, the manipulator moves to a predefined hovering position above the target station.
Once it receives the precise pose of the target object from the perception module, the manipulator approaches and grasps it with the end-effector gripper. Real-time load-cell measurements then verify the grasp; once confirmed, the object is lifted and moved to the robot’s front storage compartment, where it stays secured until delivery.
Object Placement
During placement, the manipulator retrieves the object from the front storage compartment and begins the placement procedure. The system first identifies the available section of the target station for placement.
Once a free slot is found, visual data from the perception module gives the precise coordinates of the target location. From this, the manipulator computes the required joint configuration and aligns the end-effector with the placement area.
Finally, the gripper releases the object in a controlled manner, completing the placement.
Performance Evaluation
To keep the placement procedure reliable, we set specific operational constraints and performance metrics:
Operational Speed: Each manipulator joint can reach a maximum angular velocity of . To keep motion controllable and stable and to ensure safe operation in the lab, though, joint velocities are capped between and , depending on conditions.
Grasping Success Rate: We ran a series of grasping experiments on objects of various shapes, sizes, and surface properties to evaluate the manipulator’s performance, testing each object over 20 independent trials. A trial counted as successful if the robot could grasp, lift, and place the object at the target location without dropping it. Table 4 summarizes the resulting success rates.
| Object | Dimensions | Weight (g) | Trials | Successes | Success Rate (%) |
| Small Aluminum Profile | mm | 50 | 20 | 19 | 95 |
| Large Aluminum Profile | mm | 185 | 20 | 18 | 90 |
| Screw | M20 100 mm | 295 | 20 | 16 | 80 |
| Small Nut | M20 | 55 | 20 | 9 | 45 |
| Large Nut | M30 | 215 | 20 | 11 | 55 |
| Steel Axis | 68 M20 | 160 | 20 | 15 | 75 |
| Bearing | 40 17 | 180 | 20 | 8 | 40 |
| Motor 755 | 66.7 42 | 320 | 20 | 15 | 75 |
| Wrench | 144 20 | 75 | 20 | 9 | 45 |
| Wera Allen Key 8 mm | mm | 15 | 20 | 7 | 35 |
| Drill | mm | 17 | 20 | 10 | 50 |
Table 4. Experimental results of grasping and placement success rates for various industrial objects, categorized by dimensions, weight, and geometric complexity.
Author Contributions
| Name(s) | Role and Contribution Summary |
| Aria Dadnavi | Developed the post-processing pipeline, adaptive LiDAR–camera mapping system, and computer vision algorithms for object detection and pick-and-place perception. |
| Matin Jamshidi | Mechanical design and integration of the robot chassis, manipulator arms, and gripper mechanism, including fabrication, assembly, and structural optimization. |
| Kian Khatibi | Embedded firmware design, real-time sensor processing, actuator interfacing, and communication system implementation. |
| Shahriar Karimi Makhsous | ROS architecture and system communication design. |
| Saina Najafi Shabestari | Trained and optimized the instance segmentation model and implemented object detection algorithms for pick-and-place perception. |
| Seyed Sobhan Hosseini Hajibekandeh, Amirhossein Gholizadeh Behbahani | Developed the autonomous navigation, motion planning, manipulator control, and task execution algorithms. |
| Zahra Maham, Reyhane Mohasseli | Electronics hardware design, sensor integration, PCB development, and power management. |
| Ghazal Ghoreishi | Implemented computer vision algorithms for station alignment, robot motion correction, and pick-and-place perception. |
| Mohammad Hossein Moaiyeri | Team supervision, university liaison, and funding coordination. |
Table 5.
Acknowledgments
The authors wish to note that the submission of this manuscript was delayed due to prolonged disruptions to research activities caused by civil unrest and subsequent military conflict in Iran. These circumstances significantly restricted access to laboratory facilities and delayed manuscript preparation. The work presented here corresponds to the robotic system as completed in early 2025.
Additionally, the authors used AI language models to assist with English-language editing, LaTeX formatting, and manuscript preparation, owing to limited fluency in English. All technical content, experiments, data, and conclusions are the authors’ own, and the authors have reviewed the text and take full responsibility for its contents.
References
- [1]Espressif Systems, “ESP32 Series Datasheet,” 2023. Available: https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf.
- [2]Intel Corporation, “Intel RealSense Depth Camera D435i Datasheet,” 2019. Available: https://www.intelrealsense.com/depth-camera-d435i/.
- [3]STMicroelectronics, “VL53L1X Time-of-Flight Ranging Sensor Datasheet,” 2021. Available: https://www.st.com/resource/en/datasheet/vl53l1x.pdf.
- [4]M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “ROS: an open-source Robot Operating System,” in ICRA Workshop on Open Source Software, 2009.
- [5]RoboCup@Work Technical Committee, atwork-commander: RoboCup@Work Referee Box Software, available at https://github.com/robocup-at-work/atwork-commander.
- [6]Fraunhofer FKIE, multimaster_fkie: FKIE Multi-Master for ROS [Deprecated], available at https://github.com/fkie/multimaster_fkie.
- [7]S. Kohlbrecher, O. von Stryk, J. Meyer, and U. Klingauf, “A flexible and scalable SLAM system with full 3D motion estimation,” in IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), 2011, pp. 155–160.
- [8]G. Jocher, A. Chaurasia, and J. Qiu, “Ultralytics YOLO,” 2023. Available: https://github.com/ultralytics/ultralytics.
- [9]M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD), 1996, pp. 226–231.
- [10]M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
- [11]H. Taheri, B. Qiao, and N. Ghaeminezhad, “Kinematic Model of a Four Mechanum Wheeled Mobile Robot,” International Journal of Computer Applications, vol. 113, no. 3, pp. 6–9, Mar. 2015. Available: https://www.ijcaonline.org/archives/volume113/number3/19805-1691/.DOI
- [12]P. E. Hart, N. J. Nilsson, and B. Raphael, “A Formal Basis for the Heuristic Determination of Minimum Cost Paths,” IEEE Transactions on Systems Science and Cybernetics, vol. 4, no. 2, pp. 100–107, Jul. 1968. Available: https://doi.org/10.1109/TSSC.1968.300136.
- [13]J. Goyal and K. Nagla, “A New Approach of Path Planning for Mobile Robots,” in Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2014, pp. 863–867. DOI: 10.1109/ICACCI.2014.6968528.
- [14]E. A. Khalili, K. Ghasemzadeh, H. Gohari, M. Jafari, M. Jamshidi, M. Khaksar, A. AkramiFard, M. Hatamzadeh, S. Sadeghi, and M. H. Moayeri, “Technical Report of Mobile Manipulator Robot for Industrial Environments,” arXiv preprint arXiv:2409.06693, Sep. 2024. Available: https://arxiv.org/abs/2409.06693.




