WorksComputer Science
Adversarial Tracking: Attacks on Image Sensors for Object Tracking in Autonomous Vehicles
Abstract
Object tracking systems estimate the motion of surrounding vehicles by linking detected objects across video frames. In particular, Tracking-By-Detection (TBD) systems combine object detectors with motion prediction and data association to maintain object identities over time. However, neural network-based detectors are vulnerable to adversarial attacks that perturb input images, and their impact on downstream trajectory estimation is not fully characterized by detection accuracy alone. Therefore, this paper evaluates image-specific and image-agnostic attacks on YOLOv3 and YOLOv4 detectors paired with SORT, Deep SORT, Strong SORT, and OC-SORT tracking systems. The evaluation uses CARLA simulator and measures tracking quality through Multiple Object Tracking Accuracy (MOTA), Higher Order Tracking Accuracy (HOTA), and Identification F1 (IDF1). Under image-specific attacks, HOTA for YOLOv3-based trackers declines from 53.45-56.19% to 14.85-17.03%, while MOTA reaches -772.60% in the worst case. YOLOv4-based trackers exhibit greater robustness, while HOTA still declines to 25.06-26.86%. In addition, image-agnostic attacks can reduce the tracking accuracy using a single Universal Adversarial Perturbation (UAP), without generating a separate perturbation for individual frames.
Introduction
Object tracking is a fundamental perception task in autonomous driving that estimates the locations and identities of surrounding objects over time [1]. Camera-based object tracking includes Single-Object Tracking (SOT) and Multi-Object Tracking (MOT). SOT follows one specified target, whereas MOT follows several objects and maintains a separate tracklet for each [2].
Although deep neural networks have substantially advanced the tracking performance, they are vulnerable to adversarial attacks that perturb input images [3]. For example, Tracking-By-Detection (TBD) systems combine neural object detectors with motion prediction and association across consecutive frames. Some TBD systems rely on pre-trained object detectors, and perturbations generated against widely used object detection models, such as YOLO [4, 5, 6], may transfer to tracking systems. Therefore, the tracking performance of TBD systems heavily rely on the accuracy of object detectors. However, detection errors do not necessarily translate directly into tracking errors, as the data association stage may reject inconsistent detections [3]. Nevertheless, persistent detection errors can disrupt data association, leading to fragmented trajectories [7], as shown in Figure 1.

Figure 1. Adversarial Attacks on Object Tracking: Adversarial perturbations disrupt Multi-Object Tracking (MOT) by fragmenting continuous tracklets and generating spurious tracklets.
Therefore, a systematic evaluation at the tracking level is necessary to characterize the impact of adversarial attacks on autonomous driving. The contributions are as follows.
CARLA Dataset: An open-source framework is developed to generate synthetic autonomous driving dataset using CARLA Simulator [8].1
Tracking Benchmark: A benchmark is established to evaluate representative real-time tracking algorithms, including SOT [9], Deep-SORT [10], Strong-SORT [11], and OC-SORT [12], in combination with three object detectors: YOLOv3, YOLOv4, and YOLOX.
Adversarial Attacks: Image-specific and image-agnostic attacks against TBD systems are evaluated using MOTA, IDF1, and HOTA.2
Background
Object Tracking
Camera-based Multi-Object Tracking (MOT) methods generally follow two approaches, Joint Detection and Tracking (JDT) and Tracking-By-Detection (TBD). JDT methods integrate detection and tracking, whereas TBD methods use a modular framework with an object detector, state estimation, an optional feature extractor, and data association [1]. Figure 2 summarizes these components and their roles in online tracking.

Figure 2. Common components of online tracking methods based on the TBD framework.
Object Detector: The object detector identifies objects in each camera frame and provides bounding boxes, confidence scores, and predicted classes. The tracking stages then associate detections across frames to maintain object identities.
State Estimation: To maintain object identities, SORT-like trackers commonly use a Kalman filter to predict the next state of each track [9]. Under a constant-velocity model, the state is represented by
where is the box center, is its scale, is its aspect ratio, and denotes the corresponding rates. The Kalman filter updates this state when a detection is associated with the predicted track. Consequently, association can reject isolated inconsistent boxes, but a sustained series of false detections may still form incorrect trajectories.
Feature Extractor (Optional): Motion state alone may be insufficient to distinguish nearby or occluded objects. Therefore, appearance-based trackers, including Deep SORT, use a convolutional neural network (CNN) to extract descriptors that support identity association [10]. This optional stage is shown in Figure 2.
Data Association: The data association module matches current detections to predicted track states and, when available, image-derived features. The Mahalanobis distance [13] can measure the discrepancy between a detection and a predicted state. Subsequently, the Hungarian algorithm [14] can solve the resulting assignment problem. Although these stages help maintain object identities, their effectiveness depends on the incoming detections.
Evaluation Metrics
MOT performance is evaluated through MOTA [15], IDF1 [16], and HOTA [17]. These metrics capture different aspects of detector errors, identity preservation, and their combined tracking effect. They are therefore reported together in Section IV.
Multiple Object Tracking Accuracy (MOTA): MOTA accounts for false positives (FP), false negatives (FN), and identity switches (IDSW) across time . It penalizes detection and association errors relative to ground-truth instances ():
Identification F1 Score (IDF1): IDF1 measures whether predicted identities remain consistent with ground-truth identities across the sequence [16]. It is defined through identification true positives (IDTP), false positives (IDFP), and false negatives (IDFN):
Higher Order Tracking Accuracy (HOTA): Provides a balanced evaluation of Detection Accuracy (DetA) and Association Accuracy (AssA) with different thresholds [17]:
DetA measures how accurately objects are detected, while AssA measures how consistently their identities are maintained across frames. This means that HOTA provides a more balanced assessment of detection and association performance.
Overall, MOTA emphasizes detection and identity errors, IDF1 focuses on identity consistency, and HOTA provides a balanced measure of detection and association performance.
Adversarial Attacks
Both JDT and TBD methods have been studied under adversarial attacks. For example, attacks on GOTURN-style single-object tracking [18] and FairMOT [19] alter learned tracking behavior. For modular TBD systems, an attacker may instead target the detector, the association process, or tracklet management. The distinction matters because detector-only attacks need not access the internal tracking model.
Attacking Object Detectors: The accuracy of TBD methods heavily relies on the object detector’s ability to produce precise bounding boxes of target objects. Missed or misplaced detections can therefore disrupt subsequent track association.
In 2019, the first adversarial attack against MOT models attacked a TBD method that used YOLOv3 as the object detector [3]. Moreover, YOLOX [6] and CenterNet [20] stand out as two widely-used object detectors for autonomous driving applications, yet they both exhibit vulnerability to adversarial attacks [21, 22]. To mitigate adversarial attacks, temporal outlier detection offers a potential approach to identifying adversarial perturbations [23].
Attacking Association Methods: Association assigns an identity to each detected object across frames. Liu et al. [24] describe attacks that alter object-label evidence and can undermine association. Unlike those methods, detector-only attacks first alter the image measurements passed to otherwise unchanged tracking modules.
Attacking Tracklets: Tracklets retain an object’s estimated position and motion, often through Kalman-filter state prediction. An error in these observations can alter subsequent matches and, in some cases, switch nearby identities. Lin et al. [7] attacked the appearance and distance cues of neighboring pedestrians to disrupt tracking. Such attacks differ from the detector-input perturbations considered here because they explicitly seek to affect temporal association.
Methodology
Although some MOT systems use 3D LiDAR measurements [25], this study focuses on vision-only TBD systems. Specifically, we focus on modular TBD methods to investigate whether traditional statistical tracking components, such as the Kalman filter, can help mitigate the effects of adversarial attacks.
Given an input image , the object detection model produces a set of candidate detections . Each detection hypothesis is represented as
where specifies the bounding-box position and size, is the confidence score, and contains class probabilities for classes. The superscript below denotes the video-frame index.
For an image at time , the detector therefore produces
Each hypothesis describes a candidate object detected in the corresponding frame.
The adversarial attack aims to generate a perturbation such that the detector produces a different set of hypotheses for the perturbed image , i.e.,
Image-Specific Attack: The image-specific attack generates a perturbation tailored to a single input image. The attack maximizes or minimizes an adversarial loss with respect to the input image, rather than updating the model parameters. Maximizing the adversarial loss encourages the detector to generate incorrect bounding boxes, while minimizing them suppresses detected objects. The adversarial perturbation at time step is updated using the following equation:
where is the learning rate and constrains the perturbation within an bound of . The learning rate is decayed after each iteration.
Image-Agnostic Attack: The same optimization principle can be extended to an image-agnostic attack, which learns a single perturbation that can be applied to multiple input images. Given a set of sample images , the perturbation is iteratively optimized by computing the adversarial gradient for each image and accumulating the resulting updates. A smaller learning rate is used to improve the transferability of the perturbation across different images:
Unlike the image-specific attack, which produces a different perturbation for each image, the image-agnostic attack produces a single universal perturbation that is optimized across the available sample images.
Lastly, the adversarial loss is defined as
where is the number of candidate boxes, is the confidence score, is the class probability, and and are the predicted box dimensions. The sigmoid function is denoted by . Thus, the adversarial loss increases the detection confidence for all objects while reducing the size of predicted bounding boxes, thereby increasing the number of detections.
Experimental Results
The experiment applies image-specific and image-agnostic attacks to SORT [9], Deep SORT [10], Strong SORT [11], and OC-SORT [12] with YOLOv3 and YOLOv4 detectors in CARLA simulator [8] across seven maps. The evaluation investigates whether conventional tracking components, such as the Kalman filter and the Hungarian algorithm, can mitigate the effects of adversarial attacks.
Image-Specific PCB Attack
The image-specific attack was tested with hyperparameters of attack strength and learning rate decay . We updated the perturbation frame-by-frame, iterating one step per frame for 300 consecutive frames on each of the 7 maps.
After applying adversarial perturbations, MOTA decreases significantly for both YOLOv3 and YOLOv4, as shown in Figure 3. YOLOv3 generates many small false-positive boxes, whereas YOLOv4 produces fewer but larger spurious detections (see Figure 3 (a) and (b)). Notably, the MOTAs of YOLOv3 drop to negative values due to a large number of false positives.

Figure 3. Image-Specific Attacks substantially degrade the MOTA by generating a large number of small false positives bounding boxes. In contrast, YOLOv4 produces fewer but larger boxes, making it comparatively more robust to these attacks.
The effectiveness of the image-specific attack can be further improved by increasing the number of optimization iterations per frame, as false positives are relatively rare in the initial frames, leaving the tracking method largely unaffected. HOTA and IDF1 also decline, as shown in the retained Appendix plots, Figure 5 and Figure 6. These findings demonstrate that perturbing the object detection model alone is sufficient to compromise TBD frameworks.

(a) YOLOv3 HOTA

(b) YOLOv4 HOTA
Figure 5. HOTA under Image-Specific and Image-Agnostic Attacks against YOLOv3 and YOLOv4.

(a) YOLOv3 IDF1

(b) YOLOv4 IDF1
Figure 6. IDF1 under Image-Specific and Image-Agnostic Attacks against YOLOv3 and YOLOv4.
Image-Agnostic Attack
The image-agnostic attack generates an Universal Adversarial Perturbation (UAP) to attack all video frames. We trained the UAP on Map 02 for 100 iterations, as Map 02 generalizes best across all maps. The training was conducted with a learning rate of , a learning rate decay of , and an attack strength of .
Figure 4 shows that the UAP reduces MOTA for both detectors. For YOLOv3, this reduction is smaller than that under the image-specific attack. However, several YOLOv4 combinations show a larger MOTA reduction with the UAP, so the relative effect depends on the detector. The HOTA and IDF1 results in Figure 5 and Figure 6 provide corresponding evidence of degraded track association and identity preservation.

Figure 4. Image-Agnostic Attacks generate a Universal Adversarial Perturbation (UAP) applied to all video frames. Although generally less effective than image-specific attacks, they can still substantially degrade MOTA.
The YOLOv3 and YOLOv4 examples predominantly contain false detections labeled as birds and chairs, respectively, as shown in Figure 4(a) and (b). Vehicle-class filtering can reject many such boxes, but the resulting MOTA losses show that downstream tracking is not fully protected. These examples do not establish how much of the remaining error comes from missed vehicles or accepted false tracks. Vehicle-targeted perturbations would therefore provide a useful further comparison.
Conclusion
This study evaluated four real-time online tracking methods—SORT, Deep SORT, Strong SORT, and OC SORT—using the CARLA dataset. With the same object detector, OC SORT achieved tracking accuracy comparable to Strong SORT without requiring additional neural networks for feature extraction. This finding suggests that improving conventional tracking components, such as the Kalman filter, can enhance tracking accuracy without introducing additional deep learning-based components and potential attack surfaces.
Furthermore, for TBD methods, the tracking performance is highly dependent on detection accuracy. Our experimental results demonstrated that both image-specific and image-agnostic attacks substantially degraded MOTA, HOTA and IDF1, indicating that perturbing the object detector alone is sufficient to compromise the tracking system.
References
Appendix
References
- [1]Z. Sun, J. Chen, L. Chao, W. Ruan, and M. Mukherjee, “A survey of multiple pedestrian tracking based on tracking-by-detection framework,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 5, pp. 1819–1833, 2020.DOI
- [2]S. Krebes, B. Durasamy, and F. Flohr, “A survey on leveraging deep neural networks for object tracking,” in IEEE International Conference on Intelligent Transportation Systems (ITSC), 2017, pp. 411–418.
- [3]Y. J. Jia, Y. Lu, J. Shen, Q. A. Chen, H. Chen, Z. Zhong, and T. W. Wei, “Fooling detection alone is not enough: Adversarial attack against multiple object tracking,” in International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1905.11026
- [4]J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767, 2018.arxiv.org/abs/1804.02767
- [5]A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.arxiv.org/abs/2004.10934
- [6]Z. Ge, S. Liu, F. Wang, Z. Li, and J. Sun, “Yolox: Exceeding yolo series in 2021,” arXiv preprint arXiv:2107.08430, 2021.arxiv.org/abs/2107.08430
- [7]D. Lin, Q. Chen, C. Zhou, and K. He, “Tracklet-switch adversarial attack against pedestrian multi-object tracking trackers,” arXiv preprint arXiv:2111.08954, 2021.arxiv.org/abs/2111.08954
- [8]A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in The 1st Annual Conference on Robot Learning, 2017, pp. 1–16.arxiv.org/abs/1711.03938
- [9]A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft, “Simple online and realtime tracking,” in IEEE International Conference on Image Processing (ICIP), 2016, pp. 3464–3468.DOI
- [10]N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in IEEE International Conference on Image Processing (ICIP), 2017, pp. 3645–3649.arxiv.org/abs/1703.07402
- [11]Y. Du, Z. Zhao, Y. Song, Y. Zhao, F. Su, T. Gong, and H. Meng, “Strong-sort: Make deepsort great again,” IEEE Transactions on Multimedia, 2023.
- [12]J. Cao, J. Pang, X. Weng, R. Khirodkar, and K. Kitani, “Observation-centric sort: Rethinking sort for robust multi-object tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 9686–9696.arxiv.org/abs/2203.14360
- [13]M. P. Chandra et al., “On the generalised distance in statistics,” in Proceedings of the national institute of sciences of India, vol. 2, no. 1, 1936, pp. 49–55.
- [14]H. W. Kuhn, “The hungarian method for the assignment problem,” Naval research logistics quarterly, vol. 2, no. 1-2, pp. 83–97, 1955.
- [15]K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: the clear mot metrics,” EURASIP Journal on Image and Video Processing, vol. 2008, pp. 1–10, 2008.
- [16]E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in IEEE European Conference on Computer Vision (ECCV), 2016, pp. 17–35.arxiv.org/abs/1609.01775
- [17]J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, “Hota: A higher order metric for evaluating multi-object tracking,” International Journal of Computer Vision (IJCV), 2020.DOI
- [18]R. R. Wiyatno and A. Xu, “Physical adversarial textures that fool visual object tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4822–4831.arxiv.org/abs/1904.11042
- [19]D. Lin, Q. Chen, C. Zhou, and K. He, “Trasw: Tracklet-switch adversarial attacks against multi-object tracking,” arXiv preprint arXiv:2111.08954, 2021.
- [20]K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian, “Centernet: Keypoint triplets for object detection,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6569–6578.arxiv.org/abs/1904.08189
- [21]T. Zhou, Q. Ye, W. Luo, K. Zhang, Z. Shi, and J. Chen, “F&f attack: Adversarial attack against multiple object trackers by inducing false negatives and false positives,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 4573–4583.DOI
- [22]H. Pang, R. Ma, J. Su, C. Liu, Y. Gao, and Q. Jin, “Blinding and blurring the multi-object tracker with adversarial perturbations,” Neural Networks, p. 106331, 2024.DOI
- [23]E. C. Erkuş and V. Purutçuoğlu, “Outlier detection and quasi-periodicity optimization algorithm: Frequency domain based outlier detection (FOD),” European Journal of Operational Research, vol. 291, no. 2, pp. 560–574, 2021.DOI
- [24]S. Liu, Z. Chen, W. Li, J. Zhu, J. Wang, W. Zhang, and Z. Gan, “Efficient universal shuffle attack for visual object tracking,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 2739–2743.
- [25]D. Liu and W. Hu. “Imperceptible transfer attack and defense on 3d point cloud classification,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 4, pp. 4727–4746, 2022.DOI