In practical applications, image recognition of birds and drones faces certain challenges, particularly in real-world scenarios where the system may encounter unknown aerial targets not included in the training data. Nevertheless, most current research still focuses primarily on classification performance under controlled conditions, assuming that test samples belong to known categories. To address this issue, this study conducted a systematic comparison of eight deep learning models (including four convolutional neural network (CNN) models and four Transformer models) under unified training and evaluation conditions. In addition to testing under closed conditions, an open-domain detection scenario was constructed by simulating real-world environments and introducing unknown categories during the testing phase. Experimental results indicate that while models achieve high accuracy under closed-domain conditions, their performance under open-domain conditions varies significantly. In particular, unknown targets that closely resemble drones are more difficult to correctly reject, while some unknown samples are easily misclassified as drones. Therefore, accuracy under closed-domain conditions does not fully reflect the reliability of model detection in real-world operational environments, and evaluation under open-domain conditions is of great significance for analyzing model performance and practical applications.
- Open Access
- Article
Bird-Drone Recognition under Closed-Set and Open-Set Scenarios: A Comparative Analysis of Deep Learning Models
- Zehua Tang *,
- Victor Lawrence,
- Hong Man
Author Information
Received: 12 Feb 2026 | Revised: 13 Apr 2026 | Accepted: 20 May 2026 | Published: 11 Jun 2026
Abstract
Keywords
deep learning | bird–drone recognition | open-set evaluation
References
- 1.
Coluccia, A.; Fascista, A.; Schumann, A.; et al. Drone vs. bird detection: Deep learning algorithms and results from a grand challenge. Sensors 2021, 21, 2824. https://doi.org/10.3390/s21082824.
- 2.
Coluccia, A.; Fascista, A.; Dimou, A.; et al. The Drone-vs-Bird Detection Grand Challenge at IJCNN 2025. In Proceedings of the 2025 International Joint Conference on Neural Networks (IJCNN), Rome, Italy, 30 June–5 July 2025.
- 3.
Shandilya, S.K.; Srivastav, A.; Yemets, K.; et al. YOLO-based segmented dataset for drone vs. bird detection for deep and machine learning algorithms. Data Brief 2023, 50, 109355. https://doi.org/10.1016/j.dib.2023.109355.
- 4.
Kaur, D.; Battish, N.; Bhavsar, A.; et al. YOLOBirDrone: Dataset for Bird vs Drone Detection and Classification and a YOLO based enhanced learning architecture. arXiv 2026, arXiv:2601.08319.
- 5.
Rahman, S.; Robertson, D.A. Classification of drones and birds using convolutional neural networks applied to radar micro-Doppler spectrogram images. IET Radar Sonar Navig. 2020, 14, 653–661. https://doi.org/10.1049/iet-rsn.2019.0493.
- 6.
Akyon, F.C.; Akagündüz, E.; Altinuc, S.O.; et al. Sequence Models for Drone vs. Bird Classification. In Proceedings of the Sixteenth International Conference on Machine Vision (ICMV 2023), Yerevan, Armenia, 15–18 November 2023.
- 7.
Scheirer, W.J.; de Rezende Rocha, A.; Sapkota, A.; et al. Toward open set recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 35, 1757–1772.
- 8.
Hendrycks, D.; Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017), Toulon, France, 24–26 April 2017.
- 9.
Yang, J.; Zhou, K.; Li, Y.; et al. Generalized out-of-distribution detection: A survey. Int. J. Comput. Vis. 2024, 132, 5635–5662. https://doi.org/10.1007/s11263-024-02117-4.
- 10.
He, K.; Zhang, X.; Ren, S.; et al. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016.
- 11.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556.
- 12.
Huang, G.; Liu, Z.; Van Der Maaten, L.; et al. Densely connected convolutional networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017.
- 13.
Tan, M.; Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019.
- 14.
Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; et al. An image is worth 16 × 16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations (ICLR 2021), Virtual, 3–7 May 2021.
- 15.
Liu, Z.; Lin, Y.; Cao, Y.; et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 10–17 October 2021.
- 16.
Bendale, A.; Boult, T.E. Towards open set deep networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016.
- 17.
Liang, S.; Li, Y.; Srikant, R. Enhancing the reliability of out-of-distribution image detection in neural networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada, 30 April–3 May 2018.
- 18.
Lee, K.; Lee, K.; Lee, H.; et al. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems 31, Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), Montreal, Canada, 3–8 December 2018; Neural Information Processing Systems Foundation, Inc. (NeurIPS): San Diego, CA, USA, 2018.
- 19.
Liu, W.; Wang, X.; Owens, J.; et al. Energy-based out-of-distribution detection. In Advances in Neural Information Processing Systems 33, Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Online, 6–12 December 2020; Neural Information Processing Systems Foundation, Inc. (NeurIPS): San Diego, CA, USA, 2020; pp. 21464–21475.
- 20.
Wang, H.; Li, Z.; Feng, L.; et al. Vim: Out-of-distribution with virtual-logit matching. In Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–20 June 2022.
- 21.
Guo, C.; Pleiss, G.; Sun, Y.; et al. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia, 6–11 August 2017.
- 22.
Sun, J.; Dong, Q. A survey on open-set image recognition. arXiv 2023, arXiv:2312.15571.
- 23.
Deng, J.; Dong, W.; Socher, R.; et al. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 20–25 June 2009.
- 24.
Sokolova, M.; Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 2009, 45, 427–437. https://doi.org/10.1016/j.ipm.2009.03.002.
- 25.
Powers, D.M. Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation. arXiv 2020, arXiv:2010.16061.
- 26.
Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 2006, 27, 861–874. https://doi.org/10.1016/j.patrec.2005.10.010.
- 27.
Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In Proceedings of the 23rd International Conference on Machine Learning, Pittsburgh, PA, USA, 25–29 June 2006.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Suite 4002 Level 4, 447 Collins Street, Melbourne, Victoria 3000, Australia
General Inquiries: info@sciltp.com