Automated classification of scanning electron microscope (SEM) defects is complicated by acquisition-domain shift, long-tailed classes, and target-only defect types that invalidate the closed-set assumption. We propose an offline open-set adaptation framework that combines a three-criterion reliable-known selector with global and defect-region-guided local prototype alignment. Confidence, entropy, and prototype distance jointly determine which unlabeled target samples may be aligned to known source prototypes; samples that fail this gate are excluded from known-class alignment and optimized by an unknown-rejection objective. Unlike global-only adaptation, foreground-weighted local prototypes emphasize compact defect morphology rather than domain-specific background texture. Averaged over four bidirectional SEM adaptation tasks and three random seeds per task, the method achieves 80.8% balanced accuracy, 78.1% Macro-F1, 88.7% unknown AUROC, 74.2% H-score, and 8.4% ECE. Compared with the strongest baseline for each individual metric, the proposed method improves balanced accuracy by 4.4 percentage points and Macro-F1 by 4.5 percentage points over M2DD, improves unknown AUROC by 6.9 percentage points and reduces ECE by 3.1 percentage points relative to the energy-based baseline, and improves H-score by 5.9 percentage points over CMU. Component removal and sensitivity analyses confirm complementary contributions from reliable-target selection, prototype alignment, unknown rejection, and localized alignment. The framework is directly applicable to offline SEM inspection when labeled historical data and unlabeled new-condition images are jointly available; its selective-alignment principle may also transfer to other localized visual-inspection tasks after domain-specific validation and calibration.
- Open Access
- Article
Open-Set Semiconductor Defect Classification in SEM Images via Uncertainty-Aware Local Prototype Alignment
- Xinyi Yuan 1,
- Sisi Chen 1,
- Boran Hu 1,
- Yikang Zhou 1,
- Aohan Mei 2,
- Nan Li 3,*
Author Information
Received: 06 Jun 2026 | Revised: 19 Jul 2026 | Accepted: 21 Jul 2026 | Published: 06 Aug 2026
Abstract
Keywords
semiconductor defect classification | SEM image analysis | open-set domain adaptation | uncertainty estimation | prototype alignment | defect localization
References
- 1.
Imoto, K.; Nakai, T.; Ike, T.; et al. A CNN-Based Transfer Learning Method for Defect Classification in Semiconductor Manufacturing. IEEE Trans. Semicond. Manuf. 2019, 32, 455–459.
- 2.
Patel, D.V.; Bonam, R.; Oberai, A.A. Deep Learning-Based Detection, Classification, and Localization of Defects in Semiconductor Processes. J. Micro/Nanolithogr. MEMS MOEMS 2020, 19, 024801.
- 3.
Qiao, Y.; Mei, Z.; Luo, Y.; et al. DeepSEM-Net: Enhancing SEM Defect Analysis in Semiconductor Manufacturing with a Dual-Branch CNN-Transformer Architecture. Comput. Ind. Eng. 2024, 193, 110301.
- 4.
Kim, J.; Nam, Y.; Kang, M.C.; et al. Adversarial Defect Detection in Semiconductor Manufacturing Process. IEEE Trans. Semicond. Manuf. 2021, 34, 365–371.
- 5.
Yang, D.; Jeong, J.; Yoon, H. Improving Semiconductor Defect Classification in SEM Images via Domain Adaptation. IEEE Open J. Ind. Electron. Soc. 2026, 7, 766–779. https://doi.org/10.1109/OJIES.2026.3663382.
- 6.
Ganin, Y.; Ustinova, E.; Ajakan, H.; et al. Domain-Adversarial Training of Neural Networks. J. Mach. Learn. Res. 2016, 17, 1–35.
- 7.
Zhang, Y.; Liu, T.; Long, M.; et al. Bridging Theory and Algorithm for Domain Adaptation. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA, 9–15 June 2019; Vol. 97, pp. 7404–7413.
- 8.
Long, M.; Cao, Z.; Wang, J.; et al. Conditional Adversarial Domain Adaptation. In Proceedings of the Advances in Neural Information Processing Systems 31 (NeurIPS 2018), Montr´eal, QC, Canada, 2–8 December 2018; Vol. 31, pp. 1647–1657.
- 9.
Cao, K.; Wei, C.; Gaidon, A.; et al. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. In Proceedings of the Advances in Neural Information Processing Systems 32 (NeurIPS 2019), Vancouver, BC, Canada, 8–14 December 2019; Vol. 32, pp. 1565–1576.
- 10.
Lin, T.Y.; Goyal, P.; Girshick, R.; et al. Focal Loss for Dense Object Detection. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988.
- 11.
Cui, Y.; Jia, M.; Lin, T.Y.; et al. Class-Balanced Loss Based on Effective Number of Samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 9268–9277.
- 12.
Bendale, A.; Boult, T.E. Towards Open Set Deep Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 1563–1572.
- 13.
Saito, K.; Yamamoto, S.; Ushiku, Y.; et al. Open Set Domain Adaptation by Backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 153–168.
- 14.
Zhou, B.; Khosla, A.; Lapedriza, A.; et al. Learning Deep Features for Discriminative Localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, 27–30 June 2016; pp.
2921–2929. - 15.
Selvaraju, R.R.; Cogswell, M.; Das, A.; et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Int. J. Comput. Vis. 2020, 128, 336–359.
- 16.
Panareda Busto, P.; Gall, J. Open Set Domain Adaptation. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 754–763.
- 17.
You, K.; Long, M.; Cao, Z.; et al. Universal Domain Adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 16–20 June 2019; pp. 2720–2729.
- 18.
Fu, B.; Cao, Z.; Long, M.; et al. Learning to Detect Open Classes for Universal Domain Adaptation. In Proceedings of the Computer Vision—ECCV 2020, 16th European Conference, Glasgow, UK, 23–28 August 2020; pp. 567–583.
- 19.
Vaze, S.; Han, K.; Vedaldi, A.; et al. Open-Set Recognition: A Good Closed-Set Classifier is All You Need. In Proceedings of the International Conference on Learning Representations, Virtual Event, 3–7 May 2021.
- 20.
Liu, W.; Wang, X.; Owens, J.D.; et al. Energy-Based Out-of-Distribution Detection. In Proceedings of the Advances in Neural Information Processing Systems 33 (NeurIPS 2020), Virtual Event, 6–12 December 2020; Vol. 33, pp. 21464–21475.
- 21.
Snell, J.; Swersky, K.; Zemel, R.S. Prototypical Networks for Few-Shot Learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS 2017), Long Beach, CA, USA, 4–9 December 2017; Vol. 30, pp. 4077–4087.
- 22.
Sun, Y.; Wang, X.; Liu, Z.; et al. Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. In Proceedings of the 37th International Conference on Machine Learning, Virtual Event, 13–18 July 2020; Vol. 119, pp. 9229–9248.
- 23.
Wang, D.; Shelhamer, E.; Liu, S.; et al. Tent: Fully Test-Time Adaptation by Entropy Minimization. In Proceedings of the International Conference on Learning Representations, Virtual Event, 3–7 May 2021.
- 24.
Zhou, K.; Liu, Z.; Qiao, Y.; et al. Domain Generalization: A Survey. IEEE Trans. Pattern Anal. Mach. Intell. 2023, 45, 4396–4415.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Suite 4002 Level 4, 447 Collins Street, Melbourne, Victoria 3000, Australia
General Inquiries: info@sciltp.com