Preprint: From Promise to Practice: Benchmarking Vision Foundation Models for Semantic Segmentation of Industrial Defects

Abstract

In computer vision, machine learning extracts image information, and semantic segmentation labels each pixel to localize industrial defects. Training requires expert-annotated masks, while foundation models promise label-efficient transfer. Across nine domains, zero-shot use, domain-specific models at nested label budgets, and one jointly trained model tested how that promise translates into practice. A frozen foundation encoder with a trained head and low-rank adaptation were compared with fully fine-tuned convolutional models using Defect-IoU, which measures predicted–annotated defect overlap. Zero-shot use detected no defect pixels, while additional labels yielded domain-dependent Validation gains. In the capped specialist–joint comparison, joint training increased the eight-domain Test mean for both foundation-model adaptations but reduced it for convolutional models. Separately, the best fully fine-tuned convolutional specialist achieved 49.1% Validation Defect-IoU on proprietary Injection Molding with 45 labeled training images. Under the evaluated conditions, pretraining did not guarantee superior segmentation; outcomes depended on adaptation and domain-specific supervision.