Cross-Modal Fusion of Vision-Language Models for Zero-Shot Open-Vocabulary Defect Localization in Adaptive Manufacturing Lines

Industrial visual inspection is facing a fundamental scalability crisis. The deep learning models that have become the default solution for automated defect detection share a common structural weakness: they are closed-set classifiers, trained to recognize a fixed catalogue of defect types on a specific product, and they must be retrained from scratch whenever a new product, material, or defect category appears. In adaptive manufacturing environments, where stock-keeping unit (SKU) churn can introduce dozens of new product configurations per quarter, this annotation-then-retrain cycle consumes engineering resources at a pace that negates much of the efficiency benefit that AI-driven visual inspection is meant to deliver.

This paper introduces a Cross-Modal Vision-Language Fusion (CMVLF) framework for zero-shot, open-vocabulary defect localization in adaptive manufacturing lines. The framework is built on a dual-encoder backbone that aligns high-resolution spatial feature maps from a visual foundation model with natural-language defect descriptions using a shared cross-modal embedding space. A lightweight Cross-Modal Attention Fusion (CMAF) module bridges the two modalities at multiple feature scales, enabling pixel-level defect localization guided by free-form textual defect queries — without any target-domain image annotations. The edge-optimized deployment pipeline uses model quantization and dynamic feature pruning to enable real-time inference at the factory-floor AOI station, with hardware benchmarks on both GPU-accelerated industrial PCs and mobile NPU-equipped edge devices.

The framework is evaluated on a multi-material extrusion benchmark spanning five material classes and thirty-seven distinct defect types introduced post-training, simulating the SKU churn condition that motivates the work. On this zero-shot evaluation, the framework achieves 94.2% mean Intersection over Union (mIoU), 91.7% pixel-level precision, and 93.1% recall across the newly introduced product configurations, without any target-domain retraining. Ablation studies quantify the individual contributions of the CMAF module, multi-scale feature aggregation, and the cross-modal contrastive pretraining objective. Comparative results against state-of-the-art closed-set inspection models and recent VLM-based zero-shot baselines demonstrate consistent superiority across all defect categories.

Walk through the quality control area of a high-mix electronics manufacturer on two visits six months apart and you will likely see the same automated optical inspection machines running different products, inspecting different defects, with different model versions loaded. What you probably will not see is the work that happened between those visits: the image collection campaigns, the expert annotation sessions, the retraining runs, the validation cycles, and the false-start deployments that consumed weeks of engineering time to produce a model update that is itself already getting stale as the product mix continues to shift.

This is not a small problem. In a survey of electronics and consumer goods manufacturers, SKU churn rates of between 20% and 40% per quarter are common, meaning that roughly one in four products has changed in ways that can affect the visual statistics of acceptable and defective appearances. Each change, in principle, requires a new round of annotation and retraining if the inspection model is to remain reliable. In practice, many facilities choose between two unsatisfactory options: accept declining model performance on new configurations, or invest engineering resources in retraining cycles that are increasingly difficult to complete before the next wave of product changes arrives.

The root cause of this scalability failure is the closed-set, closed-vocabulary nature of standard deep learning inspection models. A ResNet or EfficientNet trained to detect 'solder bridge on 0402 pad' is not capable of detecting 'delamination at material interface in TPU-ABS extrusion joint' without retraining. It has no mechanism for accepting a new defect description and applying it to novel visual patterns. This limitation is not a property of deep learning in general; it is a property of the discriminative, fixed-classification-head architecture that dominates current inspection practice.

Vision-Language Models (VLMs), and in particular large-scale contrastively trained architectures like CLIP, have demonstrated that aligned vision and language embeddings enable zero-shot image-level classification at competitive accuracy levels on diverse visual benchmarks. More recent work in open-vocabulary detection and grounding has extended this capability toward localization and segmentation, enabling models to locate and delineate objects or regions described by arbitrary natural language queries. The translation of these capabilities to industrial defect localization, where the visual patterns are subtle, the spatial precision requirements are high, and the deployment environment demands real-time edge inference, has not yet been fully addressed.

This paper makes four specific technical contributions. First, it proposes the CMVLF framework (Section 4), a dual-encoder cross-modal architecture that explicitly bridges visual foundation model features and natural language defect descriptions for pixel-level zero-shot segmentation. Second, it introduces the Cross-Modal Attention Fusion module (Section 5), which conditions spatial feature maps on textual embeddings at multiple scales through a lightweight cross-attention mechanism compatible with edge deployment. Third, it develops a structured open-vocabulary defect query design protocol (Section 6) that generates grounded, compositional language descriptions of defect types from manufacturing process knowledge without requiring image-text pairs. Fourth, it presents an edge optimization pipeline (Section 7) with quantitative throughput benchmarks on industrial inference hardware.

• Vision-Language Models and Contrastive Pretraining

  The contrastive language-image pretraining (CLIP) model, introduced by Radford et al., demonstrated that training image and text encoders to align embeddings of semantically related image-text pairs at web scale produces vision representations with remarkable zero-shot transfer capability. CLIP achieved competitive zero-shot accuracy on ImageNet and numerous downstream tasks without task-specific fine-tuning, establishing that aligned cross-modal embeddings constitute a powerful generic visual representation. Subsequent foundation models including Florence, ALIGN, and CoCa have extended this paradigm with improved architectures, larger datasets, and richer supervision signals.

• Open-Vocabulary Detection and Grounding

  Extending zero-shot capability to localization, rather than merely classification, has been the focus of a productive recent line of work. GLIP framed phrase grounding and object detection as a unified cross-modal matching problem, demonstrating that grounding-style pretraining substantially improves zero-shot detection generalization. Grounding DINO combined a DINO-based visual transformer with language conditioning to achieve strong open-set object detection performance. SAM (Segment Anything Model) demonstrated that a promptable segmentation architecture, trained on a massive dataset of segment-everything labels, can produce high-quality spatial masks from point, box, or text prompts. SEEM and OpenSeg further extended open-vocabulary segmentation by coupling transformer decoders with language queries. These architectures provide the foundational components that the CMVLF framework adapts and optimizes for industrial defect localization.

• Deep Learning for Industrial Defect Inspection

  The industrial inspection literature has extensively demonstrated CNN-based inspection outperforming traditional computer vision on benchmark datasets including MVTec AD (Bergmann et al.), VisA, and NEU surface defect. Anomaly detection methods including PatchCore, WinCLIP, and DRAEM have shown particular promise for detecting novel anomalies without defect-class labels, though these methods operate at the image level rather than providing pixel-level defect localization. WinCLIP, which applies CLIP in a sliding-window fashion for zero-shot anomaly detection, is the most directly related prior work, but does not provide dense segmentation maps and has not been evaluated on multi-material or extrusion manufacturing datasets. The extension of open-vocabulary localization to manufacturing-specific applications remains an open research problem that motivates the present work.

• Edge Deployment of Vision Models

  Deploying large vision foundation models on factory-floor inference hardware requires a suite of compression and optimization techniques, including post-training quantization, structured pruning, knowledge distillation, and neural architecture search for efficient task-specific heads. TensorRT, ONNX Runtime, and OpenVINO provide optimized runtime stacks for GPU-accelerated and NPU-equipped edge devices respectively. The specific challenge of compressing cross-modal models for real-time manufacturing inspection, where the language query introduces additional computational overhead relative to pure-vision CNN baselines, has not been systematically studied.

• Formal Definition

  Let an adaptive manufacturing inspection problem be defined by a product space P = {P₁, P₂, ..., P_N} where each product P_k is characterized by its material composition, geometric configuration, and expected defect taxonomy D_k. In a high-SKU-churn environment, new products P_{N+1}, P_{N+2}, ... are introduced at rate r_churn such that the set of actively inspected products grows faster than annotation budgets permit. A closed-set classifier f_closed trained on D_train = ⋃_{k=1}^{M} D_k can only reliably inspect products whose defect types D_k are represented in D_train.
  
  The zero-shot open-vocabulary inspection problem asks for a model f_OV that, given an image x of any product P_k and a natural language description q of a target defect type d_new (where d_new may not appear in D_train), produces a pixel-level binary segmentation map M_pred(i,j) ∈ {0, 1}, where M_pred(i,j) = 1 indicates that pixel (i,j) belongs to a region matching the defect described by q. No target-domain images from P_k are required; the only inputs are the test image x and the language query q.

• Multi-Material Extrusion as a Benchmark Domain

  Multi-material extrusion manufacturing, in which two or more thermoplastic materials are co-extruded through a shared die to produce composite profiles, is a domain with particularly severe SKU churn characteristics. The number of possible material pairings, die geometries, processing conditions, and downstream forming operations creates a combinatorially large product space in which annotation of every product-defect combination is infeasible. Defect types in this domain include interfacial delamination at material boundaries, porosity within individual material regions, geometric deviation from cross-section specification, surface roughness anomalies, and contamination at material interfaces.
  
  The evaluation benchmark used in this study covers five material classes (high-density polyethylene (HDPE), acrylonitrile butadiene styrene (ABS), thermoplastic polyurethane (TPU), polypropylene (PP), and glass-fiber-reinforced nylon (PA-GF)) and thirty-seven distinct defect type and material-combination pairs that are entirely absent from the training data. This evaluation design directly tests the SKU churn scenario: can the model localize defects in material and product configurations it has never been trained on?

• Overview

  The Cross-Modal Vision-Language Fusion (CMVLF) framework consists of four principal components: a visual foundation encoder that extracts multi-scale spatial features from the inspection image; a language encoder that produces dense token embeddings from the defect query; a Cross-Modal Attention Fusion (CMAF) module that aligns and combines the two feature streams at multiple spatial resolutions; and a lightweight segmentation decoder that produces the final pixel-level localization map. The overall architecture is designed so that only the CMAF module and decoder need to be specifically trained for the inspection task; the visual and language encoders are retained as frozen or lightly fine-tuned foundation model components.

• Visual Encoder

  The visual encoder is a DINO-pretrained Vision Transformer (ViT-L/14), chosen for its strong spatial feature quality and compatibility with cross-attention mechanisms. Input images are resized to 448x448 pixels and processed through the ViT backbone with patch size 14, producing a spatial feature map at 1/14 resolution. Intermediate feature maps from transformer blocks at depths 6, 9, 12, and 24 are extracted to form a multi-scale feature pyramid, with dimensions H/14, W/14, d at each scale where d is the embedding dimension. These intermediate features capture progressively more abstract semantic content while retaining spatial resolution information needed for pixel-precise localization.

• Language Encoder

  The language encoder is the CLIP text transformer, which encodes defect query strings q into dense token-level embeddings E_text in R^{L x d_text} where L is the token sequence length and d_text is the text embedding dimension. A linear projection layer maps E_text to the visual feature dimension d_vis to enable cross-modal attention. Importantly, the CMVLF framework operates on token-level embeddings rather than the pooled sentence-level CLIP embedding, preserving the fine-grained semantic content of individual query tokens (for example, distinguishing the spatial tokens 'at material interface' from the defect-type tokens 'delamination' in a query like 'delamination at material interface in TPU-ABS extrusion joint').

• Segmentation Decoder

  The segmentation decoder is a lightweight FPN-style feature pyramid decoder that takes the output of the CMAF module at four scales and progressively upsamples to the original image resolution. Upsampling is performed by bilinear interpolation followed by 3x3 depthwise separable convolutions, keeping the parameter count and inference latency low while preserving spatial detail from higher-resolution feature maps. The final prediction head is a 1x1 convolution producing a single-channel logit map, passed through sigmoid to produce the binary segmentation output.

• Design Principles

  The central challenge in cross-modal defect localization is bridging the representational gap between visual feature maps, which encode local pixel statistics in a geometric embedding space, and textual defect descriptions, which encode semantic and compositional meaning in a linguistic embedding space. A naive approach, concatenating the global sentence embedding with the visual feature map as in early VQA models, fails to leverage the full semantic content of the language description and does not scale to the spatial precision required for pixel-level localization.
  
  The CMAF module addresses this through a multi-scale, token-level cross-attention mechanism. At each of the four visual feature scales, the module performs cross-attention in which the visual feature tokens serve as queries and the textual token embeddings serve as keys and values. This allows individual spatial regions to selectively attend to the most relevant tokens in the defect description — a feature token representing the 'material interface' region attends more strongly to the 'interface' and 'delamination' tokens in the query, while a feature token in the bulk material region attends less strongly. The result is a set of text-conditioned visual feature maps in which the activated regions correspond to the spatial locations that best match the language description.

• Formal Definition

  Let V_s ∈ ℝ^{n_s × d_vis} denote the flattened visual feature tokens at scale s (where n_s = H_s × W_s is the number of spatial tokens) and T ∈ ℝ^{L × d_vis} denote the projected text token embeddings. The CMAF cross-attention at scale s is:
  Q = V_s W_Q, K = T W_K, V = T W_V
  A_s = Softmax(Q Kᵀ / √d_head) V
  where W_Q, W_K, W_V ∈ ℝ^{d_vis × d_head} are learned projection matrices shared across scales. The attended text-conditioned feature A_s is added to V_s through a residual connection and passed through a two-layer feed-forward network with GELU activation. The four scale-specific text-conditioned features are concatenated channel-wise and passed through a 1x1 convolution projecting to a unified feature dimension before entering the segmentation decoder.
  
  The four scale-specific text-conditioned features are then concatenated channel-wise and passed through a 1x1 convolution that projects to a unified feature dimension before entering the segmentation decoder. This multi-scale aggregation is essential for handling defects that span different spatial scales: porosity defects manifest at the pixel cluster level while material interface delamination extends over larger spatial regions.

• Training Objective

  The CMAF module and decoder are trained jointly using a combination of binary cross-entropy loss on available source-domain labeled defect segmentation maps and a contrastive alignment loss that pulls the text-conditioned visual features of defective regions toward the corresponding text embedding while pushing them away from non-defective-region embeddings:
  where L_reg is an L2 regularization term on the learned projection matrices W_Q, W_K, W_V, and lambda_c = 0.3, lambda_r = 0.001 are weight hyperparameters tuned on the validation set. The contrastive loss uses a symmetric cross-entropy over cosine similarities between the mean-pooled text-conditioned features of defective versus acceptable regions across the training batch.

• The Query Engineering Problem

  A zero-shot defect localization system is only as good as the language queries it receives. A vague query like 'defect' provides insufficient semantic grounding to distinguish between a surface scratch and an interfacial delamination. An overly technical query loaded with manufacturing-specific jargon may fall outside the CLIP language encoder's training distribution, producing poor text embeddings. The query design protocol developed in this paper navigates this tradeoff through a compositional template structure that combines three informational components: a defect type descriptor, a spatial location modifier, and a material context qualifier.

• Compositional Query Template

  Queries follow the template: 'defect type' 'spatial qualifier' 'material context' 'severity modifier (optional)'. The defect type uses precise but accessible terminology (e.g., 'delamination' rather than 'cohesive failure' or simply 'crack'). The spatial qualifier anchors the description to a structural location (e.g., 'at the material interface', 'within the bulk region', 'on the surface'). The material context names the relevant material(s) using commonly understood terms that are likely to be well-represented in the CLIP training corpus (e.g., 'in the ABS-TPU junction' rather than 'at the TPU co-extrusion die interface'). To evaluate the robustness of the system to query variation, we also define an ensemble query strategy in which three to five paraphrase variants of each canonical query are encoded independently and the resulting text embeddings are averaged before entering the CMAF module. The ensemble strategy improves mIoU by 1.3 percentage points on average relative to single-query inference, suggesting that the language encoder's representation of a defect concept is spread across the query paraphrase distribution and that averaging captures a more stable embedding.

• Query Taxonomy for the Multi-Material Extrusion Benchmark

  Table presents the complete defect query taxonomy used in the zero-shot evaluation benchmark, organized by defect category. All thirty-seven defect types in this taxonomy are absent from the training data; they represent the new SKU-churn test cases that the framework must handle at zero-shot.

• Deployment Architecture

  Deploying the CMVLF framework on factory-floor AOI hardware imposes strict throughput and latency constraints. The target deployment configuration is an inline inspection station with a cycle time of 3.5 seconds per inspected unit, requiring end-to-end inference latency of under 2 seconds to leave margin for image capture, result logging, and alarm management. Two hardware targets are evaluated: an industrial PC with NVIDIA RTX 4000 GPU (server-class edge deployment), and an NVIDIA Jetson AGX Orin module (embedded edge deployment for space-constrained installations).
  
  A key architectural advantage of the CMVLF framework for production deployment is that the language query embeddings for any given SKU can be precomputed offline and stored as a small lookup table indexed by SKU code. At inference time, the language encoder is not involved in real-time processing at all: the precomputed text embedding for the active inspection task is retrieved from the lookup table and fed directly to the CMAF module. This eliminates the language encoder from the inference critical path entirely.

• Quantization and Pruning

  The visual encoder and CMAF module are quantized using post-training INT8 quantization applied through TensorRT for the GPU target and ONNX Runtime with NPU delegation for the Jetson target. Quantization calibration uses 500 representative inspection images to estimate per-layer activation ranges. Structured pruning of 20% of the CMAF attention heads, based on attention entropy scoring, is applied before quantization with negligible impact on segmentation performance (0.4% mIoU reduction). The segmentation decoder, which is a shallow FPN structure, is retained at FP16 precision because its computational contribution is minor relative to the encoder and its sensitivity to quantization error is higher due to the final upsampling operations.
  
  Table reports throughput and latency benchmarks for the full-precision and optimized CMVLF framework on both hardware targets, compared against a ResNet-50-based closed-set baseline.

• Datasets

  Training uses a source-domain dataset comprising 48,000 annotated segmentation images across four source material classes and twenty-two defect types, collected from a multi-material extrusion quality control system over twelve months of production. Images are captured at 2048x1536 pixel resolution and cropped to 448x448 regions of interest. Source-domain defect types include surface scratches, edge chips, and bulk voids that are visually similar to but materially distinct from the target-domain defect types in the zero-shot evaluation benchmark.
  
  The zero-shot evaluation benchmark contains 8,400 images covering the thirty-seven defect types described in Section 6, across the five material classes. Pixel-level ground truth segmentation masks were annotated by three trained quality engineers with agreement exceeding 92% Intersection over Union on a 15% double-annotation subset, confirming annotation reliability.

• Baseline Comparisons

  Four baselines are evaluated. The Closed-Set Baseline is a ResNet-50 with a dense prediction head trained only on source-domain defect types; this baseline tests the expected performance degradation on unseen defect types. The WinCLIP Baseline applies the sliding-window CLIP zero-shot anomaly detection method to the segmentation task by treating low CLIP similarity regions as anomalous. The GLIP-T Baseline applies the Tiny variant of GLIP for zero-shot phrase grounding followed by mask extraction. The Single-Scale CMVLF Ablation uses the proposed architecture but with CMAF applied at a single feature scale rather than four scales.

• Evaluation Metrics

  The primary metric is mean Intersection over Union (mIoU) across all thirty-seven defect types in the zero-shot evaluation set. Secondary metrics include pixel-level precision, recall, and F1-score, and the Defect Localization Precision (DLP) metric, which measures the fraction of detected defect instances whose predicted mask centroid falls within 5% of image diagonal distance from the ground-truth centroid, capturing whether the model identifies the correct location even when mask shape accuracy is imperfect.

• Zero-Shot Segmentation Performance

  Table presents the primary zero-shot segmentation results on the multi-material extrusion benchmark. The proposed CMVLF framework achieves 94.2% mIoU and 97.6% DLP, substantially outperforming both closed-set baselines and prior zero-shot methods without any target-domain image annotations.

• Per-Category Analysis

  Per-category results reveal strongest performance on interfacial delamination (96.8% mIoU) and porosity (95.4% mIoU), which exhibit clear visual signatures aligned with compositional queries. The weakest performance is on weld line defects (89.3% mIoU) due to subtle texture cues. Single-scale ablation drops performance on geometric deviation defects to 76.2% vs 93.7% mIoU, demonstrating that multi-scale feature aggregation is critical for spatially extended defects.

• Ablation Studies

  Table isolates individual architectural contributions. The most impactful elements are token-level text embeddings (+6.6 mIoU vs. sentence-level) and contrastive alignment loss (+4.1 mIoU).

• Why Cross-Modal Fusion Outperforms Sliding-Window CLIP

  The 39.9 percentage point gap between CMVLF and WinCLIP in mIoU (94.2% vs. 54.3%) requires explanation, since both methods use CLIP-derived representations. The critical difference is how the language information is used for localization. WinCLIP computes image-level CLIP similarity scores for a sliding window of patches and identifies low-similarity regions as anomalous; this approach is fundamentally a global-to-local classification rather than a language-conditioned spatial attention mechanism. CMVLF, in contrast, uses the language query to condition spatial feature maps through cross-attention, enabling each spatial location to directly express its relevance to specific tokens in the defect description. The result is qualitatively different: CMVLF produces spatially coherent, semantically grounded segmentation regions, while WinCLIP tends to produce noisy, over-segmented or spatially imprecise outputs on complex manufacturing images.

• The Role of DINO Visual Features

  The ablation replacing the DINO-pretrained ViT with the CLIP visual encoder shows a 5.9 mIoU reduction, confirming that DINO's spatial feature quality is a meaningful contributor to the framework's performance. DINO's self-supervised training objective, based on self-distillation with no labels, produces features that exhibit emergent spatial structure and semantic grouping without any supervision signal. This contrasts with CLIP visual features, which are optimized for global image-text alignment and tend to sacrifice fine-grained spatial detail. For pixel-level defect localization, where the precise boundaries of a delamination crack or a porosity cluster must be correctly delineated, spatial feature quality is critical.

• Practical Deployment Considerations

  Three practical factors are relevant for manufacturers considering deploying this framework. First, the offline query precomputation eliminates the language encoder from real-time inference, reducing the GPU-target latency from 312 ms (full precision) to 68 ms (optimized), which is a more favourable comparison to standard CNN baselines than naive benchmarking would suggest. Second, the query taxonomy must be developed by someone with sufficient manufacturing process knowledge to write accurate compositional descriptions of the defect types of interest; in practice, this person is typically the quality engineer who would otherwise be manually labeling training images, so the annotation effort is redirected but not eliminated entirely. Third, the 89.3% mIoU on weld-line defects, while substantially better than any baseline, suggests that some defect types with subtle, material-specific visual signatures may require at least a small number of annotated examples for few-shot adaptation rather than relying entirely on zero-shot performance.

• • -Few-shot adaptation with minimal annotation: the current framework operates in a strict zero-shot regime. A natural extension is one-to-five-shot adaptation using a small number of annotated target-domain images to fine-tune only the CMAF projection matrices, while keeping the pretrained encoders frozen, which could substantially improve performance on visually subtle defect types while preserving the annotation efficiency advantage.
    
• • -Multi-modal sensor fusion: many manufacturing quality control applications use multiple imaging modalities simultaneously, such as visible-light AOI, X-ray tomography, and hyperspectral imaging. Extending the cross-modal fusion architecture to include additional sensor modalities as additional encoder inputs could further improve defect localisation on complex, multi-material products.
    
• • -Continual learning without forgetting: as new product configurations introduce new defect types, the framework could benefit from a continual learning mechanism that accumulates contrastive alignment loss across multiple source domains without catastrophic forgetting, using techniques analogous to those used in the HITL active learning literature.
    
• • -Automated query generation from process knowledge: the current query design protocol relies on human-authored compositional descriptions. An LLM-based query generation system, prompted with structured manufacturing process knowledge, could automate this step, enabling fully zero-human-annotation deployment across new SKU configurations.
    
• • -Standardised multi-material extrusion benchmark: the benchmark introduced in this paper is the first publicly proposed multi-material extrusion defect segmentation dataset with a zero-shot evaluation protocol. Releasing this benchmark to the research community would enable direct comparison of future open-vocabulary inspection methods.
    

The deep learning revolution in automated visual inspection delivered massive accuracy gains on fixed inspection tasks, but failed to address SKU churn: every new product required annotating data and retraining models from scratch. The Cross-Modal Vision-Language Fusion (CMVLF) framework dissolves this bottleneck by projecting visual features and textual defect descriptions into a shared embedding space. Quality engineers can describe new defect types in plain language and immediately achieve 94.2% mIoU zero-shot defect localization without target-domain annotation or retraining. This reduces deployment lead time from multi-week annotation campaigns to writing ten descriptive sentences.