WorksComputer Science
Defect-Aware Masked Autoencoders for Unsupervised Clustering of Steel Surface Defects
Abstract
Reliable recognition of steel surface defects is essential for automated quality inspection, but supervised methods require manually annotated images that are costly to produce. This creates a need for methods that can learn useful defect representations directly from unlabelled surface images. This work presents Defect aware MAE, a label free masked autoencoder for clustering steel surface defects. Each image is divided into 196 patches, with 147 masked and 49 processed by a Vision Transformer encoder. Texture energy based masking preferentially hides structurally informative patches, while a shallow decoder reconstructs HOG descriptors that capture local texture and edge orientation. Strong augmentation and an intermediate layer readout are used to form the final image features. Pretraining uses no class labels, bounding boxes, detection branch, or contrastive loss. On the held out NEU DET split, UMAP followed by Ward clustering achieves 98.6% Hungarian matched accuracy, 0.968 NMI, 0.967 ARI, and 0.838 silhouette score. Under an identical encoder and training schedule, it exceeds a SimCLR contrastive baseline in accuracy, NMI, ARI and silhouette scores across 5 seeds.
Introduction
Automated recognition of steel surface defects supports quality inspection, but supervised classification and detection methods depend on class labels or bounding box annotations [1, 2, 3, 4]. Self supervised learning reduces this dependence by constructing training objectives directly from unlabelled images [5]. Masked autoencoders learn visual representations by encoding visible image patches and reconstructing the missing information with a lightweight decoder [6], [7]. Structured targets such as histograms of oriented gradients can direct masked models toward local texture and edge orientation rather than raw pixel intensity [8], [9].
This work presents Defect aware MAE, a label free masked autoencoder for clustering the six NEU DET defect categories [10], [11]. The method combines HOG reconstruction, texture energy based masking, data augmentation, a shallow decoder and an intermediate layer feature readout. Pretraining uses no class labels, bounding boxes, detection branch or contrastive loss. Frozen features are projected with UMAP [12] and grouped through Ward clustering. On 360 held out images never seen during pretraining, Defect aware MAE achieves Hungarian matched accuracy across 5 seeds, while also producing higher NMI, ARI and silhouette values than a SimCLR baseline [5] trained with the identical encoder and schedule.
Literature Review
Song and Yan introduced the NEU surface defect dataset with six categories of hot rolled steel defects and investigated texture based recognition through completed local binary patterns [10]. Lv et al. later provided a metallic surface defect benchmark and a deep detection network [11]. Subsequent research has largely treated steel defect recognition as a supervised classification or detection problem. Examples include hierarchical feature fusion [3], Swin Transformer based classification and detection [2], broader deep learning classification methods [4], and MESC DETR, which combines a ConvNeXtV2 based backbone, edge enhanced feature fusion, and an adapted regression loss for metal defect detection [11]. These methods depend on labelled classes or spatial annotations during training.
Self supervised learning offers a route for reducing this dependence on annotation. Xu et al. combined self supervised representation learning with a segmentation network for surface defect detection [13]. Hu et al. applied contrastive representation learning with an Earth Mover’s Distance matching metric to steel surface defects and transferred the pretrained encoder to supervised detectors [24]. SimCLR provides a general contrastive framework in which two augmented views of each image are encoded and optimized through an agreement objective [5]. Although these approaches reduce the amount of labelled data required for representation learning, their learned features are primarily evaluated through supervised detection or segmentation tasks [13].
Masked image modelling provides a noncontrastive alternative. The masked autoencoder encodes a small visible subset of image patches and reconstructs the missing content with an asymmetric lightweight decoder [6]. Its encoder follows the Vision Transformer architecture, which represents an image as a sequence of patch tokens processed through self attention [7]. Autoencoder based representation learning has also been applied beyond visual inspection, for example to predict volumetric deformation from InSAR interferograms [14]. MaskFeat extends masked prediction by reconstructing structured visual features and reports that HOG targets provide an effective representation of local gradient structure [8]. HOG describes local appearance through distributions of gradient magnitude and orientation [9]. Stochastic Gumbel top sampling further provides a method for selecting elements without replacement according to weighted probabilities [15]. These ideas support the proposed combination of HOG reconstruction and texture energy guided masking.
Interpretation methods provide an additional way to examine learned visual representations. Grad CAM uses activation gradients to produce spatial importance maps [16], while Grad CAM++ refines the weighting of individual activations to improve localization [17]. Both methods have also been applied to explain deep learning models in medical imaging [18]. MESC DETR applies Grad CAM++ to examine attention in a supervised metal defect detector [1]. In the present work, Grad CAM and Grad CAM++ are instead applied to a frozen label free encoder by targeting similarity between an image feature and its assigned cluster centroid. The resulting features are projected with UMAP [12] and grouped without label supervision, while annotations are used only to evaluate clustering and explanation maps.
Proposed Methodology
The proposed Defect aware MAE uses a Vision Transformer based masked autoencoder to learn steel surface defect representations without class labels or spatial annotations. The released model was trained for 1,600 epochs with batch size 64. Training took 51 minutes on a single NVIDIA RTX A4000 GPU. The implementation, configurations, stored results, and reproduction instructions are publicly available at https://github.com/ShreyPatel1311/NEU-Defect-Aware-Autoencoder. The released model weights are available at https://huggingface.co/Godseye1311/defect-aware-MAE. The NEU DET dataset is publicly available on Kaggle.
Data Collection and Augmentation
The NEU DET dataset contains 1,800 grayscale images across six defect categories, with 300 images per category [10], [11]. The distributed split contains 1,440 training images and 360 held out validation images. Pretraining uses only the training images and does not use labels or bounding boxes. The sample images of each defect are represented in Figure 1.

Figure 1. Sample of each labelled image present in the NEU-DET dataset
Each image is enhanced using CLAHE [19], resized to pixels, and scaled to the interval from 0 to 1. Training augmentation includes a random resized crop with an area fraction between 0.35 and 1, horizontal and vertical flips, rotations by multiples of 90 degrees, brightness and contrast adjustment, Gaussian blur, and Gaussian noise.
Model Pretraining
Defect aware MAE follows the asymmetric masked autoencoder framework [6]. Each augmented grayscale image is divided into non overlapping patches of size . The method masks 147 patches and passes the remaining 49 patches to the encoder. A 2 layer decoder then predicts the HOG descriptor of every masked patch. Pretraining minimizes the reconstruction error at masked positions only.
Encoder
The encoder follows the Vision Transformer architecture [7]. Each image patch is projected into a 768 dimensional token by a convolutional patch embedding. A learnable class token is prepended, and learnable positional embeddings are added:
where denotes the patch projection. Positional embeddings are added before patch selection so that each visible token retains its spatial position.
The visible sequence is processed by 6 Transformer layers. Each layer contains pre normalization multihead self attention with 12 heads, followed by a 3,072 dimensional feed forward network with GELU activation. For layer , the operations are
Defect Aware Masking
Mask selection is based on the texture energy of each patch. For patch , its energy is defined as the standard deviation of its pixel intensities. The energy values are standardized within each image:
where and are the mean and standard deviation of the 196 patch energy values.
Each patch receives a score combining standardized texture energy with Gumbel noise:
The 147 patches with the highest scores are masked, while the 49 patches with the lowest scores remain visible. The released model uses . This gives patches with higher texture energy a greater probability of being masked while retaining stochastic selection through the Gumbel term [15]. With mask ratio , the number of visible patches is
HOG Reconstruction Target
The reconstruction target is a histogram of oriented gradients rather than pixel intensity [8], [9]. Horizontal and vertical Sobel filters produce gradients and for each pixel. Gradient magnitude and unsigned orientation are calculated as
The orientation interval from to is divided into bins. Each pixel contributes its gradient magnitude to the corresponding orientation bin. Histograms are averaged within pixel cells. Since each patch contains cells, their histograms are concatenated into a dimensional descriptor , which is normalized as
Decoder and Reconstruction Loss
The decoder projects the encoded tokens from to dimensions. A shared learnable mask token is inserted at every masked position, and decoder positional embeddings are added to the complete sequence. The sequence is processed by pre normalization Transformer layers with attention heads. A final linear layer predicts a dimensional HOG descriptor for each patch.
Let denote the set of masked patch indices, the HOG target, and the decoder prediction. The training objective is the mean squared reconstruction error over masked positions:
Only one augmented view of each image is required. Class labels, bounding boxes, detection losses, and contrastive losses are not used during pretraining. The model architecture is shown in Figure 2.
Figure 2. Model architecture of Defect aware MAE for predicting the HOG descriptors of masked patches
Optimization
Training minimizes the masked HOG reconstruction loss through one backward pass per batch. The model is trained for epochs, with batch size using AdamW [20], with , , a peak learning rate of , and weight decay of . The learning rate increases linearly during the first epochs and then follows cosine decay to zero. Training uses mixed precision and gradient norm clipping at . Progress is recorded through the reconstruction loss and learning rate, as shown in Figure 3.
Figure 3. Masked HOG reconstruction loss and learning rate during pretraining
Dimensionality Reduction and Clustering
After pretraining, the encoder is frozen and each image is processed without masking. Defect aware MAE uses the patch tokens from encoder layers to rather than the class token. The tokens from each layer are normalized, averaged across the patch positions, and then averaged across the layers:
where is the representation of patch from encoder layer , , and LN denotes layer normalization without learned affine parameters. The resulting feature matrix is standardized to zero mean and unit variance along each feature dimension.
Dimensionality Reduction
The standardized features of the held out images are projected into dimensions using UMAP [12]. The projection uses cosine distance, neighbours, a minimum distance of , and random seed . UMAP is fitted to the complete held out feature matrix before clustering.
Clustering
The dimensional UMAP representation is partitioned into clusters through agglomerative clustering with Ward linkage. The value of corresponds to the number of defect categories and is the only class information provided to the clustering procedure. Ward clustering begins with individual samples and repeatedly merges the pair of clusters that produces the smallest increase in within cluster variance.
Evaluation
Ground truth labels are used only after clustering to measure agreement between the discovered clusters and the six defect categories. Since cluster indices have no direct correspondence with class indices, the Hungarian algorithm determines the one to one mapping that maximizes agreement [21]. With ground truth label and cluster assignment , Hungarian matched accuracy is
The Adjusted Rand Index measures agreement between pairs of samples while correcting for chance [22]. Define
The Adjusted Rand Index is then
where denotes the number of samples shared by cluster and class , while and are the corresponding row and column sums.
Normalized Mutual Information measures the shared information between cluster assignments and ground truth labels [23]:
where is mutual information and denotes entropy.
Cluster compactness and separation are measured through the silhouette coefficient, which does not use ground truth labels [25]. The mean silhouette is
where, for sample , is its mean distance to the other samples in its assigned cluster and is its mean distance to the samples of the nearest other cluster.
The main evaluation reports accuracy, ARI, NMI, and silhouette for UMAP followed by Ward clustering. The same evaluation settings are applied to Defect aware MAE and the SimCLR baseline.
Results and Discussions
Defect aware MAE and SimCLR were evaluated on the 360 held out NEU DET images using UMAP followed by Ward clustering. Table I reports the mean and standard deviation across 5 seeds. Defect aware MAE achieved higher mean accuracy, NMI, ARI, and silhouette than SimCLR.
| Model | Accuracy (%) | NMI | ARI | Silhouette |
| SimCLR | 97.8 ± 1.3 | 0.955 ± 0.021 | 0.951 ± 0.027 | 0.736 ± 0.019 |
| Defect aware MAE | 98.6 ± 0.2 | 0.968 ± 0.003 | 0.967 ± 0.004 | 0.838 ± 0.019 |
Table I. Clustering results on the held out NEU DET split. Values are mean ± standard deviation across 5 seeds.
The silhouette score measured after PCA with 50 components was around 0.50 for Defect aware MAE and 0.20 for SimCLR. A linear probe trained on frozen features achieved 99.5% accuracy for Defect aware MAE and 98.8% for SimCLR. These measurements indicate that the Defect aware MAE features form more separated clusters and are more linearly separable under the same evaluation protocol.
Table II isolates the contribution of each design choice under the same held out evaluation. In the component ablation, all four models share the same learning rate, decoder and training length, and differ only in the reconstruction target and the masking strategy. Replacing pixel reconstruction with HOG targets raises accuracy from 76.9% to 92.8% and silhouette from 0.530 to 0.802. Adding defect aware masking to the HOG target raises accuracy further to 96.4%, with NMI and ARI increasing from 0.888 and 0.849 to 0.921 and 0.916. The two components are complementary: defect aware masking is most effective when the decoder predicts texture descriptors, as the masked high energy patches must then be reconstructed as edge and orientation structure rather than as pixel intensities.
| Configuration | Seeds | Accuracy (%) | NMI | ARI | Silhouette |
| Component ablation | |||||
| Pixel target, random masking | 1 | 76.9 | 0.701 | 0.631 | 0.530 |
| Pixel target, defect aware masking | 1 | 71.9 | 0.747 | 0.638 | 0.624 |
| HOG target, random masking | 1 | 92.8 | 0.888 | 0.849 | 0.802 |
| HOG target, defect aware masking | 1 | 96.4 | 0.921 | 0.916 | 0.824 |
| Incremental configuration | |||||
| Standard MAE (pixel target, random masking) | 1 | 73.9 | 0.772 | 0.667 | 0.584 |
| + HOG target and defect aware masking | 1 | 96.7 | 0.932 | 0.923 | 0.817 |
| + strong augmentation | 5 | 96.9 ± 1.8 | 0.938 | 0.930 | 0.849 |
| + 2 layer decoder | 3 | 97.9 ± 0.3 | 0.950 | 0.950 | 0.855 |
| + 1,600 epochs (Defect aware MAE) | 5 | 98.6 ± 0.2 | 0.968 | 0.967 | 0.838 |
Table II. Ablation study on the held out NEU DET split. Values with ± are mean ± standard deviation across seeds. All models use the intermediate layer readout.
The incremental configuration starts from a standard masked autoencoder with pixel targets and random masking, which reaches 73.9% accuracy. HOG reconstruction with defect aware masking raises accuracy to 96.7%. Strong augmentation increases the silhouette from 0.817 to 0.849, a 2 layer decoder raises accuracy to 97.9%, and training for 1,600 epochs gives the final 98.6% with NMI 0.968 and ARI 0.967.
Figure 4 shows the UMAP representation of the trained encoder. The six Ward clusters correspond to the six defect categories after Hungarian matching. The model achieved 98.33% accuracy, 0.963 NMI, 0.961 ARI, and a silhouette score of 0.827.

Figure 4. UMAP representation and Ward clusters of the released Defect aware MAE checkpoint
The corresponding confusion matrix is shown in Figure 5. All 60 images were assigned correctly for crazing, inclusion, patches, and rolled in scale. For pitted surface, 58 images were assigned correctly and 2 were assigned to crazing incorrectly. Moreover, for scratches, 56 images were assigned correctly and 4 predictions were inaccurately clustered as inclusion. In total, 354 of the 360 held out images were assigned to their true labels precisely.
Figure 5. Confusion matrix between ground truth labels and predicted labels for the Defect aware MAE.
The encoder representation was examined with Grad CAM and Grad CAM++ using similarity between each image feature and its assigned cluster centroid as the explanation target. No classification head or class label was used to generate the maps. Across all 360 held out images, Grad CAM produced an energy in box value of 0.405, an energy in box lift of 1.15, and a pointing game score of 0.43. Grad CAM++ produced corresponding values of 0.394, 0.98, and 0.38. Deletion scores were 0.640 for Grad CAM and 0.642 for Grad CAM++, compared with 0.664 under random patch order.
Figure 6 presents one example from each defect category. In the inclusion example, the Grad CAM response follows the vertical defect regions, while the scratches example places its main response along the horizontal scratch. Grad CAM++ distributes its response across the crazing texture and the pitted surface region.

Figure 6. Input images with annotations and explanation maps produced by Grad CAM and Grad CAM++ for the Defect aware MAE
surface region. The patches and rolled in scale examples show responses distributed across multiple textured regions. Across each class, Grad CAM achieved an energy in box lift of 2.18 for inclusion, 1.49 for scratches, and 1.04 for rolled in scale. The pointing game score for pitted surface was 0.55 with Grad CAM and 0.57 with Grad CAM++, while the Grad CAM++ score for crazing was 0.58.
Conclusion and Future Works
This work presented Defect aware MAE, a label free masked autoencoder for clustering steel surface defects. The method combines HOG reconstruction, texture energy based masking, image augmentation, a 2 layer decoder, and an intermediate layer feature readout. Pretraining uses no class labels, bounding boxes, detection branch, or contrastive loss. Across 5 seeds, UMAP followed by Ward clustering achieved 98.6% Hungarian matched accuracy with 0.968 NMI, 0.967 ARI values and got 0.838 silhouette score. Under the same encoder, data, optimizer, and learning rate schedule, SimCLR resulted in 97.8% accuracy with 0.955 NMI, 0.951 ARI, and 0.736 silhouette score. Grad CAM and Grad CAM++ provided spatial maps of the image regions contributing to cluster similarity. These results show that masked HOG reconstruction with texture guided masking produces more accurate and stable clustering than the evaluated contrastive baseline.
References
- [1]S. Zhou, Y. Cai, Z. Zhang, and J. Yin, “MESC DETR: An improved RT DETR algorithm for steel surface defect detection,” Electronics, vol. 14, no. 11, Art. no. 2232, 2025.DOI
- [2]W. Zhu, H. Zhang, C. Zhang, X. Zhu, Z. Guan, and J. Jia, “Surface defect detection and classification of steel using an efficient Swin Transformer,” Advanced Engineering Informatics, vol. 57, Art. no. 102061, 2023.DOI
- [3]Y. He, K. Song, Q. Meng, and Y. Yan, “An end to end steel surface defect detection approach via fusing multiple hierarchical features,” IEEE Transactions on Instrumentation and Measurement, vol. 69, no. 4, pp. 1493–1504, 2019.DOI
- [4]Y. Gao, G. Lv, D. Xiao, X. Han, T. Sun, and Z. Li, “Research on steel surface defect classification method based on deep learning,” Scientific Reports, vol. 14, Art. no. 8254, 2024.DOI
- [5]T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proc. Int. Conf. Mach. Learn., 2020, pp. 1597–1607.arxiv.org/abs/2002.05709
- [6]K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 16000–16009.arxiv.org/abs/2111.06377
- [7]A. Dosovitskiy et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. Learn. Represent., 2021.
- [8]C. Wei, H. Fan, S. Xie, C. Y. Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 14668–14678.arxiv.org/abs/2112.09133
- [9]N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 886–893.DOI
- [10]K. Song and Y. Yan, “A noise robust method based on completed local binary patterns for hot-rolled steel strip surface defects,” Applied Surface Science, vol. 285, pp. 858–864, 2013.DOI
- [11]X. Lv, F. Duan, J. Jiang, X. Fu, and L. Gan, “Deep metallic surface defect detection: The new benchmark and detection network,” Sensors, vol. 20, no. 6, Art. no. 1562, 2020.DOI
- [12]L. McInnes, J. Healy, and J. Melville, “UMAP: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018.arxiv.org/abs/1802.03426
- [13]R. Xu, R. Hao, and B. Huang, “Efficient surface defect detection using self supervised learning strategy and segmentation network,” Advanced Engineering Informatics, vol. 52, Art. no. 101566, 2022.DOI
- [14]A. Govind, C. P. Dubey, and J. Ray, “Multimodal deep learning based prediction of volumetric deformation of Barren Island using InSAR interferograms,” Authorea Preprints, 2025.
- [15]W. Kool, H. van Hoof, and M. Welling, “Stochastic beams and where to find them: The Gumbel top k trick for sampling sequences without replacement,” in Proc. Int. Conf. Mach. Learn., vol. 97, 2019, pp. 3499–3508.arxiv.org/abs/1903.06059
- [16]R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad CAM: Visual explanations from deep networks via gradient based localization,” in Proc. IEEE Int. Conf. Comput. Vis., Venice, Italy, 2017, pp. 618–626.
- [17]A. Chattopadhay, A. Sarkar, P. Howlader, and V. N. Balasubramanian, “Grad CAM++: Generalized gradient based visual explanations for deep convolutional neural networks,” in Proc. IEEE Winter Conf. Appl. Comput. Vis., Lake Tahoe, NV, USA, 2018, pp. 839–847.DOI
- [18]A. B. George and S. Bathini, “Grad CAM and Grad CAM++ for explainable oral squamous cell carcinoma detection using deep learning on orthopantomograms,” in Proc. 2025 Int. Conf. Sensors Relat. Netw., Special Focus on Digital Healthcare, 2025.
- [19]K. Zuiderveld, “Contrast limited adaptive histogram equalization,” in Graphics Gems IV, P. S. Heckbert, Ed. San Diego, CA, USA: Academic Press, 1994, pp. 474–485.DOI
- [20]I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. Int. Conf. Learn. Represent., 2019.
- [21]H. W. Kuhn, “The Hungarian method for the assignment problem,” Naval Research Logistics Quarterly, vol. 2, nos. 1 and 2, pp. 83–97, 1955.
- [22]L. Hubert and P. Arabie, “Comparing partitions,” Journal of Classification, vol. 2, no. 1, pp. 193–218, 1985.DOI
- [23]N. X. Vinh, J. Epps, and J. Bailey, “Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance,” Journal of Machine Learning Research, vol. 11, pp. 2837–2854, 2010.
- [24]X. Hu, J. Yang, F. Jiang, A. Hussain, K. Dashtipour, and M. Gogate, “Steel surface defect detection based on self supervised contrastive representation learning with matching metric,” Applied Soft Computing, vol. 145, 2023.
- [25]P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics, vol. 20, pp. 53–65, 1987.DOI


