Attention Weights Research Articles

Development of an Algorithm for Detecting Real-Time Defects in Steel

The integration of artificial intelligence with steel manufacturing operations holds great potential for enhancing factory efficiency. Object detection algorithms, as a category within the field of artificial intelligence, have been widely adopted for steel defect detection purposes. However, mainstream object detection algorithms often exhibit a low detection accuracy and high false-negative rates when it comes to detecting small and subtle defects in steel materials. In order to enhance the production efficiency of steel factories, one approach could be the development of a novel object detection algorithm to improve the accuracy and speed of defect detection in these facilities. This paper proposes an improved algorithm based on the YOLOv5s-7.0 version, called YOLOv5s-7.0-FCC. YOLOv5s-7.0-FCC integrates the basic operator C3-Faster (C3F) into the C3 module. Its special T-shaped structure reduces the redundant calculation of channel features, increases the attention weight on the central content, and improves the algorithm’s computational speed and feature extraction capability. Furthermore, the spatial pyramid pooling-fast (SPPF) structure is replaced by the Content Augmentation Module (CAM), which enriches the image feature content with different convolution rates to simulate the way humans observe things, resulting in enhanced feature information transfer during the process. Lastly, the upsampling operator Content-Aware ReAssembly of Features (CARAFE) replaces the “nearest” method, transforming the receptive field size based on the difference in feature information. The three modules that act on feature information are distributed reasonably in YOLOv5s-7.0, reducing the loss of feature information during the convolution process. The results show that compared to the original YOLOv5 model, YOLOv5s-7.0-FCC increases the mean average precision (mAP) from 73.1% to 79.5%, achieving a 6.4% improvement. The detection speed also increased from 101.1 f/s to 109.4 f/s, an improvement of 8.3 f/s, further meeting the accuracy requirements for steel defect detection.

Optic cup and disc segmentation model based on linear attention and dual attention

Glaucoma is one of blind causing diseases. The cup-to-disc ratio is the main basis for glaucoma screening. Therefore, it is of great significance to precisely segment the optic cup and disc. In this article, an optic cup and disc segmentation model based on the linear attention and dual attention is proposed. Firstly, the region of interest is located and cropped according to the characteristics of the optic disc. Secondly, linear attention residual network-34 (ResNet-34) is introduced as a feature extraction network. Finally, channel and spatial dual attention weights are generated by the linear attention output features, which are used to calibrate feature map in the decoder to obtain the optic cup and disc segmentation image. Experimental results show that the intersection over union of the optic disc and cup in Retinal Image Dataset for Optic Nerve Head Segmentation (DRISHTI-GS) dataset are 0.962 3 and 0.856 4, respectively, and the intersection over union of the optic disc and cup in retinal image database for optic nerve evaluation (RIM-ONE-V3) are 0.956 3 and 0.784 4, respectively. The proposed model is better than the comparison algorithm and has certain medical value in the early screening of glaucoma. In addition, this article uses knowledge distillation technology to generate two smaller models, which is beneficial to apply the models to embedded device.

LWR-Net: Robust and Lightweight Place Recognition Network for Noisy and Low-Density Point Clouds.

Point cloud-based retrieval for place recognition is essential in robotic applications like autonomous driving or simultaneous localization and mapping. However, this remains challenging in complex real-world scenes. Existing methods are sensitive to noisy, low-density point clouds and require extensive storage and computation, posing limitations for hardware-limited scenarios. To overcome these challenges, we propose LWR-Net, a lightweight place recognition network for efficient and robust point cloud retrieval in noisy, low-density conditions. Our approach incorporates a fast dilated sampling and grouping module with a residual MLP structure to learn geometric features from local neighborhoods. We also introduce a lightweight attentional weighting module to enhance global feature representation. By utilizing the Generalized Mean pooling structure, we aggregated the global descriptor for point cloud retrieval. We validated LWR-Net's efficiency and robustness on the Oxford robotcar dataset and three in-house datasets. The results demonstrate that our method efficiently and accurately retrieves matching scenes while being more robust to variations in point density and noise intensity. LWR-Net achieves state-of-the-art accuracy and robustness with a lightweight model size of 0.4M parameters. These efficiency, robustness, and lightweight advantages make our network highly suitable for robotic applications relying on point cloud-based place recognition.

Adapt to small-scale and long-term time series forecasting with enhanced multidimensional correlation

Multivariate time series forecasting aims to predict time series data comprising several linked variables or characteristicsand is frequently used in stock forecasting, energy forecasting, etc. The tough task is to acquire furtherhistorical data to forecast future values while boosting the capacity to mine relationships between and within sequences. The existing methods neglect the weighted influence of various neighbors and relationships on the sequence instance, as well as the semantic information of the instance itself, making the inter-instance correlation measure inaccurate. Meanwhile, the length of the sequence input is limited and the variable features in the multivariate sequence are treated equally, short-term and multi-feature interference cannot be eliminated, causing the learnt time series features to deviate from the real features. In this work, we provide a Multivariate Time Series Forecasting model that emphasizes Relationships between and within sequences (MTSFR). Use the BERT model to characterize the text attributes of instances and construct basic semantic embeddings. Meanwhile, instance-level and relation-level attention are used to model topological relations among different instances. Computes the cross-correlation of multivariate sequences within instances and performs attention weighting for multivariate sequence encoding. And choose the Transformer model to realize the trend prediction of long-term multivariate series. Experimental results show that the F1 value of our approach achieves 68.50% and 74.66% under the CSI300 and S&P500 data sets respectively, both of which are superior to the SOTA technique. Furthermore, the model is suited for small-scale sequence relationship modeling and is effective at handling long-term multivariate sequence forecasting problems.

Research on construction and management strategy of carbon neutral stadiums based on CNN-QRLSTM model combined with dynamic attention mechanism

IntroductionLarge-scale construction projects such as sports stadiums are known for their significant energy consumption and carbon emissions, raising concerns about sustainability. This study addresses the pressing issue of developing carbon-neutral stadiums by proposing an integrated approach that leverages advanced convolutional neural networks (CNN) and quasi-recurrent long short-term memory (QRLSTM) models, combined with dynamic attention mechanisms.MethodsThe proposed approach employs the CNN-QRLSTM model, which combines the strengths of CNN and QRLSTM to handle both image and sequential data. Additionally, dynamic attention mechanisms are integrated to adaptively adjust attention weights based on varying situations, enhancing the model's ability to capture relevant information accurately.ResultsExperiments were conducted using four datasets: EnergyPlus, ASHRAE, CBECS, and UCl. The results demonstrated the superiority of the proposed model compared to other advanced models, achieving the highest scores of 97.79% accuracy, recall rate, F1 score, and AUC.DiscussionThe integration of deep learning models and dynamic attention mechanisms in stadium construction and management offers a more scientific decision support system for stakeholders. This approach facilitates sustainable choices in carbon reduction and resource utilization, contributing to the development of carbon-neutral stadiums.

A Machine Learning Framework for Enhancing Short-Term Water Demand Forecasting Using Attention-BiLSTM Networks Integrated with XGBoost Residual Correction

Accurate short-term water demand forecasting assumes a pivotal role in optimizing water supply control strategies, constituting a cornerstone of effective water management. In recent times, the rise of machine learning technologies has ushered in hybrid models that exhibit superior performance in this domain. Given the intrinsic non-linear fluctuations and variations in short-term water demand sequences, achieving precise forecasts presents a formidable challenge. Against this backdrop, this study introduces an innovative machine learning framework for short-term water demand prediction. The maximal information coefficient (MIC) is employed to select high-quality input features. A deep learning architecture is devised, featuring an Attention-BiLSTM network. This design leverages attention weights and the bidirectional information in historical sequences to highlight influential factors and enhance predictive capabilities. The integration of the XGBoost algorithm as a residual correction module further bolsters the model’s performance by refining predicted results through error simulation. Hyper-parameter configurations are fine-tuned using the Keras Tuner and random parameter search. Through rigorous performance comparison with benchmark models, the superiority and stability of this method are conclusively demonstrated. The attained results unequivocally establish that this approach outperforms other models in terms of predictive accuracy, stability, and generalization capabilities, with MAE, RMSE, MAPE, and NSE values of 544 m3/h, 915 m3/h, 1.00%, and 0.99, respectively. The study reveals that the incorporation of important features selected by the MIC, followed by their integration into the attention mechanism, essentially subjects these features to a secondary filtration. While this enhances model performance, the potential for improvement remains limited. Our proposed forecasting framework offers a fresh perspective and contribution to the short-term water resource scheduling in smart water management systems.

BiLSTM- and GNN-Based Spatiotemporal Traffic Flow Forecasting with Correlated Weather Data

The timely and accurate forecasting of urban road traffic is crucial for smart city traffic management and control. It can assist both drivers and traffic controllers in selecting efficient routes and diverting traffic to less congested roads. However, estimating traffic volume while taking into account external factors such as weather and accidents is still a challenge. In this research, we propose a hybrid deep learning framework, double attention graph neural network BiLSTM (DAGNBL), that utilizes a graph neural network to represent spatial characteristics and bidirectional LSTM units to capture temporal dependencies between features. Attention modules are added to the GNN and BLSTM to find high-impact attention weight values for the chosen road section. Our model offers the best prediction accuracy with a mean absolute percentage error of 5.21% and a root mean squared error of 4. It can be utilized as a useful tool for predicting traffic flow on certain stretches of road.

Histopathologic brain age estimation via multiple instance learning

Understanding age acceleration, the discordance between biological and chronological age, in the brain can reveal mechanistic insights into normal physiology as well as elucidate pathological determinants of age-related functional decline and identify early disease changes in the context of Alzheimer’s and other disorders. Histopathological whole slide images provide a wealth of pathologic data on the cellular level that can be leveraged to build deep learning models to assess age acceleration. Here, we used a collection of digitized human post-mortem hippocampal sections to develop a histological brain age estimation model. Our model predicted brain age within a mean absolute error of 5.45 ± 0.22 years, with attention weights corresponding to neuroanatomical regions vulnerable to age-related changes. We found that histopathologic brain age acceleration had significant associations with clinical and pathologic outcomes that were not found with epigenetic based measures. Our results indicate that histopathologic brain age is a powerful, independent metric for understanding factors that contribute to brain aging.

Road Scene Instance Segmentation Based on Improved SOLOv2

Road instance segmentation is vital for autonomous driving, yet the current algorithms struggle in complex city environments, with issues like poor small object segmentation, low-quality mask edge contours, slow processing, and limited model adaptability. This paper introduces an enhanced instance segmentation method based on SOLOv2. It integrates the Bottleneck Transformer (BoT) module into VoVNetV2, replacing the standard convolutions with ghost convolutions. Additionally, it replaces ResNet with an improved VoVNetV2 backbone to enhance the feature extraction and segmentation speed. Furthermore, the algorithm employs Feature Pyramid Grids (FPGs) instead of Feature Pyramid Networks (FPNs) to introduce multi-directional lateral connections for better feature fusion. Lastly, it incorporates a convolutional Block Attention Module (CBAM) into the detection head for refined features by considering the attention weight coefficients in both the channel and spatial dimensions. The experimental results demonstrate the algorithm’s effectiveness, achieving a 27.6% mAP on Cityscapes, a 4.2% improvement over SOLOv2. It also attains a segmentation speed of 8.9 FPS, a 1.7 FPS increase over SOLOv2, confirming its practicality for real-world engineering applications.

A Network Intrusion Detection Model Based on BiLSTM with Multi-Head Attention Mechanism

A network intrusion detection tool can identify and detect potential malicious activities or attacks by monitoring network traffic and system logs. The data within intrusion detection networks possesses characteristics that include a high degree of feature dimension and an unbalanced distribution across categories. Currently, the actual detection accuracy of some detection models is relatively low. To solve these problems, we propose a network intrusion detection model based on multi-head attention and BiLSTM (Bidirectional Long Short-Term Memory), which can introduce different attention weights for each vector in the feature vector that strengthen the relationship between some vectors and the detection attack type. The model also utilizes the advantage that BiLSTM can capture long-distance dependency relationships to obtain a higher detection accuracy. This model combined the advantages of the two models, adding a dropout layer between the two models to improve the detection accuracy while preventing training overfitting. Through experimental analysis, the network intrusion detection model that utilizes multi-head attention and BilSTM achieved an accuracy of 98.29%, 95.19%, and 99.08% on the KDDCUP99, NSLKDD, and CICIDS2017 datasets, respectively.

IFRN: Insensitive feature removal network for zero-shot mechanical fault diagnosis across fault severity

Zero-shot learning is a promising technique for diagnosing mechanical faults in complex and uncertain environments. However, when diagnosing mechanical faults across different severities using zero-shot learning, the impact of insensitive features should be minimized due to the susceptibility of zero-shot prototypes and the vulnerability of vibration signals. To accomplish this, an Insensitive Feature Removal Network (IFRN) with an entire attention mechanism (EAM) module and a denoise autoencoder module is proposed to remove insensitive features hierarchically by dividing them into two categories: common insensitive features (CIF) and private insensitive features (PIF), each with different properties depending on their corresponding sub-labels presence in mechanical faults. Concretely, EAM removes insensitive features that the classifier cannot differentiate with the help of the entire attention weight comparison by attention generation, attention comparison, and attention limitation parts. Then, the denoise autoencoder module with long short memory term (LSTM) is utilized to remove the insensitive features that are not completely removed, especially for the insensitive features that independently arise. The IFRN’s effectiveness is demonstrated through comparative experiments and ablation studies using the Case Western Reserve University (CWRU) dataset, where the experimental result shows that IFRN outperforms conventional zero-shot learning methods. Furthermore, an analysis with prototype distance and sample aggregation is presented to further justify the effectiveness of the proposed method in reducing the prototype shift and improving classification accuracy by removing insensitive features.

Improving agricultural field parcel delineation with a dual branch spatiotemporal fusion network by integrating multimodal satellite data

Accurate spatial information for agricultural field parcels is important for agricultural production management and understanding agro-industrialization and intensification. However, traditional remote sensing methods that rely on single-modal or single-date data struggle to identify heterogeneous field parcels, particularly in regions dominated by smallholder farming systems. To address this challenge, we proposed a Dual branch Spatiotemporal Fusion Network (DSTFNet) that integrated very high-resolution (VHR) images and medium-resolution satellite image time series (MRSITS) to extract agricultural field parcels over various landscapes. The DSTFNet consisted of two branches: a spatial branch that extracted spatial features from VHR images and a temporal branch that explored seasonal spectral dynamics from MRSITS data by using ConvLSTM units and an attention module. We evaluated the DSTFNet in four regions across China by using GF-2 and Sentinel-2 data. The results showed that DSTFNet performed well in delineating agricultural field parcels, achieving the highest Matthew’s correlation coefficient (MCC) = 0.823 for the field extent, the highest F1-score of edge (Fedge) = 0.865 for field boundary, and the lowest errors of segmentation evaluation index (SEI) = 0.191 for the vectorized field parcels in Hubei province. In addition, DSTFNet significantly outperformed two single-branch models that used VHR or MRSITS alone, as well as existing BsiNet, ResUNet_a, UNet and RAUNet models. DSTFNet also showed good spatial transferability in distinct regions without training data (on average, MCC = 0.728, Fedge = 0.729, and SEI = 0.281 for three target regions). Using limited training data to fine-tune the DSTFNet can further improve its ability to delineate field parcels over complex regions. The visualization analysis of temporal attention weights demonstrated that DSTFNet can well capture cropland spectral dynamics, making it advantageous in extracting diverse cropland parcels. By exploiting important spectral, spatial and temporal information from multimodal satellite data, DSTFNet provided an effective, robust, and transferable solution for accurately delineating agricultural field parcels across heterogeneous farming systems.

A Lightweight Network for Contextual and Morphological Awareness for Hepatic Vein Segmentation.

Accurate segmentation of the hepatic vein is crucial for enhancing the precision of liver disease diagnosis and treatment. Since the hepatic venous system is a small target and sparsely distributed, with various and diverse morphology, data labeling is difficult. Therefore, automatic hepatic vein segmentation is extremely challenging. We propose a lightweight contextual and morphological awareness network and design a novel morphology aware module based on attention mechanism and a 3D reconstruction module. The morphology aware module can obtain the slice similarity awareness mapping, which can enhance the continuous area of the hepatic veins in two adjacent slices through attention weighting. The 3D reconstruction module connects the 2D encoder and the 3D decoder to obtain the learning ability of 3D context with a very small amount of parameters. The experimental results on the public dataset show that when using the proposed network structure, it only needs less than 1/14 of the parameters of U-Net to outperform other comparison methods. Meanwhile, compared with other SOTA methods, using the proposed method demonstrates an enhancement in the dice coefficient, achieving an improvement of 2.86% and 1.45% on the two datasets, respectively. A small number of parameters can reduce hardware requirements and potentially have stronger generalization, which is an advantage in clinical deployment.

Partial Point Cloud Registration With Deep Local Feature

How to accurately register partial point cloud still remains a challenging task, because of its irregular and unordered structure in a non-Euclidean space, noise, outliers, and other unfavorable factors. In this paper, an effective partial point cloud registration network is proposed, by devising a two-stage deep local feature extraction process and an outlier filtering strategy. To be specific, on the one hand, to effectively capture geometric interdependency in low-level space, a local attention feature extraction module is explored to extract local contextual attention features, by highlighting different attention weights on neighborhoods. On the other hand, in local feature aggregation module, two position encoding blocks are applied to increase the receptive field of each point in high-level space. Of these, an attentive pooling can automatically learn important local features to alleviate the possible information loss. Furthermore, to derive the weight of the putative correspondence, an outlier filtering module is designed by consisting of point context normalization block, differentiable pooling and differentiable unpooling layer. Moreover, in order to enhance robustness, a weighting point cloud registration model is formulated to alleviate outliers, by considering the contribution of each correspondence. Experiments on multiple datasets demonstrate that the proposed approach is competitive to several state-of-the-art algorithms. Code is publicly available at <uri>https://github.com/zhlSunLab/DLF</uri>.

Short-term wind power forecasting by bidirectional attention mechanism LSTM and its probability interval prediction by sliding-window KDE

Deterministic point wind power forecasting (DP-WPF) and its probability interval prediction (PIP) are indispensable to short-term peak alleviation and frequency regulation in power systems with large-scale wind power injection. To improve short-term DP-WPF by long short-term memory (LSTM), a horizontal/vertical bidirectional feature attention (BFA) based LSTM model is proposed. More specifically, the BFA-LSTM model has three parts: first, multivariate time series are fed into LSTM to extract long-short-term temporal features; second, the LSTM outputs are processed horizontally as well as vertically for retrieving step-wise/multistep-wise temporal features, respectively, namely, in the bidirectional attention sense; third, both the horizontal and vertical attention weights are adaptively adjusted according to the feature importance in DP-WPF. Cases comparison shows that the suggested modeling is stably superior to most common counterparts. To address PIP by kernel density estimation (KDE), sliding-window KDE is leveraged for probability analysis. More precisely, probability density functions (PDF) and probability intervals are estimated with sliding-window samples, which are non-parametric operations and involve finitely many local samples. Superior performances of PIP by sliding-window KDE at different confidence levels indicate that the sliding-window PDF approach is highly effective in contrast to those with all samples.

Vicinity Vision Transformer.

Vision transformers have shown great success on numerous computer vision tasks. However, their central component, softmax attention, prohibits vision transformers from scaling up to high-resolution images, due to both the computational complexity and memory footprint being quadratic. Linear attention was introduced in natural language processing (NLP) which reorders the self-attention mechanism to mitigate a similar issue, but directly applying existing linear attention to vision may not lead to satisfactory results. We investigate this problem and point out that existing linear attention methods ignore an inductive bias in vision tasks, i.e., 2D locality. In this paper, we propose Vicinity Attention, which is a type of linear attention that integrates 2D locality. Specifically, for each image patch, we adjust its attention weight based on its 2D Manhattan distance from its neighbouring patches. In this case, we achieve 2D locality in a linear complexity where the neighbouring image patches receive stronger attention than far away patches. In addition, we propose a novel Vicinity Attention Block that is comprised of Feature Reduction Attention (FRA) and Feature Preserving Connection (FPC) in order to address the computational bottleneck of linear attention approaches, including our Vicinity Attention, whose complexity grows quadratically with respect to the feature dimension. The Vicinity Attention Block computes attention in a compressed feature space with an extra skip connection to retrieve the original feature distribution. We experimentally validate that the block further reduces computation without degenerating the accuracy. Finally, to validate the proposed methods, we build a linear vision transformer backbone named Vicinity Vision Transformer (VVT). Targeting general vision tasks, we build VVT in a pyramid structure with progressively reduced sequence length. We perform extensive experiments on CIFAR-100, ImageNet-1k, and ADE20K datasets to validate the effectiveness of our method. Our method has a slower growth rate in terms of computational overhead than previous transformer-based and convolution-based networks when the input resolution increases. In particular, our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous approaches.

Online Intention Recognition With Incomplete Information Based on a Weighted Contrastive Predictive Coding Model in Wargame.

The incomplete and imperfect essence of the battlefield situation results in a challenge to the efficiency, stability, and reliability of traditional intention recognition methods. For this problem, we propose a deep learning architecture that consists of a contrastive predictive coding (CPC) model, a variable-length long short-term memory network (LSTM) model, and an attention weight allocator for online intention recognition with incomplete information in wargame (W-CPCLSTM). First, based on the typical characteristics of intelligence data, a CPC model is designed to capture more global structures from limited battlefield information. Then, a variable-length LSTM model is employed to classify the learned representations into predefined intention categories. Next, a weighted approach to the training attention of CPC and LSTM is introduced to allow for the stability of the model. Finally, performance evaluation and application analysis of the proposed model for the online intention recognition task were carried out based on four different degrees of detection information and a perfect situation of ideal conditions in a wargame. Besides, we explored the effect of different lengths of intelligence data on recognition performance and gave application examples of the proposed model to a wargame platform. The simulation results demonstrate that our method not only contributes to the growth of recognition stability, but it also improves recognition accuracy by 7%-11%, 3%-7%, 3%-13%, and 3%-7%, the recognition speed by 6-32x, 4-18x, 13-*x, and 1-6x compared with the traditional LSTM, classical FCN, OctConv, and OctFCN models, respectively, which characterizes it as a promising reference tool for command decision-making.

Research on Video Anomaly Detection Based on Cascaded Memory-augmented Autoencoder

&lt;p&gt;Video anomaly detection (VAD) is crucial in public safety and intelligent video surveillance systems and has been widely researched and applied in academia. This paper proposes a video anomaly detection method based on the Cascaded Memory-augmented Autoencoder (CMAAE). CMAAE stores feature prototypes of standard samples in a memory pool and embed multiple memory-enhancing modules in the encoder-decoder structure. SE attention is introduced into memory modules to improve their performance, and skip connections are used to share attention weights among memory modules, enabling the model to learn more comprehensive feature information and enhance the quality of reconstructed video frames. Multiple loss constraint models are used during training to improve anomaly detection accuracy. CMAAE achieves outstanding performance of 99.2% on the UCSD Ped2 dataset and 89.4% on the CUHK Avenue dataset, demonstrating the effectiveness of our approach.&lt;/p&gt; &lt;p&gt;&amp;nbsp;&lt;/p&gt;

Hybrid Regularizations for Multi-Aspect Category Sentiment Analysis

Aspect level sentiment classification aims to identify the sentiment polarity towards a particular aspect in a sentence. Previous attention-based methods generate an aspect-specific representation for each aspect and employ it to classify the sentiment polarity. However, normalized attention scores scatter over every word in the sentence, resulting in two issues. First, the attention may inherently introduce noise and downgrade the performance. Second, the opinion words may be “diluted” by other words, while the opinion feature should dominate for sentiment analysis. The issues become more severe in multi-aspect sentences. In this paper, we address the above two issues via hybrid regularizations, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">aspect-level</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">task-level regularizations</i> . Concretely, the aspect-level regularizations constrain the attention weights to alleviate noise. Among them, orthogonal regularization is designed for multi-aspect sentences and sparse regularization is for single-aspect sentences. To extract sentiment-dominant features, task-level regularization is proposed by introducing an orthogonal auxiliary task, i.e., aspect category detection. This regularization can allocate task-oriented context information for specific downstream tasks. Extensive experimental results on three public datasets demonstrate the effectiveness of the proposed approach in both single-task and multi-task scenarios.

Medimatrix: innovative pre-training of grayscale images for rheumatoid arthritis diagnosis revolutionises medical image classification.

Efficient and accurate medical image classification (MIC) methods face two major challenges: (1) high similarity between images of different disease classes; and (2) generating large medical image datasets for training deep neural networks is challenging due to privacy restrictions and the need for expert ground truth annotations. In this paper, we introduce a novel deep learning method called pre-training grayscale images with supervised learning for MIC (MediMatrix). Instead of pre-training on color ImageNet, our approach uses MediMatrix on grayscale ImageNet. To improve the performance of the network, we introduce ShuffleAttention (SA), a self-attention mechanism. By combining SA with the multiple residual structure (ResSA block) and replacing short-cut connections with dense residual connections between corresponding layers (densepath), our network can dynamically adjust channel attention weights and receive image inputs of different sizes, resulting in improved feature representation and better discrimination of similarities between different categories. MediMatrix effectively classifies X-ray images of rheumatoid arthritis (RA), enabling efficient screening without the need for expert analysis or invasive testing. Through extensive experiments, we demonstrate the superiority of MediMatrix over state-of-the-art methods and that color is not critical for rich natural image classification. Our results highlight the potential of computer-aided diagnosis combined with MediMatrix as a valuable screening tool for early detection and intervention in RA.