Fine-tuning Strategy Research Articles

COMPARATIVE ANALYSIS OF MULTILINGUAL QA MODELS AND THEIR ADAPTATION TO THE KAZAKH LANGUAGE

This paper presents a comparative analysis of large pretrained multilingual models for question-answering (QA) systems, with a specific focus on their adaptation to the Kazakh language. The study evaluates models including mBERT, XLM-R, mT5, AYA, and GPT, which were tested on QA tasks using the Kazakh sKQuAD dataset. To enhance model performance, fine-tuning strategies such as adapter modules, data augmentation techniques (back-translation, paraphrasing), and hyperparameter optimization were applied. Specific adjustments to learning rates, batch sizes, and training epochs were made to boost accuracy and stability. Among the models tested, mT5 achieved the highest F1 score of 75.72%, showcasing robust generalization across diverse QA tasks. GPT-4-turbo closely followed with an F1 score of 73.33%, effectively managing complex Kazakh QA scenarios. In contrast, native Kazakh models like Kaz-RoBERTa showed improvements through fine-tuning but continued to lag behind larger multilingual models, underlining the need for additional Kazakh-specific training data and further architectural enhancements. Kazakh’s agglutinative morphology and the scarcity of high-quality training data present significant challenges for model adaptation. Adapter modules helped mitigate computational costs, allowing efficient fine-tuning in resource-constrained environments without significant performance loss. Data augmentation techniques, such as back-translation and paraphrasing, were instrumental in enriching the dataset, thereby enhancing model adaptability and robustness. This study underscores the importance of advanced fine-tuning and data augmentation strategies for QA systems tailored to low-resource languages like Kazakh. By addressing these challenges, this research aims to make AI technologies more inclusive and accessible, offering practical insights for improving natural language processing (NLP) capabilities in underrepresented languages. Ultimately, these findings contribute to bridging the gap between high-resource and low-resource language models, fostering a more equitable distribution of AI solutions across diverse linguistic contexts.

A deep learning approach for deriving wheat phenology from near-surface RGB image series using spatiotemporal fusion

Accurate monitoring of wheat phenological stages is essential for effective crop management and informed agricultural decision-making. Traditional methods often rely on labour-intensive field surveys, which are prone to subjective bias and limited temporal resolution. To address these challenges, this study explores the potential of near-surface cameras combined with an advanced deep-learning approach to derive wheat phenological stages from high-quality, real-time RGB image series. Three deep learning models based on three different spatiotemporal feature fusion methods, namely sequential fusion, synchronous fusion, and parallel fusion, were constructed and evaluated for deriving wheat phenological stages with these near-surface RGB image series. Moreover, the impact of different image resolutions, capture perspectives, and model training strategies on the performance of deep learning models was also investigated. The results indicate that the model using the sequential fusion method is optimal, with an overall accuracy (OA) of 0.935, a mean absolute error (MAE) of 0.069, F1-score (F1) of 0.936, and kappa coefficients (Kappa) of 0.924 in wheat phenological stages. Besides, the enhanced image resolution of 512 × 512 pixels and a suitable image capture perspective, specifically a sensor viewing angle of 40° to 60° vertically, introduce more effective features for phenological stage detection, thereby enhancing the model’s accuracy. Furthermore, concerning the model training, applying a two-step fine-tuning strategy will also enhance the model’s robustness to random variations in perspective. This research introduces an innovative approach for real-time phenological stage detection and provides a solid foundation for precision agriculture. By accurately deriving critical phenological stages, the methodology developed in this study supports the optimization of crop management practices, which may result in improved resource efficiency and sustainability across diverse agricultural settings. The implications of this work extend beyond wheat, offering a scalable solution that can be adapted to monitor other crops, thereby contributing to more efficient and sustainable agricultural systems.

Passive control of thermoacoustic instability in a high-efficiency domestic gas stove with fine-tuning burner

Thermoacoustic oscillation is a significant challenge in the development of advanced thermal power and propulsion systems. The present work focuses on the whistling phenomenon in the practical burners of a developing high-efficiency domestic gas stove. The acoustic analysis is conducted by a Helmholtz acoustic solver, identifying the 1/2 wave longitudinal mode in the outer ring or center nozzle and its upstream pipe and estimating the modal eigenfrequencies fa=599Hz and fb=647Hz which are close to the main frequency of the whistling measured in experiments. The sequential flame images captured by a high-speed camera are processed by a dynamic mode decomposition method to illustrate flame structure oscillations at the peak frequency mode. The results verify that thermoacoustic oscillations are the generation mechanism of the whistling and display the periodic fluctuation characteristics of the flame and the spatial distribution of heat release. Moreover, both structural fine-tuning designs help to achieve better suppression for thermoacoustic oscillations and avoid the whistling phenomenon. These structural fine-tuning strategies have been researched for several decades but rarely reported to be applied to industrial burners in previous work. Motivated by this, it will provide a new insight into the investigation of burner structures on thermoacoustic behaviors and the application of these passive control strategies.

Synthesis and Upconversion Luminescence Fine-tuning of Yb3+/Ho3+-Doped Indium and Gallium Oxide Nanoparticles.

Rare earth (RE) dopants can modulate the bandgap of oxides of indium and gallium and provide extra upconversion luminescence (UCL) abilities. However, relevant UCL fine-tuning strategies and energy mechanisms have been less studied. In this research, InGaO, Ho3+ monodoped and Yb3+/Ho3+ codoped In2O3, and Ho3+ monodoped Yb3Ga5O12 nanoparticles (NPs) were synthesized by a solvothermal method. The effects of Yb3+ and Ho3+ dopants on the crystal structures, UCL properties, and optical bandgaps of the oxides were characterized by X-ray diffraction (XRD), transmission electron microscopy (TEM), UCL spectroscopy, and measurements of decay times, pump power dependence, and transmittance spectra. The crystal structures of oxide products of indium and gallium were significantly modified with RE dopants. In2O3 and Yb3Ga5O12 were selected as the host materials. For Yb3+/Ho3+ codoped In2O3 NPs, there existed energy transfers from the defect states of In2O3 to Ho3+ and from Yb3+ to Ho3+. With a fixed Ho3+ concentration, In2O3:0%Yb3+,2%Ho3+ NPs showed the optimal UCL properties mainly due to In2O3-Ho3+ energy transfer and Ho3+-Yb3+ energy-back-transfer, while with a fixed Yb3+ concentration, In2O3:5%Yb3+,3%Ho3+ NPs with a slight Yb2O3 impurity and Yb3Ga5O12:2%Ho3+ NPs did mainly due to Ho3+-Ho3+ cross-relaxation. Besides, the optical bandgaps of In2O3 and Yb3Ga5O12 were noticeably broadened with RE dopants. These findings can offer feasible directions for the synthesis and UCL fine-tuning of RE-doped oxides of indium and gallium and improve their multifunction application prospects in the fields of semiconductor and UCL nanomaterials.

SSRLM: A self-supervised representation learning method for identifying one ship with multi-MMSI codes

It is of great practical significance for maritime safety and shipping development to achieve “One Ship with One Maritime Mobile Service Identity (MMSI) Code” in accordance with regulations. However, the current “One Ship with Multiple MMSI Codes” violation identification method lacks practicability and generalization. Aiming at the these problems, this paper proposes a self-supervised representation learning model for the “One Ship with Multiple MMSI Codes” abnormal behavior recognition task, named SSRLM. Firstly, we construct a new dataset about “One Ship with Multiple MMSI Codes”, named HN_MulMI. Secondly, the SSRLM model learns the contextual interdependencies among different trajectory features in the HN_MulMI dataset through an unsupervised pre-training scheme, and extracts a dense vectorial representation of the ship’s motion trajectories. Finally, by using the unique characteristics of the “One Ship with Multiple MMSI Codes” trajectory sequence, the SSRLM model captures its feature dependencies through a fine-tuning scheme to accomplish the abnormal behavior detection task. Experimental results in the two scenarios of HN_MulMI test set demonstrate that the proposed model outperform the state-of-the-art models in recent years, the average precision, recall and F1-score rates is up to about 99.26% and 93.5%. Meanwhile, we also conducted comparative experiments on three public datasets to verify the generalization performance, achieving 90.31%, 93.2% and 95.14% results in the F1 score evaluation indicators, further confirming the good recognition performance of the SSLLM model. Finally, our model has been applied to the actual scene and achieved good results.

A high-efficiency local and global detector for diatom-based drowning diagnosis

Detection and classification of diatoms has been considered as the most effective method for the drowning diagnosis in forensic practice. However, challenges (such as limited samples, complex background interference, and detecting multiple objects) remain during detection and classification. To address these challenges, we propose a MDMS (multi-scale dynamic multi-head self-attention)-based deep learning network for the diatom-based drowning diagnosis. This framework uses TFFT (two-stage full fine-tuning training strategy) to solve the challenges of limited diatom samples and detecting multiple objects, and builds a multi-scale dynamic multi-head self-attention module to extract the local and global features of diatom images. In the meantime, we introduce an online hard example mining strategy to attenuate the complex background interference. The experimental results show that the proposed framework can effectively reduce the missing and false detection rates of diatom objects, with the mAP (mean Average Precision) reaching 92.434%, which is better than the result using the mainstream methods.

Risks and Opportunities Brought by Artificial Intelligence Empowering the Financial Industry

Intelligent risk management and trade optimization play pivotal roles in the realm of investment banking. Leveraging AI's capabilities, institutions harness vast datasets and machine learning algorithms to detect potential risks and fine-tune trading strategies. This results in more accurate decision-making processes and better risk mitigation measures. Furthermore, AI enables personalized customer service and precise market forecasting, elevating client satisfaction levels and enabling the anticipation of market trends with greater precision. Tailored services foster stronger client relationships, driving long-term partnerships and loyalty. In addition to enhancing operational efficiency and client satisfaction, AI drives innovation and development within investment banking. This includes the introduction of intelligent advisory services and the implementation of digital perception systems, revolutionizing traditional banking practices. However, the integration of AI also presents challenges such as data privacy concerns, regulatory compliance issues, the need for talent cultivation, and organizational transformation. Looking ahead, the future of investment banking is expected to witness increased AI adoption, leading to higher levels of intelligence, more sophisticated learning techniques, and collaborative innovation across the industry. This trajectory promises to reshape the landscape of investment banking, creating new opportunities for growth and advancement.

Personalized Federated Graph Learning on Non-IID Electronic Health Records.

Understanding the latent disease patterns embedded in electronic health records (EHRs) is crucial for making precise and proactive healthcare decisions. Federated graph learning-based methods are commonly employed to extract complex disease patterns from the distributed EHRs without sharing the client-side raw data. However, the intrinsic characteristics of the distributed EHRs are typically non-independent and identically distributed (Non-IID), significantly bringing challenges related to data imbalance and leading to a notable decrease in the effectiveness of making healthcare decisions derived from the global model. To address these challenges, we introduce a novel personalized federated learning framework named PEARL, which is designed for disease prediction on Non-IID EHRs. Specifically, PEARL incorporates disease diagnostic code attention and admission record attention to extract patient embeddings from all EHRs. Then, PEARL integrates self-supervised learning into a federated learning framework to train a global model for hierarchical disease prediction. To improve the performance of the client model, we further introduce a fine-tuning scheme to personalize the global model using local EHRs. During the global model updating process, a differential privacy (DP) scheme is implemented, providing a high-level privacy guarantee. Extensive experiments conducted on the real-world MIMIC-III dataset validate the effectiveness of PEARL, demonstrating competitive results when compared with baselines.

Rainforest: A three-stage distribution adaptation framework for unsupervised time series domain adaptation

Solving the unsupervised domain adaptation (UDA) task in time series is of great significance for practical applications, such as human activity recognition and machine fault diagnosis. Compared to UDA for computer vision, UDA in time series is more challenging due to the dynamics of time series data and the complex dependencies among different time steps. Existing UDA methods for time series fail to adequately capture the temporal dependencies, limiting their ability to learn domain-invariant temporal patterns. Furthermore, most UDA methods only focus on distribution adaptation on the backbone network without considering how the classifier adapts to the data distribution of the target domain. In this paper, we propose Rainforest, a three-stage UDA framework for time series. We first pre-train the backbone network through a self-supervised method called bidirectional autoregression, so that the model can comprehensively learn the temporal dependencies in time series. Next, we propose a novel meta-learning-based distribution adaptation method to achieve the joint alignment of the global and local distributions while encouraging the model to adaptively reduce the temporal dynamic differences among different domains. Finally, we design a pseudo-label-guided fine-tuning strategy to help the classifier estimate the data distribution of the target domain more accurately. Extensive experiments on four real-world time series datasets show that our Rainforest outperforms state-of-the-art methods, with an average improvement of 2.19% in accuracy and 2.41% in MF1-score.

Multimodal self-supervised learning for remote sensing data land cover classification

Deep learning has revolutionized the remote sensing image processing techniques over the past few years. Nevertheless, annotating high-quality samples is difficult and time-consuming, which limits the performance of deep neural networks because of insufficient supervision information. Aiming to solve this contradiction, we investigate the multimodal self-supervised learning (MultiSSL) paradigm for pre-training and classification of remote sensing image. Specifically, the proposed self-supervised feature learning model consists of asymmetric encoder–decoder structure, in which deep unified encoder learns high-level key information characterizing multimodal remote sensing data and task-specific lightweight decoders are developed to reconstruct original data. To further enhance feature extraction capability, the cross-attention layers are utilized to exchange information contained in heterogeneous characteristics, thus learning more complementary information from multimodal remote sensing data. In fine-tuning stage, the pre-trained encoder and cross-attention layer serve as feature extractor, and leaned characteristics are combined with corresponding spectral information for land cover classification through a lightweight classifier. The self-supervised pre-training model can learn high-level key features from unlabeled samples, thereby utilizing the feature extraction capability of deep neural networks while reducing their dependence on annotated samples. Compared with existing classification paradigms, the proposed multimodal self-supervised pre-training and fine-tuning scheme achieves superior performance for remote sensing image land cover classification.

Cross-modal Transfer Learning Based on an Improved CycleGAN Model for Accurate Kidney Segmentation in Ultrasound Images

ObjectiveDeep-learning algorithms have been widely applied in the field of automatic kidney ultrasound (US) image segmentation. However, obtaining a large number of accurate kidney labels clinically is very difficult and time-consuming. To solve this problem, we have proposed an efficient cross-modal transfer learning method to improve the performance of the segmentation network on a limited labeled kidney US dataset. MethodsWe aim to implement an improved image-to-image translation network called Seg-CycleGAN to generate accurate annotated kidney US data from labeled abdomen computed tomography images. The Seg-CycleGAN framework primarily consists of two structures: (i) a standard CycleGAN network to visually simulate kidney US from a publicly available labeled abdomen computed tomography dataset; (ii) and a segmentation network to ensure accurate kidney anatomical structures in US images. Based on the large number of simulated kidney US images and small number of real annotated kidney US images, we then aimed to employ a fine-tuning strategy to obtain better segmentation results. ResultsTo validate the effectiveness of the proposed method, we tested this method on both normal and abnormal kidney US images. The experimental results showed that the proposed method achieved a segmentation accuracy of 0.8548 in dice similarity coefficient on all testing datasets and 0.7622 on the abnormal testing dataset. ConclusionsCompared with existing data augmentation and transfer learning methods, the proposed method improved the accuracy and generalization of the kidney US image segmentation network on a limited number of training datasets. It therefore has the potential to significantly reduce annotation costs in clinical settings.

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM’s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM’s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM’s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.

MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples

Although In-Context Learning (ICL) brings remarkable performance gains to Large Language Models (LLMs), the improvements remain lower than fine-tuning on downstream tasks. This paper introduces Multi-Modal In-Context Tuning (MMICT), a novel multi-modal fine-tuning paradigm that boosts multi-modal fine-tuning by fully leveraging the promising ICL capability of multi-modal LLMs (MM-LLMs). We propose the Multi-Modal Hub (M-Hub), a unified module that captures various multi-modal features according to different inputs and objectives. Based on M-Hub, MMICT enables MM-LLMs to learn from in-context visual-guided textual features and subsequently generate outputs conditioned on the textual-guided visual features. Moreover, leveraging the flexibility of M-Hub, we design a variety of in-context demonstrations. Extensive experiments on a diverse range of downstream multi-modal tasks demonstrate that MMICT significantly outperforms traditional fine-tuning strategy and the vanilla ICT method that directly takes the concatenation of all information from different modalities as input. Our implementation is available at: https://github.com/KDEGroup/MMICT.

Ventricular segmentation algorithm for echocardiography based on transfer learning and GAN

With the swift progression of computer technology, utilizing deep learning for left ventricular image segmentation in echocardiography is of great significance for automated cardiac function assessment. In this study, a left ventricular segmentation algorithm for echocardiography based on transfer learning and generative adversarial networks (GAN) was proposed. A hierarchical fine-tuning strategy was employed to adaptively adjust network parameters, thereby improving segmentation accuracy. This strategy was based on combining multi-scale GAN and attention mechanisms to enhance segmentation performance, and incorporating the idea of transfer learning into the segmentation network. The algorithm demonstrated efficacy in heart image segmentation tasks, achieving high levels of classification accuracy and producing satisfactory segmentation outcomes. In terms of image segmentation of dual and four chamber images, this method achieved high Dice similarity coefficients and average intersection values. Furthermore, this method demonstrated excellent performance in image segmentation experiments on clinical data, with low average absolute errors in the calculated end diastolic volume, end systolic volume, and ejection fraction. The experimental results demonstrate that the algorithm exhibits satisfactory performance in left ventricular segmentation in echocardiography and is suitable for use in medical clinical data. The study presents a novel approach to ventricular segmentation in echocardiography, which enhances the precision and efficiency of automated cardiac function assessment and has significant clinical applicability.

Automatic detection and visualization of temporomandibular joint effusion with deep neural network

This study investigated the usefulness of deep learning-based automatic detection of temporomandibular joint (TMJ) effusion using magnetic resonance imaging (MRI) in patients with temporomandibular disorder and whether the diagnostic accuracy of the model improved when patients’ clinical information was provided in addition to MRI images. The sagittal MR images of 2948 TMJs were collected from 1017 women and 457 men (mean age 37.19 ± 18.64 years). The TMJ effusion diagnostic performances of three convolutional neural networks (scratch, fine-tuning, and freeze schemes) were compared with those of human experts based on areas under the curve (AUCs) and diagnosis accuracies. The fine-tuning model with proton density (PD) images showed acceptable prediction performance (AUC = 0.7895), and the from-scratch (0.6193) and freeze (0.6149) models showed lower performances (p < 0.05). The fine-tuning model had excellent specificity compared to the human experts (87.25% vs. 58.17%). However, the human experts were superior in sensitivity (80.00% vs. 57.43%) (all p < 0.001). In gradient-weighted class activation mapping (Grad-CAM) visualizations, the fine-tuning scheme focused more on effusion than on other structures of the TMJ, and the sparsity was higher than that of the from-scratch scheme (82.40% vs. 49.83%, p < 0.05). The Grad-CAM visualizations agreed with the model learned through important features in the TMJ area, particularly around the articular disc. Two fine-tuning models on PD and T2-weighted images showed that the diagnostic performance did not improve compared with using PD alone (p < 0.05). Diverse AUCs were observed across each group when the patients were divided according to age (0.7083–0.8375) and sex (male:0.7576, female:0.7083). The prediction accuracy of the ensemble model was higher than that of the human experts when all the data were used (74.21% vs. 67.71%, p < 0.05). A deep neural network (DNN) was developed to process multimodal data, including MRI and patient clinical data. Analysis of four age groups with the DNN model showed that the 41–60 age group had the best performance (AUC = 0.8258). The fine-tuning model and DNN were optimal for judging TMJ effusion and may be used to prevent true negative cases and aid in human diagnostic performance. Assistive automated diagnostic methods have the potential to increase clinicians’ diagnostic accuracy.

Gastric Cancer Image Classification: A Comparative Analysis and Feature Fusion Strategies.

Gastric cancer is the fifth most common and fourth deadliest cancer worldwide, with a bleak 5-year survival rate of about 20%. Despite significant research into its pathobiology, prognostic predictability remains insufficient due to pathologists' heavy workloads and the potential for diagnostic errors. Consequently, there is a pressing need for automated and precise histopathological diagnostic tools. This study leverages Machine Learning and Deep Learning techniques to classify histopathological images into healthy and cancerous categories. By utilizing both handcrafted and deep features and shallow learning classifiers on the GasHisSDB dataset, we conduct a comparative analysis to identify the most effective combinations of features and classifiers for differentiating normal from abnormal histopathological images without employing fine-tuning strategies. Our methodology achieves an accuracy of 95% with the SVM classifier, underscoring the effectiveness of feature fusion strategies. Additionally, cross-magnification experiments produced promising results with accuracies close to 80% and 90% when testing the models on unseen testing images with different resolutions.

1D -FNO - Residual convolutional neural network autoregressive prediction model for deep water wave train evolution based on high-order spectral (HOS) data

Accurate prediction of deep-water wave dynamics is essential for ocean engineering, meteorology, and maritime safety. Traditional physical models, though effective, face significant computational and time challenges with complex nonlinear dynamics. This study introduces an autoregressive prediction model using a one-dimensional adaptive Fourier Neural Operator-Residual Convolutional Neural Network (1D-FNO-ResNet) to enhance prediction accuracy. We developed a high-precision single-step prediction model capable of capturing wave characteristics such as height and fluid particle velocity. A fine-tuning training strategy based on multi-step prediction errors was proposed for autoregressive evolution. Performance comparisons between the ResNet and FNO-ResNet models were made under various fine-tuning steps. A correction model was also introduced to control error accumulation, improving autoregressive prediction steps. The model's generalization performance was validated with modulated and random wave trains under different conditions, optimizing predictive stability and accuracy. This approach demonstrates efficient 1D deep-water wave evolution prediction, offering a novel deep learning solution for wave dynamics prediction in ocean engineering, supporting wave energy conversion device design and optimization.

STELLM: Spatio-temporal enhanced pre-trained large language model for wind speed forecasting

As a renewable and clean energy source, wind energy has caught worldwide attention. To ensure the reliability and stability of wind energy production, wind speed forecasting is of great importance. Although existing forecasting models have shown promising results, they remain relatively small in scale and are incapable of learning comprehensive sequential representations from large-scale data to reason intricate spatio-temporal patterns obscured in entangled wind series. Recently, pre-trained large models have demonstrated impressive performance in natural language processing (NLP) and computer vision (CV). However, large models are rarely applied to wind speed forecasting due to the lack of large-scale wind data for training. To resolve this issue, we propose Spatio-Temporal Enhanced Pre-Trained Large Language Model, namely STELLM, innovatively leveraging the powerful reasoning ability of Large Language Models (LLMs) for accurate wind speed forecasting. Specifically, we first perform the series decomposition to separate wind series data into seasonal and trend components, capturing the distinct properties from disentangled patterns. Then, to empower LLMs with the spatio-temporal modeling capability, we introduce the spatial and temporal prompts, which fuse the geographic distribution information and short-term temporal patterns of wind turbines to enrich the input wind series. Moreover, we leverage an autoregressive fine-tuning strategy to align LLMs with wind series data and learn patch-level representations. Afterwards these representations are fed into a localized spatial module to capture the spatial dependencies between wind turbines. Extensive experiments on four public wind datasets demonstrate the performance superiority of STELLM.

Convolutional neural network transformer (CNNT) for fluorescence microscopy image denoising with improved generalization and fast adaptation

Deep neural networks can improve the quality of fluorescence microscopy images. Previous methods, based on Convolutional Neural Networks (CNNs), require time-consuming training of individual models for each experiment, impairing their applicability and generalization. In this study, we propose a novel imaging-transformer based model, Convolutional Neural Network Transformer (CNNT), that outperforms CNN based networks for image denoising. We train a general CNNT based backbone model from pairwise high-low Signal-to-Noise Ratio (SNR) image volumes, gathered from a single type of fluorescence microscope, an instant Structured Illumination Microscope. Fast adaptation to new microscopes is achieved by fine-tuning the backbone on only 5–10 image volume pairs per new experiment. Results show that the CNNT backbone and fine-tuning scheme significantly reduces training time and improves image quality, outperforming models trained using only CNNs such as 3D-RCAN and Noise2Fast. We show three examples of efficacy of this approach in wide-field, two-photon, and confocal fluorescence microscopy.

Embracing Large Natural Data: Enhancing Medical Image Analysis via Cross-Domain Fine-Tuning.

With the rapid advancements of Big Data and computer vision, many large-scale natural visual datasets are proposed, such as ImageNet-21K, LAION-400M, and LAION-2B. These large-scale datasets significantly improve the robustness and accuracy of models in the natural vision domain. However, the field of medical images continues to face limitations due to relatively small-scale datasets. In this article, we propose a novel method to enhance medical image analysis across domains by leveraging pre-trained models on large natural datasets. Specifically, a Cross-Domain Transfer Module (CDTM) is proposed to transfer natural vision domain features to the medical image domain, facilitating efficient fine-tuning of models pre-trained on large datasets. In addition, we design a Staged Fine-Tuning (SFT) strategy in conjunction with CDTM to further improve the model performance. Experimental results demonstrate that our method achieves state-of-the-art performance on multiple medical image datasets through efficient fine-tuning of models pre-trained on large natural datasets.