Paired Training Research Articles

SCAM-GAN: Generating brain MR images from CT scan data based on CycleGAN combined with attention module

The diagnosis of epilepsy often depends heavily on Magnetic Resonance Imaging (MRI). Unfortunately, the utilization of MRI is constrained, due to its expensive price and lengthy operating times. More significantly, certain people with claustrophobia or cardiac pacemakers are not candidates for MRI owing to the risk of harm. Computed tomography (CT) images, in comparison, are considerably faster, cheaper, and free from the same restrictions. As opposed to conventional medical imaging synthetic techniques, which rely on abundant unpaired data or a minimum number of paired data, the proposed method in this study utilizes unpaired training data to predict an MR picture from a CT scan. This method overcomes tight registration problem in paired training and reduces the challenge of context mismatch in unpaired training. It is possible to convert 2D brain MR pictures from CT scans using the Spatial & Channel Attention Mechanism-Generative Adversarial Network (SCAM-GAN) that includes cycle-consistent and adversarial losses. The superiority of the proposed strategy was established by quantitative comparisons versus separate training approaches which are paired and unpaired.

Real-time monitoring of CO2 transport pipelines using deep learning

Real-time pipeline monitoring is important for the safe transportation of captured CO2. A dynamic modeling method, which is one of pipeline monitoring methods, can provide reliable diagnostic results for various anomalies. In the dynamic modeling method, anomalies are detected by comparing the predictions and observations of pipeline monitoring variables. However, the licensing costs associated with the use of pipeline flow simulators that provides the predictions are high. In this study, we developed a real-time deep-learning-based pipeline monitoring method that can save the licensing cost of the pipeline flow simulators. The predictions are obtained using deep-learning models where the pipeline flow simulator is required only in the training step. Two improvements were made to enhance both the prediction and anomaly detection accuracies. First, the prediction accuracy for the monitoring variables can be improved by considering a delay time interval between inlet and outlet points in pairing training input and output data. Second, the anomaly detection accuracy can be also improved by conditionally choosing observations based on the normal operation ranges of the observations. As part of a field demonstration, the proposed deep-learning-based pipeline monitoring method was applied to the monitoring of a CO2 transport pipeline located in the East Sea gas field. The results showed the anomaly detection accuracy of the proposed method was improved by more than 25 %.

Cross-Domain Unpaired Learning for Low-Dose CT Imaging.

Supervised deep-learning techniques with paired training datasets have been widely studied for low-dose computed tomography (LDCT) imaging with excellent performance. However, the paired training datasets are usually difficult to obtain in clinical routine, which restricts the wide adoption of supervised deep-learning techniques in clinical practices. To address this issue, a general idea is to construct a pseudo paired training dataset based on the widely available unpaired data, after which, supervised deep-learning techniques can be adopted for improving the LDCT imaging performance by training on the pseudo paired training dataset. However, due to the complexity of noise properties in CT imaging, the LDCT data are difficult to generate in order to construct the pseudo paired training dataset. In this paper, we propose a simple yet effective cross-domain unpaired learning framework for pseudo LDCT data generation and LDCT image reconstruction, which is denoted as CrossDuL. Specifically, a dedicated pseudo LDCT sinogram generative module is constructed based on a data-dependent noise model in the sinogram domain, and then instead of in the sinogram domain, a pseudo paired dataset is constructed in the image domain to train an LDCT image restoration module. To validate the effectiveness of the proposed framework, clinical datasets are adopted. Experimental results demonstrate that the CrossDuL framework can obtain promising LDCT imaging performance in both quantitative and qualitative measurements.

GAN inversion-based semi-supervised learning for medical image segmentation

The crux of medical image segmentation stems from learning pixel-wise semantic consistency from a mount of labeled samples through exhaustive annotation. Most existing methods rely heavily on labeled image pairs and are optimized in a fully supervised manner, often resulting in unsatisfactory results due to limited paired training samples. In this paper, we propose a GAN inversion-based semi-supervised learning framework InvSSL for medical image segmentation, which generates corresponding variants from labeled images by GAN inversion. Specifically, it first trains a StyleGAN as an inversion generator which is able to generate samples with the same distribution as training samples, and then obtains variant samples from a training sample by performing GAN inversion to strengthen the segmentation performance in a semi-supervised learning manner. In particular, inversion generator inverts a training sample into the latent space and adds interference to the latent codes to evolve variant samples. To model high-level consistency between pixels and decouple semantic components in the latent space, our InvSSL designs a multi-level dense contrastive learning mechanism. Extensive experiments on lung segmentation and skin lesion segmentation demonstrate that our InvSSL outperforms state-of-the-art methods for medical image segmentation. Our code will be released at https://github.com/funkdub/InvSSL.

MARGANVAC: metal artifact reduction method based on generative adversarial network with variable constraints

Objective. Metal artifact reduction (MAR) has been a key issue in CT imaging. Recently, MAR methods based on deep learning have achieved promising results. However, when deploying deep learning-based MAR in real-world clinical scenarios, two prominent challenges arise. One limitation is the lack of paired training data in real applications, which limits the practicality of supervised methods. Another limitation is that image-domain methods suitable for more application scenarios are inadequate in performance while end-to-end approaches with better performance are only applicable to fan-beam CT due to large memory consumption. Approach. We propose a novel image-domain MAR method based on the generative adversarial network with variable constraints (MARGANVAC) to improve MAR performance. The proposed variable constraint is a kind of time-varying cost function that can relax the fidelity constraint at the beginning and gradually strengthen the fidelity constraint as the training progresses. To better deploy our image-domain supervised method into practical scenarios, we develop a transfer method to mimic the real metal artifacts by first extracting the real metal traces and then adding them to artifact-free images to generate paired training data. Main results. The effectiveness of the proposed method is validated in simulated fan-beam experiments and real cone-beam experiments. All quantitative and qualitative results demonstrate that the proposed method achieves superior performance compared with the competing methods. Significance. The MARGANVAC model proposed in this paper is an image-domain model that can be conveniently applied to various scenarios such as fan beam and cone beam CT. At the same time, its performance is on par with the cutting-edge dual-domain MAR approaches. In addition, the metal artifact transfer method proposed in this paper can easily generate paired data with real artifact features, which can be better used for model training in real scenarios.

Domain-Scalable Unpaired Image Translation via Latent Space Anchoring.

Unpaired image-to-image translation (UNIT) aims to map images between two visual domains without paired training data. However, given a UNIT model trained on certain domains, it is difficult for current methods to incorporate new domains because they often need to train the full model on both existing and new domains. To address this problem, we propose a new domain-scalable UNIT method, termed as latent space anchoring, which can be efficiently extended to new visual domains and does not need to fine-tune encoders and decoders of existing domains. Our method anchors images of different domains to the same latent space of frozen GANs by learning lightweight encoder and regressor models to reconstruct single-domain images. In the inference phase, the learned encoders and decoders of different domains can be arbitrarily combined to translate images between any two domains without fine-tuning. Experiments on various datasets show that the proposed method achieves superior performance on both standard and domain-scalable UNIT tasks in comparison with the state-of-the-art methods.

Blind deep-learning based preprocessing method for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a technique for tackling the trade-off between the resolution and the imaging field of view by combining the techniques from aperture synthesis and phase retrieval to estimate the complex object from a series of low-resolution intensity images captured under angle-varied illumination. The captured images are commonly corrupted by multiple noise, leading to the degradation of the reconstructed image quality. Typically speaking, the noise model and noise level of the experimental images are unknown, and the traditional image denoising methods have limited effect. In this paper we model the FPM forward imaging process corrupted by noise and divide the noise in the captured images into two parts: the signal-dependent part and the signal-independent part. Based on the noise model we propose a novel blind deep-learning based Fourier ptychographic microscopy preprocessing method, termed BDFP, for removing these two components of noise. First, from a portion of the captured low-resolution images, a set of blocks corresponding to the smooth area of the object are extracted to model signal-independent noise. Second, under the assumption that the signal-dependent noise follows a Poisson distribution, we add Poisson noise and signal-independent noise blocks to clean images to form a paired training dataset, which is then used for training a deep convolutional neural network (CNN) model to reduce both signal-dependent noise and signal-independent noise. The proposed blind preprocessing method, combining with typical FPM reconstruction algorithms, is tested on simulated data and experimental images. Experimental results show that our preprocessing method can significantly reduce the noise in the captured images and bring about effective improvements in reconstructed image quality.

Self-supervised depth super-resolution with contrastive multiview pre-training

Many low-level vision tasks, including guided depth super-resolution (GDSR), struggle with the issue of insufficient paired training data. Self-supervised learning is a promising solution, but it remains challenging to upsample depth maps without the explicit supervision of high-resolution target images. To alleviate this problem, we propose a self-supervised depth super-resolution method with contrastive multiview pre-training. Unlike existing contrastive learning methods for classification or segmentation tasks, our strategy can be applied to regression tasks even when trained on a small-scale dataset and can reduce information redundancy by extracting unique features from the guide. Furthermore, we propose a novel mutual modulation scheme that can effectively compute the local spatial correlation between cross-modal features. Exhaustive experiments demonstrate that our method attains superior performance with respect to state-of-the-art GDSR methods and exhibits good generalization to other modalities.

Transforming traditional physiotherapy hands-on skills teaching into video-based learning

BackgroundPandemic-induced restrictions forced curriculum transformation from on-site education to virtual learning options. This report describes this transition, the challenge of creating technology-enhanced learning for hands-on psychomotor skills teaching in physiotherapy, and students’ evaluations of the new technology-enhanced learning approach in Complex Decongestive Physiotherapy.MethodsOn-site theoretical background lectures were replaced with e-learning sessions. Faculty hands-on skills demonstrations for the entire class were replaced with video-recorded demonstrations. Videos included verbal and written instructions and were complemented with checklists guiding the students, training in pairs, through their learning tasks. A cross-sectional observational survey for teaching quality evaluated this new technology-enhanced learning approach and assessed students’ preference for traditional or video-based hands-on skills learning.ResultsSurvey return rate was > 50% (46 participating students). Teaching quality was rated between 1.5 ± 0.5 and 1.8 ± 0.4 (Likert scale from − 2 to + 2). Most students (66.7%) preferred the new approach. They appreciated for example that videos were available all the time, enabling self-paced learning, providing an equally good view on skills demonstrations, and the convenience to be able to rewind, re-view, and use speed adjustment options.ConclusionsStudents preferred the new video-based learning of skills for Complex Decongestive Physiotherapy. Because in-class live skills demonstrations were omitted, faculty had more time to provide individual feedback and answer questions. The shift from teacher- to student-centered learning enabled students to control their own learning pace. The innovative program was maintained after pandemic-induced restrictions were lifted. The success of this approach should be tested in other physiotherapy settings and different educational institutions.

Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features.

Decoding human visual neural representations is a challenging task with great scientific significance in revealing vision-processing mechanisms and developing brain-like intelligent machines. Most existing methods are difficult to generalize to novel categories that have no corresponding neural data for training. The two main reasons are 1) the under-exploitation of the multimodal semantic knowledge underlying the neural data and 2) the small number of paired (stimuli-responses) training data. To overcome these limitations, this paper presents a generic neural decoding method called BraVL that uses multimodal learning of brain-visual-linguistic features. We focus on modeling the relationships between brain, visual and linguistic features via multimodal deep generative models. Specifically, we leverage the mixture-of-product-of-experts formulation to infer a latent code that enables a coherent joint generation of all three modalities. To learn a more consistent joint representation and improve the data efficiency in the case of limited brain activity data, we exploit both intra- and inter-modality mutual information maximization regularization terms. In particular, our BraVL model can be trained under various semi-supervised scenarios to incorporate the visual and textual features obtained from the extra categories. Finally, we construct three trimodal matching datasets, and the extensive experiments lead to some interesting conclusions and cognitive insights: 1) decoding novel visual categories from human brain activity is practically possible with good accuracy; 2) decoding models using the combination of visual and linguistic features perform much better than those using either of them alone; 3) visual perception may be accompanied by linguistic influences to represent the semantics of visual stimuli.

Distinguishing and Matching-Aware Unsupervised Point Cloud Completion

Real-scanned point clouds are often incomplete due to occlusion, light reflection and limitations of sensor resolution, which impedes the related progress of downstream tasks, e.g., shape classification and object detection. Although there has been impressive research progress on the point cloud completion topic, they rely on the premise of extensive paired training data. However, collecting complete point clouds in some specified scenarios is labor-intensive and even impractical. To mitigate this problem, we propose DMNet, a distinguishing and matching-aware unsupervised point cloud completion network. Our work belongs to the group of unsupervised completion methods but goes beyond previous studies. Firstly, we propose a distinguishing-aware feature extractor to learn discriminable semantic information for different instances, simultaneously enhancing the robust invariant representation under noise disturbances. Secondly, we design a hierarchy-aware hyperbolic decoder to recover the complete geometry of point clouds, which not only can capture the implicit hierarchical relationships in data but also has an explicit extended nature. Finally, we develop a matching-aware refiner to eliminate noise points via aligning the topology structure of the input and predicted partial point clouds. Extensive experiments on MVP, Completion3D and KITTI datasets prove the effectiveness of our method, which performs favorably over state-of-the-art methods both quantitatively and qualitatively.

Electrical impedance tomography with deep Calderón method

Electrical impedance tomography (EIT) is a noninvasive medical imaging modality utilizing the current-density/voltage data measured on the surface of the subject. Calderón's method is a relatively recent EIT imaging algorithm that is non-iterative, fast, and capable of reconstructing complex-valued electric impedances. However, due to the regularization via low-pass filtering and linearization, the reconstructed images suffer from severe blurring and under-estimation of the exact conductivity values. In this work, we develop an enhanced version of Calderón's method, using deep convolution neural networks (i.e., U-net) as an effective targeted post-processing step, and term the resulting method by deep Calderón's method. Specifically, we learn a U-net to postprocess the EIT images generated by Calderón's method so as to have better resolutions and more accurate estimates of conductivity values. We simulate chest configurations with which we generate the current-density/voltage boundary measurements and the corresponding reconstructed images by Calderón's method. With the paired training data, we learn the deep neural network and evaluate its performance on real tank measurement data. The experimental results indicate that the proposed approach indeed provides a fast and direct (complex-valued) impedance tomography imaging technique, and substantially improves the capability of the standard Calderón's method.

Survey on Sketch-to-photo Translation

Sketch-based understanding is involved in human communication and cognitive development, making it essential in visual perception. A specific task in this domain is sketch-to-photo translation, where a model produces realistic images from simple drawings. To this end, large paired training datasets are commonly required, which is impractical in real applications. Thus, this work studies conditional generative models for sketch-to-photo translation, overcoming the lack of training datasets by a self-supervised approach that produces sketch-photo pairs from a target catalog. Our study shows the benefit of cycle-consistency loss and UNet architectures that, together with the proposed dataset generation, improve performance in real applications like eCommerce. Our results also reveal the weakness of conditional DDPMs for generating images resembling the input sketch, even though they achieve a high FID score.

A Zero-Shot Low Light Image Enhancement Method Integrating Gating Mechanism.

Photographs taken under harsh ambient lighting can suffer from a number of image quality degradation phenomena due to insufficient exposure. These include reduced brightness, loss of transfer information, noise, and color distortion. In order to solve the above problems, researchers have proposed many deep learning-based methods to improve the illumination of images. However, most existing methods face the problem of difficulty in obtaining paired training data. In this context, a zero-reference image enhancement network for low light conditions is proposed in this paper. First, the improved Encoder-Decoder structure is used to extract image features to generate feature maps and generate the parameter matrix of the enhancement factor from the feature maps. Then, the enhancement curve is constructed using the parameter matrix. The image is iteratively enhanced using the enhancement curve and the enhancement parameters. Second, the unsupervised algorithm needs to design an image non-reference loss function in training. Four non-reference loss functions are introduced to train the parameter estimation network. Experiments on several datasets with only low-light images show that the proposed network has improved performance compared with other methods in NIQE, PIQE, and BRISQUE non-reference evaluation index, and ablation experiments are carried out for key parts, which proves the effectiveness of this method. At the same time, the performance data of the method on PC devices and mobile devices are investigated, and the experimental analysis is given. This proves the feasibility of the method in this paper in practical application.

Paradoxical effects of feeding status on food consumption and learning performance in zebrafish (Danio rerio)

Associative learning is often studied using food reward as the unconditioned stimulus (US). With warm-blooded species, to get the subject more motivated the solution has been to feed less, making the subject hungrier. Here we show the opposite with zebrafish. We randomly assigned zebrafish to two groups: a once-a-day-fed and a five-times-a-day-fed group, with the same amount of food fed per occasion for fish of both groups, a feeding regimen that lasted for three months. Subsequently, we trained fish by pairing food (US) with a red cue card (the conditioned stimulus, CS), which were placed together in one arm of a plus-maze across eight training sessions. We also ran unpaired training, in which the CS and US were presented in different arms. We found the previously once-a-day-fed zebrafish to consume less food throughout habituation and training sessions compared to the previously five-times-a-day-fed ones. Furthermore, five-times-a-day-fed fish in the paired group swam significantly closer to the CS during a post-training probe trial compared to the five-times-a-day-fed unpaired fish, a paired training effect that was absent in once-a-day-fed fish. Groups did not differ in health or general activity. In sum, elevated chronic feeding improved food consumption and enhanced learning and memory performance without affecting activity levels in adult zebrafish.

VAEs with structured image covariance applied to compressed sensing MRI

Objective.This paper investigates how generative models, trained on ground-truth images, can be used as priors for inverse problems, penalizing reconstructions far from images the generator can produce. The aim is that learned regularization will provide complex data-driven priors to inverse problems while still retaining the control and insight of a variational regularization method. Moreover, unsupervised learning, without paired training data, allows the learned regularizer to remain flexible to changes in the forward problem such as noise level, sampling pattern or coil sensitivities in MRI. Approach. We utilize variational autoencoders that generate not only an image but also a covariance uncertainty matrix for each image. The covariance can model changing uncertainty dependencies caused by structure in the image, such as edges or objects, and provides a new distance metric from the manifold of learned images. Main results. We evaluate these novel generative regularizers on retrospectively sub-sampled real-valued MRI measurements from the fastMRI dataset. We compare our proposed learned regularization against other unlearned regularization approaches and unsupervised and supervised deep learning methods. Significance. Our results show that the proposed method is competitive with other state-of-the-art methods and behaves consistently with changing sampling patterns and noise levels.

Unsupervised Low-Light Image Enhancement via Virtual Diffraction Information in Frequency Domain

With the advent of deep learning, significant progress has been made in low-light image enhancement methods. However, deep learning requires enormous paired training data, which is challenging to capture in real-world scenarios. To address this limitation, this paper presents a novel unsupervised low-light image enhancement method, which first introduces the frequency-domain features of images in low-light image enhancement tasks. Our work is inspired by imagining a digital image as a spatially varying metaphoric “field of light”, then subjecting the influence of physical processes such as diffraction and coherent detection back onto the original image space via a frequency-domain to spatial-domain transformation (inverse Fourier transform). However, the mathematical model created by this physical process still requires complex manual tuning of the parameters for different scene conditions to achieve the best adjustment. Therefore, we proposed a dual-branch convolution network to estimate pixel-wise and high-order spatial interactions for dynamic range adjustment of the frequency feature of the given low-light image. Guided by the frequency feature from the “field of light” and parameter estimation networks, our method enables dynamic enhancement of low-light images. Extensive experiments have shown that our method performs well compared to state-of-the-art unsupervised methods, and its performance approximates the level of the state-of-the-art supervised methods qualitatively and quantitatively. At the same time, the light network structure design allows the proposed method to have extremely fast inference speed (near 150 FPS on an NVIDIA 3090 Ti GPU for an image of size 600×400×3). Furthermore, the potential benefits of our method to object detection in the dark are discussed.

Improving face morph detection with the pairs training effect

AbstractIt is becoming increasingly common for face morphs (weighted combinations of two people's photographs) to be submitted for inclusion in an official document, such as a passport. These images may sufficiently resemble both individuals that they can be used by either person in a ‘fraudulently obtained genuine’ document. Problematically, people are poor at detecting face morphs and there is limited evidence that this can be improved. Here, we tested whether the ‘pairs training effect’ (working in pairs, which we know improves unfamiliar face matching) can improve face morph detection. We found morph detection was more accurate when working in a pair. Further, the lower performer in the pair maintained this benefit when completing the task again individually. We conclude that the pairs training effect translates to face morph detection, and these findings have important implications for improving the detection of face morphs at the initial application stage.

THE EFFECT OF PAIR TRAINING VARIATION ON THE SEPAK SILA SKILL OF TAKRAW ATHLETES

The skill of Sepak sila is one of the basic techniques in the game of takraw. The focus of this research is extracurricular students at SMA Negeri 1 Palopo. The problem is the lack of variation in pair training. The purpose of this study is to determine whether or not there is an impact of variations in pair training on the sepak sila skills of the takraw extracurricular students at SMA Negeri 1 Palopo. The term "quantitative research using experimental methods" refers to a method for determining, under controlled conditions, how one treatment affects another treatment. In SMA Negeri 1 Palopo there are 25 male students who take part in takraw extracurricular activities. Purposive sampling was used as a sampling method, meaning that each student will be included in the research sample based on the special considerations of the researcher. SPSS application program version 23 is used for data analysis methods. With a t = count 22.733, this value is greater than t = table 2.059 or t = count 22.733 > t = table 2.059, and the result is sig. (2-tailed) of 0.000 <0.05. The results showed that there were variations in pair training for students' sepak sila skills in the extracurricular sepak takraw at SMA Negeri 1 Palopo. This shows that variations in pair training have a significant effect on the sepak sila skills of the takraw extracurricular students of SMA Negeri 1 Palopo. 059 or t = count 22.733 > t = table 2.059, and the result is sig. (2-tailed) of 0.000 <0.05. The results showed that there were variations in pair training for students' sepak sila in the extracurricular sepak takraw at SMA Negeri 1 Palopo.

Underwater Image Enhancement Based on Zero-Reference Deep Network

Due to underwater light absorption and scattering, underwater images usually suffer from severe color attenuation and contrast reduction. Most mainstream underwater image processing methods based on deep learning require a large amount of underwater paired training data, leading to a complex network structure, longer training time, and higher computational cost. To address this problem, a novel Zero-Reference Deep Network for Underwater Image Enhancement (Zero-UIE) is proposed in this paper, which transforms the enhancement of an underwater image into a specific parameter map estimation by using a deep network. The underwater curve model based on the classical haze image formation principle is specially designed to remove underwater color dispersion and cast. A lightweight deep network is designed to estimate the dynamic adjustment parameters of the underwater curve model, and then adjust the dynamic range of the given image pixels according to the model. A set of non-reference loss functions are designed according to the characteristics of underwater images, which can implicitly drive the network learning. In addition, adaptive color compensation can be optionally used as the pre-processing step to further improve the robustness and visual performance. The significant contribution of the proposed method is zero reference, i.e., it does not require any paired or unpaired reference data for training. Extensive experiments on various benchmarks demonstrate that the proposed method is superior to state-of-the-art methods subjectively and objectively, which is competitive and applicable to diverse underwater conditions. Most importantly, it is an innovative exploration of zero reference for underwater image enhancement.