Additional Loss Function Research Articles

Exploring charge sharing compensation using inter-pixel coincidence counters for photon counting detectors by deep-learning from local information

Photon counting detectors (PCDs) have well-acknowledged advantages in computed tomography (CT) imaging. However, charge sharing and other problems prevent PCDs from fully realizing the anticipated potential in diagnostic CT. PCDs with multi-energy inter-pixel coincidence counters (MEICC) have been proposed to provide particular information about charge sharing, thereby achieving lower Cramér-Rao Lower Bound (CRLB) than conventional PCDs when assessing its performance by estimating material thickness or virtual monochromatic attenuation integrals (VMAIs). This work explores charge sharing compensation using local spatial coincidence counter information for MEICC detectors through a deep-learning method.
Approach: By analyzing the impact of charge sharing on photon count detection, we designed our network with a focus on individual pixels. Employing MEICC data of patches centered on POIs as input, we utilized local information for effective charge sharing compensation. The output was VMAI at different energies to address real detector issues without knowledge of primary counts. To achieve data diversity, a fast and online data generation method was proposed to provide adequate training data. A new loss function was introduced to reduce bias for training with high-noise data. The proposed method was validated by Monte Carlo (MC) simulation data for MEICC detectors that were compared with conventional PCDs. 
Main-Results: For conventional data as a reference, networks trained on low-noise data yielded results with a minimal bias (about 0.7%) compared with > 3% for the polynomial fitting method. The results of networks trained on high-noise data exhibited a slightly increased bias (about 1.3%) but a significantly reduced standard deviation (STD) and normalized root mean square error (NRMSE). The simulation study of the MEICC detector demonstrated superior compared to the conventional detector across all the metrics. Specifically, for both networks trained on high-noise and low-noise data, their biases were reduced to about 1% and 0.6%, respectively. Meanwhile, the results from a MEICC detector were of about 10% lower noise than a conventional detector. Moreover, an ablation study showed that the additional loss function on bias was beneficial for training on high-noise data.
Significance: We demonstrated that a network-based method could utilize local information in PCDs effectively by patch-based learning to reduce the impact of charge sharing. MEICC detectors provide very valuable local spatial information by additional coincidence counters. Compared with MEICC detectors, conventional PCDs only have limited local spatial information for charge sharing compensation, resulting in higher bias and standard deviation in VMAI estimation with the same patch strategy.&#xD.

Deep learning model for fiber optic distributed acoustic sensing seismic data compression based on autoencoder and recurrent neural networks

Because of the extremely high sampling rate of fiber optic distributed acoustic sensing (DAS) equipment, the amount of DAS seismic data collected is enormous, which poses great challenges to the transmission and storage of DAS seismic data. Therefore, it is essential to study the compression methods of DAS seismic data. Existing data compression methods such as wavelet transform, cosine transform, and convolutional autoencoder (CAE) still have room for improvement in the compression performance and compression ratio (CR). Thus, we have proposed what we believe to be is a novel deep learning model called the recurrent autoencoder (RAE) for high-performance compression of DAS seismic data. Under different CRs, we have designed performance evaluation experiments for RAE models based on different RNN modules with different loss functions. When the CR of the RAE model is 8, the signal-to-noise ratio (SNR) of the reconstructed DAS seismic data reaches 40.60 dB, which is better than that of the CAE’s 11.58 dB. The ultimate CR was increased to 512 without reducing the compression quality, which is 4.12 times higher than the CAE model. The LSTM with a weighted loss function improves the SNR to 43.66 dB at a CR of 8, which is 3.06 dB higher than the LSTM with additive loss function. The results show that the RAE model proposed with a weighted loss function in this paper has excellent DAS seismic data compression performance and provides a high CR, which can be widely applied in large-scale DAS seismic data compression.

Multi-scale V-net architecture with deep feature CRF layers for brain extraction

BackgroundBrain extraction is a computational necessity for researchers using brain imaging data. However, the complex structure of the interfaces between the brain, meninges and human skull have not allowed a highly robust solution to emerge. While previous methods have used machine learning with structural and geometric priors in mind, with the development of Deep Learning (DL), there has been an increase in Neural Network based methods. Most proposed DL models focus on improving the training data despite the clear gap between groups in the amount and quality of accessible training data between.MethodsWe propose an architecture we call Efficient V-net with Additional Conditional Random Field Layers (EVAC+). EVAC+ has 3 major characteristics: (1) a smart augmentation strategy that improves training efficiency, (2) a unique way of using a Conditional Random Fields Recurrent Layer that improves accuracy and (3) an additional loss function that fine-tunes the segmentation output. We compare our model to state-of-the-art non-DL and DL methods.ResultsResults show that even with limited training resources, EVAC+ outperforms in most cases, achieving a high and stable Dice Coefficient and Jaccard Index along with a desirable lower Surface (Hausdorff) Distance. More importantly, our approach accurately segmented clinical and pediatric data, despite the fact that the training dataset only contains healthy adults.ConclusionsUltimately, our model provides a reliable way of accurately reducing segmentation errors in complex multi-tissue interfacing areas of the brain. We expect our method, which is publicly available and open-source, to be beneficial to a wide range of researchers.

Regularizing cross-attention learning for end-to-end speech translation with ASR and MT attention matrices

The cross-attention mechanism enables Transformer to capture correspondences between the input and output. However, in the domain of end-to-end (E2E) speech-to-text translation (ST), the learned cross-attention weights often struggle to accurately correspond with actual alignments, given the need to align speech and text across different modalities and languages. In this paper, we present a simple yet effective method called regularized cross-attention learning, for end-to-end speech translation in a multitask learning (MTL) framework. RCAL leverages the knowledge from auxiliary automatic speech recognition (ASR) and machine translation (MT) tasks to generate a teacher cross-attention matrix, serving as prior alignment knowledge to enhance cross-attention learning within the ST task. An additional loss function is introduced as part of the MTL framework to facilitate this process. We conducted experiments on the MuST-C benchmark dataset to evaluate the effectiveness of RCAL. The results demonstrate that the proposed approach yields significant improvements over the baseline, with an average enhancement of +0.8 BLEU across four translation directions in two experimental settings, outperforming state-of-the-art E2E and cascaded speech translation models. Further analysis and visualization reveal that the model with RCAL effectively learns high-quality alignment information from auxiliary ASR and ST tasks, thereby improving the ST alignment quality. Moreover, the experiments with different sizes of MT and ST data provide strong evidence supporting our model’s robustness in various scenarios.

An adaptive classification and updating method for unknown network traffic in open environments

In recent years, to address the classification problem of unknown traffic in open network environments, numerous deep learning (DL)-based classification techniques have been proposed. Among these techniques, the single threshold-based DL method has gained significant popularity. However, the classification boundary constructed by this method is inaccurate, thus leading to the misclassification of known and unknown traffic. In this paper, we propose an adaptive classification and updating method for accurate application-level classification of known and unknown traffic in open environments. We adaptively construct individualized and accurate decision boundaries for each class of known traffic, which avoids misclassification cases caused by a single threshold and achieves accurate separation of known and unknown traffic. To provide a more accurate feature representation for the construction of the decision boundary, we also introduce an additional angular loss function to guide the model in learning more accurate flow features. In addition, to cope with the constant emergence of new classes of traffic in real network environments, our proposed method adaptively updates the model to respond to changes in network traffic classes. The proposed method is evaluated on two datasets, and the experimental results show that our proposed method has excellent classification accuracy and outperforms the state-of-the-art unknown traffic classification methods.

Multimodal and Multitask Learning with Additive Angular Penalty Focus Loss for Speech Emotion Recognition

Speech emotion recognition has lots of applications such as human-computer interaction and health management. The current methods are challenged with the problems of fuzzy decision boundary and imbalance between difficult and easy samples in the training data. This paper first proposes an additive angle penalty focus loss function (APFL), which strictly refines the fuzzy decision boundary by introducing angle penalty factors to improve the compactness within the class and enlarge the distance between classes. It also assigns the larger loss to difficult samples to make the model pay more attention to them, as they are easily misclassified. Simultaneously, due to the lack of training samples, the framework of multimodal and multitask learning with APFL is further proposed, which extracts spectrogram features by deep neural network, text features by the pretrained language model, and audio features by the pretrained sound model. It uses the gender recognition as an auxiliary task. The experimental results verify the effectiveness of the proposed loss function and framework.

Segmentation-Free PVC for Cardiac SPECT Using a Densely-Connected Multi-Dimensional Dynamic Network.

In nuclear imaging, limited resolution causes partial volume effects (PVEs) that affect image sharpness and quantitative accuracy. Partial volume correction (PVC) methods incorporating high-resolution anatomical information from CT or MRI have been demonstrated to be effective. However, such anatomical-guided methods typically require tedious image registration and segmentation steps. Accurately segmented organ templates are also hard to obtain, particularly in cardiac SPECT imaging, due to the lack of hybrid SPECT/CT scanners with high-end CT and associated motion artifacts. Slight mis-registration/mis-segmentation would result in severe degradation in image quality after PVC. In this work, we develop a deep-learning-based method for fast cardiac SPECT PVC without anatomical information and associated organ segmentation. The proposed network involves a densely-connected multi-dimensional dynamic mechanism, allowing the convolutional kernels to be adapted based on the input images, even after the network is fully trained. Intramyocardial blood volume (IMBV) is introduced as an additional clinical-relevant loss function for network optimization. The proposed network demonstrated promising performance on 28 canine studies acquired on a GE Discovery NM/CT 570c dedicated cardiac SPECT scanner with a 64-slice CT using Technetium-99m-labeled red blood cells. This work showed that the proposed network with densely-connected dynamic mechanism produced superior results compared with the same network without such mechanism. Results also showed that the proposed network without anatomical information could produce images with statistically comparable IMBV measurements to the images generated by anatomical-guided PVC methods, which could be helpful in clinical translation.

Deep learning-based attenuation map generation with simultaneously reconstructed PET activity and attenuation and low-dose application

Objective. In PET/CT imaging, CT is used for positron emission tomography (PET) attenuation correction (AC). CT artifacts or misalignment between PET and CT can cause AC artifacts and quantification errors in PET. Simultaneous reconstruction (MLAA) of PET activity (λ-MLAA) and attenuation (μ-MLAA) maps was proposed to solve those issues using the time-of-flight PET raw data only. However, λ-MLAA still suffers from quantification error as compared to reconstruction using the gold-standard CT-based attenuation map (μ-CT). Recently, a deep learning (DL)-based framework was proposed to improve MLAA by predicting μ-DL from λ-MLAA and μ-MLAA using an image domain loss function (IM-loss). However, IM-loss does not directly measure the AC errors according to the PET attenuation physics. Our preliminary studies showed that an additional physics-based loss function can lead to more accurate PET AC. The main objective of this study is to optimize the attenuation map generation framework for clinical full-dose 18F-FDG studies. We also investigate the effectiveness of the optimized network on predicting attenuation maps for synthetic low-dose oncological PET studies. Approach. We optimized the proposed DL framework by applying different preprocessing steps and hyperparameter optimization, including patch size, weights of the loss terms and number of angles in the projection-domain loss term. The optimization was performed based on 100 skull-to-toe 18F-FDG PET/CT scans with minimal misalignment. The optimized framework was further evaluated on 85 clinical full-dose neck-to-thigh 18F-FDG cancer datasets as well as synthetic low-dose studies with only 10% of the full-dose raw data. Main results. Clinical evaluation of tumor quantification as well as physics-based figure-of-merit metric evaluation validated the promising performance of our proposed method. For both full-dose and low-dose studies, the proposed framework achieved <1% error in tumor standardized uptake value measures. Significance. It is of great clinical interest to achieve CT-less PET reconstruction, especially for low-dose PET studies.

Feature Map Regularized CycleGAN for Domain Transfer

CycleGAN domain transfer architectures use cycle consistency loss mechanisms to enforce the bijectivity of highly underconstrained domain transfer mapping. In this paper, in order to further constrain the mapping problem and reinforce the cycle consistency between two domains, we also introduce a novel regularization method based on the alignment of feature maps probability distributions. This type of optimization constraint, expressed via an additional loss function, allows for further reducing the size of the regions that are mapped from the source domain into the same image in the target domain, which leads to mapping closer to the bijective and thus better performance. By selecting feature maps of the network layers with the same depth d in the encoder of the direct generative adversarial networks (GANs), and the decoder of the inverse GAN, it is possible to describe their d-dimensional probability distributions and, through novel regularization term, enforce similarity between representations of the same image in both domains during the mapping cycle. We introduce several ground distances between Gaussian distributions of the corresponding feature maps used in the regularization. In the experiments conducted on several real datasets, we achieved better performance in the unsupervised image transfer task in comparison to the baseline CycleGAN, and obtained results that were much closer to the fully supervised pix2pix method for all used datasets. The PSNR measure of the proposed method was, on average, 4.7% closer to the results of the pix2pix method in comparison to the baseline CycleGAN over all datasets. This also held for SSIM, where the described percentage was 8.3% on average over all datasets.

One-Class Adversarial Fraud Detection Nets with Class Specific Representations

Most fraud detection algorithms identify the anomaly by learning from a small number of existing fraud samples, therefore they are often ineffective when deal with complex and unknown situations. This paper proposes a new one-class classification model called one-class adversarial fraud detection nets with class specific representations (CS-OCAN), which consists of modified autoencoders and Complementary generative adversarial networks (GAN). Firstly, the two-iteration framework is designed to make reasonable use of the reference data generated by Complementary GAN. Secondly, an additional loss function is added in the latent space of the autoencoder, which transforms the semi-supervised problem into a supervised problem that aims to maximize inter-class distances between two classes and minimize intra-class variances. We have conducted experiments on UMD Wikipedia dataset and Credit card fraud detection dataset. Experimental results show that our method CS-OCAN has significantly improved the detection accuracy and stability compared with the state-of-the-art one-class classification models.

Divergence-Driven Consistency Training for Semi-Supervised Facial Age Estimation

Facial age estimation has attracted considerable attention owing to its great potential in applications. However, it still falls short of reliable age estimation due to the lack of sufficient training data with accurate age labels. Using conventional semi-supervised methods to exploit unlabeled data appears to be a good solution, but it does not yield sufficient performance gains while significantly increasing training time. Therefore, to tackle these problems, we present a Divergence-driven Consistency Training (DCT) method for enhancing both efficiency and performance in this paper. Following the idea of pseudo-labeling and consistency regularization, we assign pseudo labels predicted by the teacher model to unlabeled samples and then train the student model on labeled and unlabeled samples based on consistency regularization. Based on this, we propose two main promotions. The first is the Efficient Sample Selection (ESS) strategy, which is based on the Divergence Score to select effective samples from massive unlabeled images to reduce the training time and improve efficiency. The second is Identity Consistency (IC) regularization as the additional loss function, which introduces a high dependency of aging traits on a person. Moreover, we propose Local Prediction (LP), which is a plug-and-play component, to capture local semantics. Extensive experiments on multiple age benchmark datasets, including CACD, Morph II, MIVIA, and Chalearn LAP 2015, indicate DCT outperforms the state-of-the-art approaches significantly.

A Data-Driven Model for Automated Chinese Word Segmentation and POS Tagging.

Chinese natural language processing tasks often require the solution of Chinese word segmentation and POS tagging problems. Traditional Chinese word segmentation and POS tagging methods mainly use simple matching algorithms based on lexicons and rules. The simple matching or statistical analysis requires manual word segmentation followed by POS tagging, which leads to the inability to meet the practical requirements for label prediction accuracy. With the continuous development of deep learning technology, data-driven machine learning models provide new opportunities for automated Chinese word segmentation and POS tagging. Therefore, a data-driven automated Chinese word segmentation and POS tagging model is proposed in order to address the above problems. Firstly, the main idea and overall framework of the proposed automated model are outlined, and the tagging strategy and neural network language model used are described. Secondly, two main optimisations are made on the input side of the model: (1) the use of word2Vec for the representation of text features, thus representing the text as a distributed word vector; and (2) the use of an improved AlexNet for efficient encoding of long-range word, and the addition of an attention mechanism to the model. Finally, on the output side, an additional auxiliary loss function was designed to optimise the Chinese text based on its frequency. The experimental results show that the proposed model can significantly improve the accuracy and operational efficiency of Chinese word segmentation and POS tagging compared with other existing models, thus verifying its effectiveness and advancement.

Grid adaptive reduced-order model of fluid flow based on graph convolutional neural network

In the interdisciplinary field of data-driven models and computational fluid mechanics, the reduced-order model for flow field prediction is mainly constructed by a convolutional neural network (CNN) in recent years. However, the standard CNN is only applicable to data with Euclidean spatial structure, while data with non-Euclidean properties can only be convolved after pixelization, which usually leads to decreased data accuracy. In this work, a novel data-driven framework based on graph convolution network (GCN) is proposed to allow the convolution operator to predict fluid dynamics on non-uniform structured or unstructured mesh data. This is achieved by the fact that the graph data inherit the spatial characteristics of the mesh and by the message passing mechanism of GCN. The conversion method from the form of mesh data to graph data and the operation mechanism of GCN are clarified. Moreover, additional relevance features and weight loss function of the dataset are also investigated to improve the model performance. The model learns an end-to-end mapping between the mesh spatial features and the physical flow field. Through our studies of various cases of internal flow, it is shown that the proposed GCN-based model offers excellent adaptability to non-uniformly distributed mesh data, while also achieving a high accuracy and three-order speedup compared with numerical simulation. Our framework generalizes the graph convolution network to flow field prediction and opens the door to further extending GCN to most existing data-driven architectures of fluid dynamics in the future.

Comparison of CNNs and Vision Transformers-Based Hybrid Models Using Gradient Profile Loss for Classification of Oil Spills in SAR Images

Oil spillage over a sea or ocean surface is a threat to marine and coastal ecosystems. Spaceborne synthetic aperture radar (SAR) data have been used efficiently for the detection of oil spills due to their operational capability in all-day all-weather conditions. The problem is often modeled as a semantic segmentation task. The images need to be segmented into multiple regions of interest such as sea surface, oil spill, lookalikes, ships, and land. Training of a classifier for this task is particularly challenging since there is an inherent class imbalance. In this work, we train a convolutional neural network (CNN) with multiple feature extractors for pixel-wise classification and introduce a new loss function, namely, “gradient profile” (GP) loss, which is in fact the constituent of the more generic spatial profile loss proposed for image translation problems. For the purpose of training, testing, and performance evaluation, we use a publicly available dataset with selected oil spill events verified by the European Maritime Safety Agency (EMSA). The results obtained show that the proposed CNN trained with a combination of GP, Jaccard, and focal loss functions can detect oil spills with an intersection over union (IoU) value of 63.95%. The IoU value for sea surface, lookalikes, ships, and land class is 96.00%, 60.87%, 74.61%, and 96.80%, respectively. The mean intersection over union (mIoU) value for all the classes is 78.45%, which accounts for a 13% improvement over the state of the art for this dataset. Moreover, we provide extensive ablation on different convolutional neural networks (CNNs) and vision transformers (ViTs)-based hybrid models to demonstrate the effectiveness of adding GP loss as an additional loss function for training. Results show that GP loss significantly improves the mIoU and F1 scores for CNNs as well as ViTs-based hybrid models. GP loss turns out to be a promising loss function in the context of deep learning with SAR images.

Dilated Generative Adversarial Networks for Underwater Image Restoration

Underwater images often come with blurriness, lack of contrast, and low saturation due to the physics of light propagation, absorption, and scattering in seawater. To improve the visual quality of underwater images, many have proposed image processing methods that vary based on different approaches. We use a generative adversarial network (GAN)-based solution and generate high-quality underwater images equivalent to given raw underwater images by training our network to specify the differences between high-quality and raw underwater images. In our proposed method, which is called dilated GAN (DGAN), we add an additional loss function using structural similarity. Moreover, this method can not only determine the realness of the entire image but also functions with classification ability on each constituent pixel in the discriminator. Finally, using two different datasets, we compare the proposed model with other enhancement methods. We conduct several comparisons and demonstrate via full-reference and nonreference metrics that the proposed approach is able to simultaneously improve clarity and correct color and restores the visual quality of the images acquired in typical underwater scenarios.

High fidelity deep learning-based MRI reconstruction with instance-wise discriminative feature matching loss.

To improve reconstruction fidelity of fine structures and textures in deep learning- (DL) based reconstructions. A novel patch-based Unsupervised Feature Loss (UFLoss) is proposed and incorporated into the training of DL-based reconstruction frameworks in order to preserve perceptual similarity and high-order statistics. The UFLoss provides instance-level discrimination by mapping similar instances to similar low-dimensional feature vectors and is trained without any human annotation. By adding an additional loss function on the low-dimensional feature space during training, the reconstruction frameworks from under-sampled or corrupted data can reproduce more realistic images that are closer to the original with finer textures, sharper edges, and improved overall image quality. The performance of the proposed UFLoss is demonstrated on unrolled networks for accelerated two- (2D) and three-dimensional (3D) knee MRI reconstruction with retrospective under-sampling. Quantitative metrics including normalized root mean squared error (NRMSE), structural similarity index (SSIM), and our proposed UFLoss were used to evaluate the performance of the proposed method and compare it with others. In vivo experiments indicate that adding the UFLoss encourages sharper edges and more faithful contrasts compared to traditional and learning-based methods with pure loss. More detailed textures can be seen in both 2D and 3D knee MR images. Quantitative results indicate that reconstruction with UFLoss can provide comparable NRMSE and a higher SSIM while achieving a much lower UFLoss value. We present UFLoss, a patch-based unsupervised learned feature loss, which allows the training of DL-based reconstruction to obtain more detailed texture, finer features, and sharper edges with higher overall image quality under DL-based reconstruction frameworks. (Code available at: https://github.com/mikgroup/UFLoss).

InfraGAN: A GAN architecture to transfer visible images to infrared domain

• Generative adversarial networks (GANs) can be useful to generate infrared (IR) images from visible images. • Using SSIM and pixel-based loss in our novel InfraGAN discriminator can help generating better IR images. • We report the state of the art results of InfraGAN in various metrics on large datasets. Utilizing both visible and infrared (IR) images in various deep learning based computer vision tasks has been a recent trend. Consequently, datasets having both visible and IR image pairs are desired in many applications. However, while large image datasets taken at the visible spectrum can be found in many domains, large IR-based datasets are not easily available in many domains. The lack of IR counterparts of the available visible image datasets limits existing deep algorithms to perform on IR images effectively. In this paper, to overcome with that challenge, we introduce a generative adversarial network (GAN) based solution and generate the IR equivalent of a given visible image by training our deep network to learn the relation between visible and IR modalities. In our proposed GAN architecture (InfraGAN), we introduce using structural similarity as an additional loss function. Furthermore, in our discriminator, we do not only consider the entire image being fake or real but also each pixel being fake or real. We evaluate our comparative results on three different datasets and report the state of the art results over five metrics when compared to Pix2Pix and ThermalGAN architectures from the literature. We report up to +16% better performance in Structural Similarity Index Measure (SSIM) over Pix2Pix and +8% better performance over ThermalGAN for VEDAI dataset. Further gains on different metrics and on different datasets are also reported in our experiments section.

AUE-Net: Automated Generation of Ultrasound Elastography Using Generative Adversarial Network.

Problem: Ultrasonography is recommended as the first choice for evaluation of thyroid nodules, however, conventional ultrasound features may not be able to adequately predict malignancy. Ultrasound elastography, adjunct to conventional B-mode ultrasound, can effectively improve the diagnostic accuracy of thyroid nodules. However, this technology requires professional elastography equipment and experienced physicians. Aim: in the field of computational medicine, Generative Adversarial Networks (GANs) were proven to be a powerful tool for generating high-quality images. This work therefore utilizes GANs to generate ultrasound elastography images. Methods: this paper proposes a new automated generation method of ultrasound elastography (AUE-net) to generate elastography images from conventional ultrasound images. The AUE-net was based on the U-Net architecture and optimized by attention modules and feature residual blocks, which could improve the adaptability of feature extraction for nodules of different sizes. The additional color loss function was used to balance color distribution. In this network, we first attempted to extract the tissue features of the ultrasound image in the latent space, then converted the attributes by modeling the strain, and finally reconstructed them into the corresponding elastography image. Results: a total of 726 thyroid ultrasound elastography images with corresponding conventional images from 397 patients were obtained between 2019 and 2021 as the dataset (646 in training set and 80 in testing set). The mean rating accuracy of the AUE-net generated elastography images by ultrasound specialists was 84.38%. Compared with that of the existing models in the visual aspect, the presented model generated relatively higher quality elastography images. Conclusion: the AUE-net generated ultrasound elastography images showed natural appearance and retained tissue information. Accordingly, it seems that B-mode ultrasound harbors information that can link to tissue elasticity. This study may pave the way to generate ultrasound elastography images readily without the need for professional equipment.

A Video Captioning Method Based on Multi-Representation Switching for Sustainable Computing

Video captioning is a problem that generates a natural language sentence as a video’s description. A video description includes not only words that express the objects in the video but also words that express the relationships between the objects, or grammatically necessary words. To reflect this characteristic explicitly using a deep learning model, we propose a multi-representation switching method. The proposed method consists of three components: entity extraction, motion extraction, and textual feature extraction. The proposed multi-representation switching method makes it possible for the three components to extract important information for a given video and description pair efficiently. In experiments conducted on the Microsoft Research Video Description dataset, the proposed method recorded scores that exceeded the performance of most existing video captioning methods. This result was achieved without any preprocessing based on computer vision and natural language processing, nor any additional loss function. Consequently, the proposed method has a high generality that can be extended to various domains in terms of sustainable computing.

Physics-informed echo state networks

We propose a physics-informed echo state network (ESN) to predict the evolution of chaotic systems. Compared to conventional ESNs, the physics-informed ESNs are trained to solve supervised learning tasks while ensuring that their predictions do not violate physical laws. This is achieved by introducing an additional loss function during the training, which is based on the system's governing equations. The additional loss function penalizes non-physical predictions without the need of any additional training data. This approach is demonstrated on a chaotic Lorenz system and a truncation of the Charney–DeVore system. Compared to the conventional ESNs, the physics-informed ESNs improve the predictability horizon by about two Lyapunov times. This approach is also shown to be robust with regard to noise. The proposed framework shows the potential of using machine learning combined with prior physical knowledge to improve the time-accurate prediction of chaotic dynamical systems.