Input Reconstruction Research Articles

Learned scalable video coding for humans and machines

AbstractVideo coding has traditionally been developed to support services such as video streaming, videoconferencing, digital TV, and so on. The main intent was to enable human viewing of the encoded content. However, with the advances in deep neural networks (DNNs), encoded video is increasingly being used for automatic video analytics performed by machines. In applications such as automatic traffic monitoring, analytics such as vehicle detection, tracking and counting, would run continuously, while human viewing could be required occasionally to review potential incidents. To support such applications, a new paradigm for video coding is needed that will facilitate efficient representation and compression of video for both machine and human use in a scalable manner. In this manuscript, we introduce an end-to-end learnable video codec that supports a machine vision task in its base layer, while its enhancement layer, together with the base layer, supports input reconstruction for human viewing. The proposed system is constructed based on the concept of conditional coding to achieve better compression gains. Comprehensive experimental evaluations conducted on four standard video datasets demonstrate that our framework outperforms both state-of-the-art learned and conventional video codecs in its base layer, while maintaining comparable performance on the human vision task in its enhancement layer.

A systematic review of progenitor survival and maturation in Parkinsonian models

Stem cell–based brain repair is a promising emergent therapy for Parkinson's disease based on years of foundational research using human fetal donors as a cell source. Unlike current therapeutic options for patients, this approach has the potential to provide long-term stem cell–derived reconstruction and restoration of the dopaminergic input to denervated regions of the brain allowing for restoration of certain functions to patients. The ultimate clinical success of stem cell–derived brain repair will depend on both the safety and efficacy of the approach and the latter is dependent on the ability of the transplanted cells to survive and differentiate into functional dopaminergic neurons in the Parkinsonian brain. Because the pre-clinical literature suggests that there is considerable variability in survival and differentiation between studies, the aim of this systematic review was to assess these parameters in human stem cell–derived dopaminergic progenitor transplant studies in animal models of Parkinson's disease. A defined systematic search of the PubMed database was completed to identify relevant studies published up to March 2024. After screening, 76 articles were included in the analysis from which 178 separate transplant studies were identified. From these, graft survival could be assessed in 52 studies and differentiation in 129 studies. Overall, we found that graft survival ranged from < 1% to 500% of cells transplanted, with a median of 51% of transplanted cells surviving in the brain; while dopaminergic differentiation of the cells ranged from 0% to 46% of cells transplanted with a median of 3%. This systematic review suggests that there is considerable scope for improvement in the differentiation of stem cell–derived dopaminergic progenitors to maximize the therapeutic potential of this approach for patients.

Can input reconstruction be used to directly estimate uncertainty of a dose prediction U-Net model?

The reliable and efficient estimation of uncertainty in artificial intelligence (AI) models poses an ongoing challenge in many fields such as radiation therapy. AI models are intended to automate manual steps involved in the treatment planning workflow. We focus in this study on dose prediction models that predict an optimal dose trade-off for each new patient for a specific treatment modality. They can guide physicians in the optimization, be part of automatic treatment plan generation or support decision in treatmentindication. Most common uncertainty estimation methods are based on Bayesian approximations, like Monte Carlo dropout (MCDO) or Deep ensembling (DE). These two techniques, however, have a high inference time (i.e., require multiple inference passes) and might not work for detecting out-of-distribution (OOD) data (i.e., overlapping uncertainty estimate for in-distribution (ID) and OOD). In this study, we present a direct uncertainty estimation method and apply it for a dose prediction U-Net architecture. It can be used to flag OOD data and give information on the quality of the doseprediction. Our method consists in the addition of a branch decoding from the bottleneck which reconstructs the CT scan given as input. The input reconstruction error can be used as a surrogate of the model uncertainty. For the proof-of-concept, our method is applied to proton therapy dose prediction in head and neck cancer patients. A dataset of 60 oropharyngeal patients was used to train the network using a nested cross-validation approach with 11 folds (training: 50 patients, validation: 5 patients, test: 5 patients). For the OOD experiment, we used 10 extra patients with a different head and neck sub-location. Accuracy, time-gain, and OOD detection are analyzed for our method in this particular application and compared with the popular MCDO andDE. The additional branch did not reduce the accuracy of the dose prediction model. The median absolute error is close to zero for the target volumes and less than 1% of the dose prescription for organs at risk. Our input reconstruction method showed a higher Pearson correlation coefficient with the prediction error (0.620) than DE (0.447) and MCDO (between 0.599 and 0.612). Moreover, our method allows an easier identification of OOD (no overlap for ID and OOD data and a Z-score of 34.05). The uncertainty is estimated simultaneously to the regression task, therefore requires less time and computationalresources. This study shows that the error in the CT scan reconstruction can be used as a surrogate of the uncertainty of the model. The Pearson correlation coefficient with the dose prediction error is slightly higher than state-of-the-art techniques. OOD data can be more easily detected and the uncertainty metric is computed simultaneously to the regression task, therefore faster than MCDO or DE. The code and pretrained model are available on the gitlab repository: https://gitlab.com/ai4miro/ct-reconstruction-for-uncertainty-quatification-of-hdunet.

Resilient Self-Triggered Model Predictive Control of Cyber-Physical Systems Under Two-Channel False Data Injection Attacks

Abstract This paper presents a novel resilient self-triggered model predictive control (ST-MPC) method to alleviate potential threats of false data injection (FDI) attacks on cyber-physical systems (CPSs). First, considering that the data transmitted via the sensor-to-controller (S–C) and controller-to-actuator (C–A) channels in CPS may be tampered with by FDI attacks, a novel input reconstruction strategy combined with the ST-MPC mechanism is proposed to alleviate the threats of FDI attacks while reducing the computational and communication resources, in which key optimal control signals are selected and protected based on systematic performance inequalities. Correspondingly, a resilient ST-MPC algorithm combined with the dual-mode strategy is further proposed. Moreover, the iterative feasibility and the closed-loop stability are strictly demonstrated. Finally, the effectiveness of the proposed strategy is verified via a simulation study.

Multi-scale input reconstruction network and one-stage instance segmentation for enhancing heart defect prediction rate

<p>Artifacts and unpredictable fetal movements can hinder clear fetal heart imaging during ultrasound scans, complicating anatomical identification. This study presents a new medical imaging approach that combines one-stage instance segmentation with ultrasound (US) video enhancement for precise fetal heart defect detection. This innovation allows real-time identification and timely medical intervention. The study acquired 100 fetal heart US videos from an Indonesian Hospital featuring cardiac septal defects, generating 1,000 frames for training, validation, and testing. Utilizing a combination of the multi-scale input reconstruction network (MIRNet) for image enhancement and YOLOv8l-seg for real-time instance segmentation, the method achieved outstanding validation results, boasting a 99.50% mAP for bounding box prediction and 98.40% for mask prediction. It delivered a remarkable real-time processing speed of 68.4 frames per second. In application to new patients, the method yielded a 65.93% mAP for bounding box prediction and 57.66% for mask prediction. This proposed approach offers a promising solution to early fetal heart defect detection using ultrasound, holding substantial potential for enhancing healthcare outcomes.</p>

Efficient sparse spiking auto-encoder for reconstruction, denoising and classification

Auto-encoders are capable of performing input reconstruction, denoising, and classification through an encoder-decoder structure. Spiking Auto-Encoders (SAEs) can utilize asynchronous sparse spikes to improve power efficiency and processing latency on neuromorphic hardware. In our work, we propose an efficient SAE trained using only Spike-Timing-Dependant Plasticity (STDP) learning. Our auto-encoder uses the Time-To-First-Spike (TTFS) encoding scheme and needs to update all synaptic weights only once per input, promoting both training and inference efficiency due to the extreme sparsity. We showcase robust reconstruction performance on the Modified National Institute of Standards and Technology (MNIST) and Fashion-MNIST datasets with significantly fewer spikes compared to state-of-the-art SAEs by 1–3 orders of magnitude. Moreover, we achieve robust noise reduction results on the MNIST dataset. When the same noisy inputs are used for classification, accuracy degradation is reduced by 30%–80% compared to prior works. It also exhibits classification accuracies comparable to previous STDP-based classifiers, while remaining competitive with other backpropagation-based spiking classifiers that require global learning through gradients and significantly more spikes for encoding and classification of MNIST/Fashion-MNIST inputs. The presented results demonstrate a promising pathway towards building efficient sparse spiking auto-encoders with local learning, making them highly suited for hardware integration.

Fault detection of industrial processes using attention-based gated recurrent unit autoencoder with skip connection

Abstract With the continuous evolution of modern industrial technology, industrial production has grown progressively complex, necessitating the use of various sensors to measure multiple process variables. However, intricate temporal dependencies and nonlinear relationships between data presented by multivariate sequences pose significant challenges to process fault detection. In response to these challenges, this paper proposes an attention-based gated recurrent unit autoencoder with skip connection (SAGRU-AE) model for monitoring large-scale, nonlinear, and multivariate industrial process faults. SAGRU-AE combines gated recurrent units, multi-head self-attention, and autoencoder to extract features from multivariable time series data efficiently. Concurrently, feature reuse is achieved through the skip connection structure, which improves the accuracy of data reconstruction. Based on the implementation of process data feature extraction and input reconstruction in SAGRU-AE, two statistics have been developed, namely the H2 statistic and square prediction error (SPE) statistic, for fault detection tasks. Ultimately, the feasibility and effectiveness of the proposed algorithms are validated through experimentation on the TE process.

Transformations establishing equivalence across neural networks: When have two networks learned the same task?

Transformations are a key tool in the qualitative study of dynamical systems: transformations to a normal form, for example, underpin the study of instabilities and bifurcations. In this work, we test, and when possible establish, an equivalence between two different artificial neural networks by attempting to construct a data-driven transformation between them, using diffusion maps with a Mahalanobis-like metric. If the construction succeeds, the two networks can be thought of as belonging to the same equivalence class. We first discuss transformation functions between only the outputs of the two networks; we then also consider transformations that take into account outputs (activations) of a number of internal neurons from each network. Whitney's theorem dictates the number of (generic) measurements from one of the networks required to reconstruct each and every feature of the second network. The construction of the transformation function relies on a consistent, intrinsic representation of the network input space. We illustrate our algorithm by matching neural network pairs trained to learn (a) observations of scalar functions, (b) observations of two-dimensional vector fields, and (c) representations of images of a moving three-dimensional object (a rotating horse). We also demonstrate reconstruction of a network's input (and output) from minimal partial observations of intermediate neuron activations. The construction of equivalences across different network instantiations clearly relates to transfer learning and will also be valuable in establishing equivalence between different machine learning-based tools.

RECONSTRUCTING RETINAL VISUAL IMAGES FROM 3T FMRI DATA ENHANCED BY UNSUPERVISED LEARNING.

The reconstruction of human visual inputs from brain activity, particularly through functional Magnetic Resonance Imaging (fMRI), holds promising avenues for unraveling the mechanisms of the human visual system. Despite the significant strides made by deep learning methods in improving the quality and interpretability of visual reconstruction, there remains a substantial demand for high-quality, long-duration, subject-specific 7-Tesla fMRI experiments. The challenge arises in integrating diverse smaller 3-Tesla datasets or accommodating new subjects with brief and low-quality fMRI scans. In response to these constraints, we propose a novel framework that generates enhanced 3T fMRI data through an unsupervised Generative Adversarial Network (GAN), leveraging unpaired training across two distinct fMRI datasets in 7T and 3T, respectively. This approach aims to overcome the limitations of the scarcity of high-quality 7-Tesla data and the challenges associated with brief and low-quality scans in 3-Tesla experiments. In this paper, we demonstrate the reconstruction capabilities of the enhanced 3T fMRI data, highlighting its proficiency in generating superior input visual images compared to data-intensive methods trained and tested on a single subject.

Optimal sensor placement for joint reconstruction of multiscale responses and unknown inputs using modal Kalman filter

SummaryMany optimal sensor placement (OSP) techniques have been developed basing on known external loads. However, it is often difficult to obtain excitation measurements. Therefore, the development of OSP under unknown inputs (OSP‐UI) is desirable. In this paper, based on modal Kalman filter (MKF), an OSP‐UI approach (MKF‐OSP‐UI) is proposed for optimally determining the number and locations of multitype sensors with the aim of minimizing the reconstructed responses errors. An MKF‐based approach previously developed by the authors is first employed for estimating multiscale structural responses and unknown loads. Then, an error covariance matrix is defined as a measure of the differences between the reconstructed responses and the corresponding actual ones. By using the covariance matrix of measurement noise for normalization, the ill‐conditioning problem caused by data fusion of multiscale responses is avoided. The sensors that have few contributions to the reconstructed responses are removed from the candidate set during iteration procedure. The sensor placement is finally determined when the estimation errors are below the preset level. Numerical results show that the sensor configuration determined by the proposed approach has a better performance on the joint estimation of multiscale responses and unknown inputs, as compared with that determined by experience.

Video anomaly detection based on a multi-layer reconstruction autoencoder with a variance attention strategy

In this paper, we propose a comprehensive framework for detecting anomalies in videos based on autoencoder (AE). Traditional AE models solely rely on input and final reconstruction, potentially limiting their capacity to fully utilize the intermediate neural network layers. To mitigate this limitation, we introduce a novel approach that concurrently trains the model using corresponding intermediate layers from both the encoder and decoder. This allows the model to capture more intricate features, thus enhancing its anomaly detection capabilities. Furthermore, we introduce a motion loss function that exclusively relies on original video frames rather than optical flow, rendering it more efficient and capable of extracting motion features. Additionally, we have devised a variance attention strategy that is parameter-free and can automatically directs our model's focus towards moving objects, further boosting the performance of our approach. Our experiments on three public datasets demonstrate the effectiveness and efficiency of our method in identifying abnormal events in complex scenarios. The code is publicly available at https://github.com/lsf2008/multRecLossAEPub.

D‐Unet: A symmetric architecture of convolutional neural network with two auxiliary outputs for dementia recognition

AbstractDementia‐associated disorders cause damage to the brains of patients and bring huge burdens to individuals and families. Electroencephalogram (EEG) monitoring is friendly to patients on account of low cost, non‐invasion, and objective. Event‐related potential (ERP) is a component of EEG that has huge potential to evaluate the cognitive function of the brain. In this study, we recorded the ERP from patients with dementia and healthy people, then proposed an ERP‐based deep learning method to realize dementia recognition via the model Dementia‐Unet (D‐Unet). To improve the performance of the model, on the base of the decoder and the primary classifier, we added new structures including a symmetric decoder and two auxiliary outputs. One of the auxiliary outputs was input reconstruction, and the other one was aimed at working like the primary classifier with the same task. The results of the experiment of four‐fold cross‐validation demonstrated the two auxiliary outputs improved the performance of the model effectively. When compared with some other machine learning methods and deep learning methods, our model obtained the best performance with an accuracy of 0.815, a precision of 0.829, a recall of 0.797, and an f1‐score of 0.812. Besides, we put up a complex training strategy with all outputs involved, but a simple testing strategy with only a primary classifier working to keep high performance but cut down the complexity burden during testing.

Vertical-horizontal latent space with iterative memory review network for multi-class anomaly detection

Anomaly detection has been recently proposed as a visual scene understanding task and widely applied in industrial detection. Traditional unsupervised methods aim to enhance anomaly detection performance by improving input reconstruction quality. However, these approaches rarely address the problem of poor quality reconstruction information. The inferior information affects the reconstruction effect of the input images, and further weakens the recognition performance of anomaly detection. Moreover, existing methods ignore the features of generated images and information interaction between modules, resulting in insufficient use of information. In this paper, a novel Iterative Memory Review Network (IMRN) based on horizontal-vertical latent space is presented to detect anomalies by selecting prior reconstruction information and strengthening the interactivity between network modules. An iterative memory module is recommended for the generator to encode high-dimensional inputs. Meanwhile, a review module is proposed for image modeling by extracting local and global features and fusing reconstruction process information. While completing the expansion of horizontal latent space, the review module also encodes vertical information in different directions from the iterative memory module. Our proposed approach is validated in multi-class anomaly detection tasks on public datasets. Extensive experimental results on the MVTec dataset verify the superiority of our proposed IMRN.

Orbital Controls on North Pacific Dust Flux During the Late Quaternary

AbstractAirborne mineral dust is sensitive to climatic changes, but its response to orbital forcing is still not fully understood. Here, we present a reconstruction of dust input to the Subarctic Pacific Ocean covering the past 190 kyr. The dust composition record is indicative of source moisture conditions, which were dominated by precessional variations. In contrast, the dust flux record is dominated by obliquity variations and displays an out‐of‐phase relationship with a dust record from the mid‐latitude North Pacific Ocean. Climate model simulations suggest precession likely drove changes in the aridity and extent of dust source regions. Additionally, the obliquity variations in dust flux can be explained by meridional shifts in the North Pacific westerly jet, driven by changes in the meridional atmospheric temperature gradient. Overall, our findings suggest that North Pacific dust input was primarily modulated by orbital‐controlled source aridity and the strength and position of the westerly winds.

An reconstruction bidirectional recurrent neural network ‐based deinterleaving method for known radar signals in open‐set scenarios

AbstractIn electronic warfare, radar signal deinterleaving is a critical task. While many researchers have applied deep learning and utilised known radar classes to construct interleaved pulse sequences training sets for deinterleaving models, these models face challenges in distinguishing between known and unknown radar classes in open‐set scenarios. To address this challenge, the authors propose a novel model, the Reconstruction Bidirectional Recurrent Neural Network (RBi‐RNN). RBi‐RNN utilises input reconstruction and employs a joint training strategy incorporating cross‐entropy loss, reconstruction loss, and centre loss. These strategies aim to maximise inter‐class latent representation distances while minimising intra‐class disparities. By incorporating an open‐set recognition method based on extreme value theory, RBi‐RNN adapts to open‐set scenarios. Simulation results demonstrate the superiority of RBi‐RNN over conventional models in both closed‐set and open‐set scenarios. In open‐set scenarios, it successfully discriminates between known and unknown radar signals within interleaved pulse sequences, deinterleaving known radar classes with high stability. The authors lay the foundation for future unsupervised deinterleaving methods designed specifically for unknown radar pulses.

Mind Reasoning Manners: Enhancing Type Perception for Generalized Zero-Shot Logical Reasoning Over Text.

Logical reasoning task involves diverse types of complex reasoning over text, based on the form of multiple-choice question answering (MCQA). Given the context, question and a set of options as the input, previous methods achieve superior performances on the full-data setting. However, the current benchmark dataset has the ideal assumption that the reasoning type distribution on the train split is close to the test split, which is inconsistent with many real application scenarios. To address it, there remain two problems to be studied: 1) how is the zero-shot capability of the models (train on seen types and test on unseen types)? and 2) how to enhance the perception of reasoning types for the models? For problem 1, we propose a new benchmark for generalized zero-shot logical reasoning, named ZsLR. It includes six splits based on the three type sampling strategies. For problem 2, a type-aware model TaCo is proposed. It utilizes the heuristic input reconstruction and builds a text graph with a global node. Incorporating graph reasoning and contrastive learning, TaCo can improve the type perception in the global representation. Extensive experiments on both the zero-shot and full-data settings prove the superiority of TaCo over the state-of-the-art (SOTA) methods. Also, we experiment and verify the generalization capability of TaCo on other logical reasoning dataset.

Asymptotical left inversion for single-input linear systems

A constructive design of a stable left inverse for single-input linear systems is proposed. Based on structural properties relating left inversion and feasibility of input reconstruction, the input observer is realized within the state-space approach, for arbitrary initial condition. While previous design is concerned with a nominal plant, that is with known parameters, we propose in a second step to apply this inversion methodology to a simplified identification problem. Reformulating the unknown parameters as a fictitious unknown input, convergent parameter estimates are retrieved from measurement. The overall methodology is illustrated by simulations.

Human-in-the-Loop Formation-Containment Control for Multiagent Systems: An Observer-Based Distributed Unknown Input Reconstruction Method

This paper studies the human-in-the-loop time-varying formation-containment tracking (TVFCT) control problem of general linear multiagent systems (MASs). The MASs consist of a non-autonomous tracking leader, multiple formation leaders and followers, and all agents are subject to unknown disturbances. A human operator is deployed to supervise and regulate the entire team by sending a command signal to the tracking leader. The control input of the tracking leader should be combined with the human operator but not available to other agents. First, a local observer associated with each agent is constructed to provide asymptotic state estimation, in which the observer error system is disturbance-free. Also, interval observers are established for the neighborhood TVFCT errors, where the disturbance of the current agent is aggregated with the disturbances and input signals of its neighbors into an unknown input. Meanwhile, a distributed asymptotic unknown input reconstruction (UIR) mechanism is proposed that decouples the control input of the current agent. The consequent human-in-the-loop TVFCT control protocol is designed by combining the local observer and distributed UIR scheme. Finally, the theoretical results are verified in multi-robot formation-containment control.

Observer‐based event‐triggered consensus control for multi‐agent systems with nonlinearity and unknown inputs

AbstractThis article investigates observer‐based consensus issues for a class of Lipschitz nonlinear multi‐agent systems (MASs) with unknown input. To address the challenges because of the unmeasurable states and the unknown input of each agent system, an interval observer is utilized to design an unknown input observer (UIO). The UIO possesses the capability of generating asymptotically the estimations of both the system states and unknown input simultaneously. Subsequently, a distributed control protocol scheme is put forwarded. The distributed controller for each agent is designed by using the state estimation and the unknown input reconstruction (UIR). The unknown input is compensated thanks for the introduction of the UIR into the controller, and thus leads to a better control performance. Besides, the event‐triggered strategy is also considered such that agents communicate with each other discontinuously. In this way, an observer‐based event‐triggered distributed control protocol scheme is developed, and under the scheme, the asymptotic convergent MASs consensus is accomplished. Finally, the effectiveness of the proposed methods is verified by a simulation example.

Detecting Adversarial Examples on Deep Neural Networks With Mutual Information Neural Estimation

Despite achieving exceptional performance, deep neural networks (DNNs) suffer from the harassment caused by adversarial examples, which are produced by corrupting clean examples with tiny perturbations. Many powerful defense methods have been presented such as training data augmentation and input reconstruction which, however, usually rely on the prior knowledge of the targeted models or attacks. A clean example and its adversarial version are very similar but have different high-level representations in a victim model. If we can obtain a space in which the representations of similar examples are also similar, then adversarial examples can be picked out by comparing the representations of input examples in this space and the high-level space of the victim model. Inspired by this, we propose a novel approach for detecting adversarial images, which can protect any pre-trained DNN classifiers and resist an endless stream of new attacks. Specifically, we first adopt a dual autoencoder to project images to a latent space. The dual autoencoder uses the self-supervised learning to ensure that small modifications to samples do not significantly alter their latent representations. Next, the mutual information neural estimation is utilized to enhance the discrimination of the latent representations. We then leverage the prior distribution matching to regularize the latent representations. To easily compare the representations of examples in the two spaces, and not rely on the prior knowledge of the targeted model, a simple fully connected neural network is used to embed the learned representations into an eigenspace, which is consistent with the output eigenspace of the targeted model. Through the distribution similarity of an input example in the two eigenspaces, we can judge whether the input example is adversarial or not. Extensive experiments on MNIST, CIFAR-10, and ImageNet show that the proposed method has superior defense performance and transferability than state-of-the-arts.