Forward Pass Research Articles

Dense open-set recognition based on training with noisy negative images

Deep convolutional models often produce inadequate predictions for inputs which are foreign to the training distribution. Consequently, the problem of detecting outlier images has recently been receiving a lot of attention. Unlike most previous work, we address this problem in the dense prediction context. Our approach is based on two reasonable assumptions. First, we assume that the inlier dataset is related to some narrow application field (e.g. road driving). Second, we assume that there exists a general-purpose dataset which is much more diverse than the inlier dataset (e.g. ImageNet-1 k). We consider pixels from the general-purpose dataset as noisy negative samples since most (but not all) of them are outliers. We encourage the model to recognize borders between the known and the unknown by pasting jittered negative patches over inlier training images. Our experiments target two dense open-set recognition benchmarks (WildDash 1 and Fishyscapes) and one dense open-set recognition dataset (StreetHazard). Extensive performance evaluation indicates competitive potential of the proposed approach. • Training with noisy negative images greatly improves dense open-set recognition. • Training with randomly pasted negatives improves generalization on mixed-content images. • Shared features improve outlier detection and allow for inference with a single forward pass. • State-of-the-art results on dense open-set recognition benchmarks: WildDash 1, Fishyscapes and StreetHazard.

Scaffolding protein functional sites using deep learning.

The binding and catalytic functions of proteins are generally mediated by a small number of functional residues held in place by the overall protein structure. Here, we describe deep learning approaches for scaffolding such functional sites without needing to prespecify the fold or secondary structure of the scaffold. The first approach, "constrained hallucination," optimizes sequences such that their predicted structures contain the desired functional site. The second approach, "inpainting," starts from the functional site and fills in additional sequence and structure to create a viable protein scaffold in a single forward pass through a specifically trained RoseTTAFold network. We use these two methods to design candidate immunogens, receptor traps, metalloproteins, enzymes, and protein-binding proteins and validate the designs using a combination of in silico and experimental tests.

Adaptive stochastic resonance based convolutional neural network for image classification

In this paper, we exploit the adaptive stochastic resonance effect in the convolutional neural network with threshold activation functions for enabling the back-propagation gradient computation. During training, the injection of noise into the threshold activation function makes it to be differentiable at forward pass, thus exact gradients of the loss function with respect to network parameters including the injected noise level can smoothly propagate at backforward pass. This training method effectively uses the non-zero noise to improve the performance of the designed convolutional neural network. In the testing phase, the differentiable activation function is decoupled into a set of threshold functions driven by mutually independent noise components, which endows a hardware-friendly feature of the designed neural network. Experiments show that the designed low-precision convolutional neural network has a comparable performance with the full-precision one with ReLU activation functions on two benchmark data sets. These results also extend the potential application of adaptive stochastic resonance into the convolutional neural network for image classification.

Kinetics of two-step catalytic sequence on nanoclusters with limited cluster occupancy

Two-step catalytic sequence was considered for cooperative kinetics of heterogeneous catalytic reactions on nanoclusters with a limited number of adsorbed species due to size restrictions allowing for derivation of the corresponding rate expressions. Dependence of the kinetic parameters on the number of adsorbed species was considered for different models of biographical (intrinsic) and induced non uniform surfaces. Explicit expressions were derived for the rates on evenly nonuniform surfaces exhibiting a linear decrease in the Gibbs energy with site occupancy. An expression for the apparent activation energy was obtained for the two step sequence with both irreversible steps. Simulations were performed for several expressions for the forward reaction rate in a two-step sequence with different number of maximal adsorbed species. The analysis illustrated that the reaction rates in the forward direction pass through maxima even without any competition of both reactants. Strong nonuniformity or lateral interactions make the rate maxima less prominent. Larger clusters, which can accommodate more adsorbed intermediates per cluster, exhibit rather wide maxima in the rates expanding over a broad range of concentrations.The experimental data on resazurin reduction with hydroxylamine over gold clusters bearing 15 to 25 gold atoms were used to illustrate applicability of the developed theoretical approach showing a dramatic switch in kinetic regularities between Au15 and Au18 clusters, which could be attributed to changes in the number of adsorbed species per cluster.

Batch-Wise Permutation Feature Importance Evaluation and Problem-Specific Bigraph for Learn-to-Branch

The branch-and-bound algorithm for combinatorial optimization typically relies on a plethora of handcraft expert heuristics, and a research direction, so-called learn-to-branch, proposes to replace the expert heuristics in branch-and-bound with machine learning models. Current studies in this area typically use an imitation learning (IL) approach; however, in practice, IL often suffers from limited training samples. Thus, it has been emphasized that a small-dataset fast-training scheme for IL in learn-to-branch is worth studying, so that other methods, e.g., reinforcement learning, may be used for subsequent training. Thus, this paper focuses on the IL part of a mixed training approach, where a small-dataset fast-training scheme is considered. The contributions are as follows. First, to compute feature importance metrics so that the state-of-the-art bigraph representation can be effectively reduced for each problem type, a batch-wise permutation feature importance evaluation method is proposed, which permutes features within each batch in the forward pass. Second, based on the evaluated importance of the bigraph features, a reduced bigraph representation is proposed for each of the benchmark problems. The experimental results on four MILP benchmark problems show that our method improves branching accuracy by 8% and reduces solution time by 18% on average under the small-dataset fast-training scheme compared to the state-of-the-art bigraph-based learn-to-branch method. The source code is available online at GitHub.

Neural mixture models with expectation-maximization for end-to-end deep clustering

Any clustering algorithm must synchronously learn to model the clusters and allocate data to those clusters in the absence of labels. Mixture model-based methods model clusters with pre-defined statistical distributions and allocate data to those clusters based on the cluster likelihoods. They iteratively refine those distribution parameters and member assignments following the Expectation-Maximization (EM) algorithm. However, the cluster representability of such hand-designed distributions that employ a limited amount of parameters is not adequate for most real-world clustering tasks. In this paper, we realize mixture model-based clustering with a neural network where the final layer neurons, with the aid of an additional transformation, approximate cluster distribution outputs. The network parameters pose as the parameters of those distributions. The result is an elegant, much-generalized representation of clusters than a restricted mixture of hand-designed distributions. We train the network end-to-end via batch-wise EM iterations where the forward pass acts as the E-step and the backward pass acts as the M-step. In image clustering, the mixture-based EM objective can be used as the clustering objective along with existing representation learning methods. In particular, we show that when mixture-EM optimization is fused with consistency optimization, it improves the sole consistency optimization performance in clustering. Our trained networks outperform single-stage deep clustering methods that still depend on k-means, with unsupervised classification accuracy of 63.8% in STL10, 58% in CIFAR10, 25.9% in CIFAR100, and 98.9% in MNIST.

Video Capsule Endoscopy Classification using Focal Modulation Guided Convolutional Neural Network.

Video capsule endoscopy is a hot topic in computer vision and medicine. Deep learning can have a positive impact on the future of video capsule endoscopy technology. It can improve the anomaly detection rate, reduce physicians' time for screening, and aid in real-world clinical analysis. Computer-Aided diagnosis (CADx) classification system for video capsule endoscopy has shown a great promise for further improvement. For example, detection of cancerous polyp and bleeding can lead to swift medical response and improve the survival rate of the patients. To this end, an automated CADx system must have high throughput and decent accuracy. In this study, we propose FocalConvNet, a focal modulation network integrated with lightweight convolutional layers for the classification of small bowel anatomical landmarks and luminal findings. FocalConvNet leverages focal modulation to attain global context and allows global-local spatial interactions throughout the forward pass. Moreover, the convolutional block with its intrinsic inductive/learning bias and capacity to extract hierarchical features allows our FocalConvNet to achieve favourable results with high throughput. We compare our FocalConvNet with other state-of-the-art (SOTA) on Kvasir-Capsule, a large-scale VCE dataset with 44,228 frames with 13 classes of different anomalies. We achieved the weighted F1-score, recall and Matthews correlation coefficient (MCC) of 0.6734, 0.6373 and 0.2974, respectively, outperforming SOTA methodologies. Further, we obtained the highest throughput of 148.02 images/second rate to establish the potential of FocalConvNet in a real-time clinical environment. The code of the proposed FocalConvNet is available at https://github.com/NoviceMAn-prog/FocalConvNet.

Latent Time Neural Ordinary Differential Equations

Neural ordinary differential equations (NODE) have been proposed as a continuous depth generalization to popular deep learning models such as Residual networks (ResNets). They provide parameter efficiency and automate the model selection process in deep learning models to some extent. However, they lack the much-required uncertainty modelling and robustness capabilities which are crucial for their use in several real-world applications such as autonomous driving and healthcare. We propose a novel and unique approach to model uncertainty in NODE by considering a distribution over the end-time T of the ODE solver. The proposed approach, latent time NODE (LT-NODE), treats T as a latent variable and apply Bayesian learning to obtain a posterior distribution over T from the data. In particular, we use variational inference to learn an approximate posterior and the model parameters. Prediction is done by considering the NODE representations from different samples of the posterior and can be done efficiently using a single forward pass. As T implicitly defines the depth of a NODE, posterior distribution over T would also help in model selection in NODE. We also propose, adaptive latent time NODE (ALT-NODE), which allow each data point to have a distinct posterior distribution over end-times. ALT-NODE uses amortized variational inference to learn an approximate posterior using inference networks. We demonstrate the effectiveness of the proposed approaches in modelling uncertainty and robustness through experiments on synthetic and several real-world image classification data.

Modified convolutional neural network with pseudo-CNN for removing nonlinear noise in digital images

Recently, deep learning techniques are widely used in various computer vision applications such as pattern recognition, data classification, object detection, image enhancement, etc. Deep learning frameworks outperform classical algorithms due to their flexibility and interoperability. The main aim of this work is to remove Gaussian-Impulse noise with blind (unknown) noise densities in digital images. The proposed work has two phases; in phase 1, the noisy image is preprocessed with a novel Pseudo–Convolutional Neural Network (P-CNN) having no tunable parameters. The Pseudo-CNN is a customized three-layered network inspired by conventional convolutional neural networks. The first-level feature detector has k number of (2d+1)×(2d+1) Weight Initialized Adaptive Window (WIAW) filters. The weights or coefficients of WIAW filters are initialized using the noise probabilistic distribution. The loss function is not estimated in P-CNN because there is only one forward pass and no backpropagation. The PCNN has excellent Impulse noise rejection capability. In phase 2, a modified Convolutional Neural Network (CNN) is applied to the preprocessed image to obtain a latent clean image. To improve the network performance, we have utilized residual learning and batch normalization. The high degree of localized pixel correlation established by P-CNN helps the modified CNN to learn compact representations and salient features of the input image. The proposed method not only denoises images corrupted with blind noise levels of Gaussian-Impulse noise but is also effective for removing Impulse noise in digital images. Our experimental results imply that the proposed method gives good qualitative and quantitative results compared to various state-of-the-art techniques. Convolutional neural networks are pertinent for parallel computation using powerful GPUs, which help to improve the denoising performance.

Accelerating in silico saturation mutagenesis using compressed sensing.

In silico saturation mutagenesis (ISM) is a popular approach in computational genomics for calculating feature attributions on biological sequences that proceeds by systematically perturbing each position in a sequence and recording the difference in model output. However, this method can be slow because systematically perturbing each position requires performing a number of forward passes proportional to the length of the sequence being examined. In this work, we propose a modification of ISM that leverages the principles of compressed sensing to require only a constant number of forward passes, regardless of sequence length, when applied to models that contain operations with a limited receptive field, such as convolutions. Our method, named Yuzu, can reduce the time that ISM spends in convolution operations by several orders of magnitude and, consequently, Yuzu can speed up ISM on several commonly used architectures in genomics by over an order of magnitude. Notably, we found that Yuzu provides speedups that increase with the complexity of the convolution operation and the length of the sequence being analyzed, suggesting that Yuzu provides large benefits in realistic settings. We have made this tool available at https://github.com/kundajelab/yuzu.

Uncertainty Quantification in Segmenting Tuberculosis-Consistent Findings in Frontal Chest X-rays.

Deep learning (DL) methods have demonstrated superior performance in medical image segmentation tasks. However, selecting a loss function that conforms to the data characteristics is critical for optimal performance. Further, the direct use of traditional DL models does not provide a measure of uncertainty in predictions. Even high-quality automated predictions for medical diagnostic applications demand uncertainty quantification to gain user trust. In this study, we aim to investigate the benefits of (i) selecting an appropriate loss function and (ii) quantifying uncertainty in predictions using a VGG16-based-U-Net model with the Monto–Carlo (MCD) Dropout method for segmenting Tuberculosis (TB)-consistent findings in frontal chest X-rays (CXRs). We determine an optimal uncertainty threshold based on several uncertainty-related metrics. This threshold is used to select and refer highly uncertain cases to an expert. Experimental results demonstrate that (i) the model trained with a modified Focal Tversky loss function delivered superior segmentation performance (mean average precision (mAP): 0.5710, 95% confidence interval (CI): (0.4021,0.7399)), (ii) the model with 30 MC forward passes during inference further improved and stabilized performance (mAP: 0.5721, 95% CI: (0.4032,0.7410), and (iii) an uncertainty threshold of 0.7 is observed to be optimal to refer highly uncertain cases.

Approximate dynamic programming with policy-based exploration for microgrid dispatch under uncertainties

Approximate dynamic programming (ADP) is a promising approach for power system scheduling and dispatch under uncertainties. This paper presents an innovative ADP-based dispatch method for a microgrid with intermittent renewable generation, battery energy storage systems, and controllable distributed generators. The proposed ADP algorithm is based on a double-pass value iteration approach and takes advantage of the underlying properties of the microgrid dispatch problem. In the forward pass, decision variables are updated moving forward in time using an ε-greedy strategy to balance exploitation and exploration. In particular, an approximate optimization method is proposed to speed up exploitation. In addition to random exploration, a policy is designed to guide the algorithm to explore some promising solution space in a probabilistic manner. In the backward pass, the value function is updated moving backward in time using the trajectory of states, decisions, and outcomes of the sample path in the forward pass. The proposed method is evaluated through numerical experiments in both deterministic and stochastic environments. Case study results show that the proposed method demonstrates improved performance in both optimization gap and computation time in comparison to conventional methods.

УСКОРЕНИЕ ПРЯМОГО ПРОХОДА ПРИ РЕАЛИЗАЦИИ СНС НА ОГРАНИЧЕННОМ ВЫЧИСЛИТЕЛЬНОМ РЕСУРСЕ

The work is devoted to the optimization of the neural network architecture for its launch ona limited computing resource. Several optimization approaches are considered, estimates of thecomplexity and execution time of the forward pass of the neural network are given. Comparativeestimates of the complexity of the network using different optimization approaches are given.The paper presents an analysis of the selected network architecture, and estimates of the computationalcomplexity of individual components (modules) of the architecture are obtained. An analysisof possible optimization methods for each module was made. The parameters of the consideredmodules, the sizes of the input and output tensors are described. Several architectures were testedto optimize the feature extraction module, ResNet 50, ResNet 18, MobileNet v3 small, MobileNetv3 large. A comparative analysis of the computational complexity and execution time of the forwardpass for each architecture is presented. Forward pass times were measured on Nvidia'sJetson AGX Xaver embedded computing device. Estimates of the execution time of the direct passfor each module of the considered neural networks are presented. The paper presents the results ofcomparing neural network accuracy estimates before and after architecture optimization. The testdata set consists of 100 video recordings. 5 different typical objects are involved in test videos,10 different scenarios are recorded for each object class. For each of the developed architectures,accuracy estimates were obtained, and a comparative analysis was made.

Efficient and portable GEMM-based convolution operators for deep neural network training on multicore processors

Convolutional Neural Networks (CNNs) play a crucial role in many image recognition and classification tasks, recommender systems, brain-computer interfaces, etc. As a consequence, there is a notable interest in developing high performance realizations of the convolution operators, which concentrate a significant portion of the computational cost of this type of neural networks.In a previous work, we introduced a portable, high performance convolution algorithm, based on the BLIS realization of matrix multiplication, which eliminates most of the runtime and memory overheads that impair the performance of the convolution operators appearing in the forward training pass, when performed via explicit im2col transform. In this paper, we extend our ideas to the full training process of CNNs on multicore processors, proposing new high performance strategies to tackle the convolution operators that are present in the more complex backward pass of the training process, while maintaining the portability of the realizations. In addition, we conduct a full integration of these algorithms into a framework for distributed training of CNNs on clusters of computers, providing a complete experimental evaluation of the actual benefits in terms of both performance and memory consumption. Compared with baseline implementation, the use of the new convolution operators using pre-allocated memory can accelerate the training by a factor of about 6%–25%, provided there is sufficient memory available. In comparison, the operator variants that do not rely on persistent memory can save up to 70% of memory.

Medical image segmentation using scalable functional variational Bayesian neural networks with Gaussian processes

Bayesian neural networks (BNNs) are widely used in medical image segmentation tasks because they provide a probabilistic view of deep learning models by placing a prior distribution over the model parameters, and allow the representation of uncertainties through posterior distribution over the model parameters. However, it is difficult to choose meaningful prior distributions and fit accurate posterior distributions in a high-dimensional weight space, which limit the practical effectiveness of BNNs. In this study, we propose a scalable functional variational BNN with Gaussian processes (GPs) for the purpose of medical image segmentation. Specifically, we regard the prior and variational posterior distributions as GPs, and perform variational inference in function space. We employ a variant of the functional evidence lower bound (fELBO) with a β-weight on the Kullback–Leibler divergence term as the loss objective, and design a content-aware UNet segmentation network, which utilizes content-aware reassembly of features (CARAFE) as an upsampling operator to extract semantic information from input feature maps. The proposed BNN framework enables efficient training and can perform predictive inference with one forward pass. Extensive experiments are conducted on four segmentation datasets; the experimental results demonstrate that the proposed approach improves the segmentation performance metrics and uncertainty estimates compared with several state-of-the-art methods.

Enhancement of Super Sort Sorting Algorithm Using Identical Value Search Concept

The Super Sort Sorting algorithm is a sorting algorithm that uses forward selection, backward selection, merging, and partition to sort a given list or data set. This study will focus on the enhancement of the Super Sort Sorting Algorithm on the forward selection part. This paper introduces an enhanced version of the algorithm where the concept of Identical Value Search was implemented on the first forward pass. This method eliminated the small sublists left behind by the original algorithm's forward pass and reduced the overall runtime of the algorithm.

Video Regeneration and Quality Enhancer using GFP-GAN

Blind face restoration usually relies on facial priors, such as facial geometry prior or reference prior, to restore realistic and faithful details. However, very low-quality inputs cannot offer accurate geometric prior while high-quality references are inaccessible, limiting the applicability in real-world scenarios. In this work, we propose GFP-GAN that leverages rich and diverse priors encapsulated in a pretrained face GAN for blind face restoration. This Generative Facial Prior (GFP) is incorporated into the face restoration process via spatial feature transform layers, which allow our method to achieve a good balance of realness and fidelity. Thanks to the powerful generative facial prior and delicate designs, our GFP-GAN could jointly restore facial details and enhance colors with just a single forward pass, while GAN inversion methods require image-specific optimization at inference. Extensive experiments show that our method achieves superior performance to prior art on both synthetic and real-world datasets.

A Novel Approach to Uncertainty Quantification in Groundwater Table Modeling by Automated Predictive Deep Learning

Uncertainty quantification (UQ) is an important benchmark to assess the performance of artificial intelligence (AI) and particularly deep learning ensembled-based models. However, the ability for UQ using current AI-based methods is not only limited in terms of computational resources but it also requires changes to topology and optimization processes, as well as multiple performances to monitor model instabilities. From both geo-engineering and societal perspectives, a predictive groundwater table (GWT) model presents an important challenge, where a lack of UQ limits the validity of findings and may undermine science-based decisions. To overcome and address these limitations, a novel ensemble, an automated random deactivating connective weights approach (ARDCW), is presented and applied to retrieved geographical locations of GWT data from a geo-engineering project in Stockholm, Sweden. In this approach, the UQ was achieved via a combination of several derived ensembles from a fixed optimum topology subjected to randomly switched off weights, which allow predictability with one forward pass. The process was developed and programmed to provide trackable performance in a specific task and access to a wide variety of different internal characteristics and libraries. A comparison of performance with Monte Carlo dropout and quantile regression using computer vision and control task metrics showed significant progress in the ARDCW. This approach does not require changes in the optimization process and can be applied to already trained topologies in a way that outperforms other models.

Analysis and Investigation of Trilateral Spinning Based on the Concentric Circle Trajectory

Several researchers have developed various spinning methods to produce non-circular hollow parts. In this paper, an approach that the rollers revolve around the roller principal axis was used to generate a concentric circular trajectory to the spinning trilateral workpiece. The spinning simulation was performed in ABAQUS 2016/Explicit software. The trajectory of the roller is derived, and the distributions of the equivalent von Mises stress, equivalent strain, wall thickness of the workpiece in typical section and spinning force are obtained. The simulation experiment proves the feasibility of using the proposed spinning method to produce workpieces with trilateral cross-section. The results demonstrated that after seven passes, in the area where the roller unloads the workpiece and the area where the roller starts the forward pass, local wrinkling may be generated; the most vulnerable area of the workpiece is also located at the area where the roller starts the forward pass, while the overall thinning rate of the workpiece is below 21%. Compared with other existent non-circular spinning methods, this novel method could process non-circular parts with a high rotational velocity, while ensuring the roller processing trajectory with high accuracy. The application of this method will improve the efficiency of the process by more than 200% in theory, while ensuring workpiece quality

Operator compression with deep neural networks

This paper studies the compression of partial differential operators using neural networks. We consider a family of operators, parameterized by a potentially high-dimensional space of coefficients that may vary on a large range of scales. Based on the existing methods that compress such a multiscale operator to a finite-dimensional sparse surrogate model on a given target scale, we propose to directly approximate the coefficient-to-surrogate map with a neural network. We emulate local assembly structures of the surrogates and thus only require a moderately sized network that can be trained efficiently in an offline phase. This enables large compression ratios and the online computation of a surrogate based on simple forward passes through the network is substantially accelerated compared to classical numerical upscaling approaches. We apply the abstract framework to a family of prototypical second-order elliptic heterogeneous diffusion operators as a demonstrating example.