Simple Network Architecture Research Articles

Kinematic Control of Manipulator with Remote Center of Motion Constraints Synthesised by a Simplified Recurrent Neural Network

Redundancy manipulators need favorable redundancy resolution to obtain suitable control actions to guarantee accurate kinematic control. Among numerous kinematic control applications, some specific tasks such as minimally invasive manipulation/surgery require the distal link of a manipulator to translate along such fixed point. Such a point is known as remote center of motion (RCM) to constrain motion planning and kinematic control of manipulators. Recurrent neural network (RNN) which possesses parallel processing ability, is a powerful alternative and has achieved success in conventional redundancy resolution and kinematic control with physical constraints of joint limits. However, up to now, there still is few related works on the RNNs for redundancy resolution and kinematic control of manipulators with RCM constraints considered yet. In this paper, for the first time, an RNN-based approach with a simplified neural network architecture is proposed to solve the redundancy resolution issue with RCM constraints, with a new and general dynamic optimization formulation containing the RCM constraints investigated. Theoretical results analyze and convergence properties of the proposed simplified RNN for redundancy resolution of manipulators with RCM constraints. Simulation results further demonstrate the efficiency of the proposed method in end-effector path tracking control under RCM constraints based on a redundant manipulator.

Wireless for Control: Over-the-Air Controller

In closed-loop wireless control systems, the state-of-the-art approach prescribes that a controller receives by wireless communications the individual sensor measurements, and then sends the computed control signal to the actuators. We propose an over-the-air controller scheme where all sensors attached to the plant simultaneously transmit scaled sensing signals directly to the actuator; then the feedback control signal is computed partially over the air and partially by a scaling operation at the actuator. Such over-the-air controller essentially adopts the over-the-air computation concept to compute the control signal for closed-loop wireless control systems. In contrast to the state-of-the-art sensor-to-controller and controller-to-actuator communication approach, the over-the-air controller exploits the superposition properties of multiple-access wireless channels to complete the communication and computation of a large number of sensing signals in a single communication resource unit. Therefore, the proposed scheme can obtain significant benefits in terms of low actuation delay and low wireless resource utilization by a simple network architecture that does not require a dedicated controller. Numerical results show that our proposed over-the-air controller achieves a huge widening of the stability region in terms of sampling time and delay, and a significant reduction of the computation error of the control signal.

Using neural networks to support high-quality evidence mapping

BackgroundThe Living Evidence Map Project at the Norwegian Institute of Public Health (NIPH) gives an updated overview of research results and publications. As part of NIPH’s mandate to inform evidence-based infection prevention, control and treatment, a large group of experts are continously monitoring, assessing, coding and summarising new COVID-19 publications. Screening tools, coding practice and workflow are incrementally improved, but remain largely manual.ResultsThis paper describes how deep learning methods have been employed to learn classification and coding from the steadily growing NIPH COVID-19 dashboard data, so as to aid manual classification, screening and preprocessing of the rapidly growing influx of new papers on the subject. Our main objective is to make manual screening scalable through semi-automation, while ensuring high-quality Evidence Map content.ConclusionsWe report early results on classifying publication topic and type from titles and abstracts, showing that even simple neural network architectures and text representations can yield acceptable performance.

A deep-learning-based workflow to deal with the defocusing problem in high-throughput experiments

The increasing throughput of experiments in biomaterials research makes automatic techniques more and more necessary. Among all the characterization methods, microscopy makes fundamental contributions to biomaterials science where precisely focused images are the basis of related research. Although automatic focusing has been widely applied in all kinds of microscopes, defocused images can still be acquired now and then due to factors including background noises of materials and mechanical errors. Herein, we present a deep-learning-based method for the automatic sorting and reconstruction of defocused cell images. First, the defocusing problem is illustrated on a high-throughput cell microarray. Then, a comprehensive dataset of phase-contrast images captured from varied conditions containing multiple cell types, magnifications, and substrate materials is prepared to establish and test our method. We obtain high accuracy of over 0.993 on the dataset using a simple network architecture that requires less than half of the training time compared with the classical ResNetV2 architecture. Moreover, the subcellular-level reconstruction of heavily defocused cell images is achieved with another architecture. The applicability of the established workflow in practice is finally demonstrated on the high-throughput cell microarray. The intelligent workflow does not require a priori knowledge of focusing algorithms, possessing widespread application value in cell experiments concerning high-throughput or time-lapse imaging.

Predicting LQ45 financial sector indices using RNN-LSTM

As one of the most popular financial market instruments, the stock has formed one of the most massive and complex financial markets in the world. It could handle millions of transactions within a short period of time and highly unpredictable. In this study, we aim to implement a famous Deep Learning method, namely the long short-term memory (LSTM) networks, for the stock price prediction. We limit the stocks to those that are included in the LQ45 financial sectors indices, i.e., BBCA, BBNI, BBRI, BBTN, BMRI, and BTPS. Rather than using too deep network architecture, we propose using a simple three-layer LSTM network architecture to predict the stocks’ closing prices. We found that the prediction results fall in the reasonable forecasting category. Moreover, it is worth noting that two of the considered stocks, namely, BBCA and BMRI, have the lowest MAPE values at 19.1020 and 18.6135, which fall in the good forecasting results. Hence, the proposed LSTM model is most recommended to be used on those two stocks.

DeepSperm: A robust and real-time bull sperm-cell detection in densely populated semen videos

Background and ObjectiveObject detection is a primary research interest in computer vision. Sperm-cell detection in a densely populated bull semen microscopic observation video presents challenges that are more difficult than those presented by other general object-detection cases. These challenges include partial occlusion, vast number of objects in a single video frame, tiny size of the object, artifacts, low contrast, low video resolution, and blurry objects because of the rapid movement of the sperm cells. This study proposes a deep neural network architecture, called DeepSperm, that solves the aforementioned problems and is more accurate and faster than state-of-the-art architectures. MethodsIn the proposed architecture, we use only one detection layer, which is specific for small object detection. For handling overfitting and increasing accuracy, we set a higher input network resolution, use a dropout layer, and perform data augmentation on saturation and exposure. Several hyper-parameters are tuned to achieve better performance. Mean average precision (mAP), confusion matrix, precision, recall, and F1-score are used to measure accuracy. Frame per second (fps) is used to measure speed. We compare our proposed method with you only look once (YOLO) v3 and YOLOv4. ResultsIn our experiment, we achieve 94.11 mAP on the test dataset, F1-score of 0.93, and a processing speed of 51.9 fps. In comparison with YOLOv4, our proposed method is 2.18 x faster on testing, and 2.9 x faster on training with a small dataset, while achieving comparative detection accuracy. The weights file size was also reduced significantly, with one-twentieth that of YOLOv4. Moreover, it requires a 1.07 x less graphical processing unit (GPU) memory than YOLOv4. ConclusionsThis study proposes DeepSperm, which is a simple, effective, and efficient deep neural network architecture with its hyper-parameters and configuration to detect bull sperm cells robustly in real time. In our experiments, we surpass the state-of-the-art in terms of accuracy, speed, and resource needs.

Multi-stage deep learning approaches to predict boarding behaviour of bus passengers

Smart card data has emerged in recent years and provide a comprehensive, and cheap source of information for planning and managing public transport systems. This paper presents a multi-stage machine learning framework to predict passengers’ boarding stops using smart card data. The framework addresses the challenges arising from the imbalanced nature of the data (e.g. many non-travelling data) and the ‘many-class’ issues (e.g. many possible boarding stops) by decomposing the prediction of hourly ridership into three stages: whether to travel or not in that one-hour time slot, which bus line to use, and at which stop to board. A simple neural network architecture, fully connected networks (FCN), and two deep learning architectures, recurrent neural networks (RNN) and long short-term memory networks (LSTM) are implemented. The proposed approach is applied to a real-life bus network. We show that the data imbalance has a profound impact on the accuracy of prediction at individual level. At aggregated level, FCN is able to accurately predict the rideship at individual stops, it is poor at capturing the temporal distribution of ridership. RNN and LSTM are able to measure the temporal distribution but lack the ability to capture the spatial distribution through bus lines.

Automated scoring of pre-REM sleep in mice with deep learning

Reliable automation of the labor-intensive manual task of scoring animal sleep can facilitate the analysis of long-term sleep studies. In recent years, deep-learning-based systems, which learn optimal features from the data, increased scoring accuracies for the classical sleep stages of Wake, REM, and Non-REM. Meanwhile, it has been recognized that the statistics of transitional stages such as pre-REM, found between Non-REM and REM, may hold additional insight into the physiology of sleep and are now under vivid investigation. We propose a classification system based on a simple neural network architecture that scores the classical stages as well as pre-REM sleep in mice. When restricted to the classical stages, the optimized network showed state-of-the-art classification performance with an out-of-sample F1 score of 0.95 in male C57BL/6J mice. When unrestricted, the network showed lower F1 scores on pre-REM (0.5) compared to the classical stages. The result is comparable to previous attempts to score transitional stages in other species such as transition sleep in rats or N1 sleep in humans. Nevertheless, we observed that the sequence of predictions including pre-REM typically transitioned from Non-REM to REM reflecting sleep dynamics observed by human scorers. Our findings provide further evidence for the difficulty of scoring transitional sleep stages, likely because such stages of sleep are under-represented in typical data sets or show large inter-scorer variability. We further provide our source code and an online platform to run predictions with our trained network.

Compressive Neural Representations of Volumetric Scalar Fields

AbstractWe present an approach for compressing volumetric scalar fields using implicit neural representations. Our approach represents a scalar field as a learned function, wherein a neural network maps a point in the domain to an output scalar value. By setting the number of weights of the neural network to be smaller than the input size, we achieve compressed representations of scalar fields, thus framing compression as a type of function approximation. Combined with carefully quantizing network weights, we show that this approach yields highly compact representations that outperform state‐of‐the‐art volume compression approaches. The conceptual simplicity of our approach enables a number of benefits, such as support for time‐varying scalar fields, optimizing to preserve spatial gradients, and random‐access field evaluation. We study the impact of network design choices on compression performance, highlighting how simple network architectures are effective for a broad range of volumes.

Simple and Effective Stochastic Neural Networks

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.

Spatiotemporal Saliency Representation Learning for Video Action Recognition

Deep convolutional neural networks (CNNs) have achieved great success in human action recognition, however they are still limited in understanding complex and noisy videos owing to the difficulties of exploiting appearance and motion information. Most existing works have been devoted to designing CNN architectures, which overlook the quality of network inputs that is of great importance. This paper provides an alternative solution of action recognition improvement by focusing on the quality of network inputs. A multi-task video salient object detection approach with object-of-interest segmentation scheme, which takes into account both human and action-relevant cues, is proposed to immunize the input video from background clutter. Further, a simple spatiotemporal residual network architecture is presented, which operates on multiple high-quality inputs for long-term action representation learning. Empirical evaluations on various challenging datasets demonstrate that the proposed framework can perform competitively against state-of-the-art. Besides better performance, learning representations of saliency can help prevent the action recognition model from overfitting and speed up the convergence of training.

Hybrid evolutionary network architecture search (HyENAS) for convolution class of deep neural networks with applications

AbstractConvolutional Neural Networks (CNNs) and its variants are increasingly used across wide domain of applications achieving high performance measures. For high performance, application specific CNN architecture is required, hence the need for network architecture search (NAS) becomes essential. This paper proposes a hybrid evolutionary approach for network architecture search (HyENAS), and targets convolution class of neural networks. One of the significant contribution of this technique is to completely evolve the high performance network by simultaneously finding network structures and their corresponding parameters. An elegant string representation has been proposed which efficiently represents the network. The concept of sparse block evolving requisite layer wise features for dense network is deployed. This permits the network to dynamically evolve for a specific application. In comparison to the other state‐of‐art methods, the high performance of the proposed HyENAS approach is demonstrated across various benchmark data sets belonging to the domain of malariology, oncology, neurology, ophthalmology, and genomics. Further, to deploy the proposed model on lower hardware specification devices, another salient feature of the HyENAS technique is to seamlessly sift out the simpler network architecture with comparable accuracy.

Efficient solution of linear inverse problems using an iterative linear neural network with a generalization training approach

Inverse problems are some of the most important mathematical problems in science and mathematics because their solution yields information about parameters that are not directly observable. Artificial neural networks have long been used as a mathematical modelling method and have been used successfully to solve inverse problems for application including denoising and medical image reconstruction. Many inverse problems result from integral processes that can be modelled using a linear formulation. These can be efficiently solved via simple networks which are easily trained with reasonable datasets. An innovative simple neural network architecture, the iterative linear neural network (ILNN), consisting of two non-hidden layer networks, one for the forward model and one for the inverse model, is proposed to solve linear inverse problems. Iteration between the two models refines network outcomes with greater accuracy than a network with only the inverse model. A training procedure accompanying the network is also introduced. The network needs to train only the inverse model with one-hot vectors as targets. The training inputs of the inverse model define the weights of the forward model. The number of targets is finite and equal to the length of the vector. With the defined targets, the training process ensures that the inverse model is at least a left inverse of the forward model. This leads to generalizable networks. The experimental results show that the ILNN produces good results even if its inverse model is not perfectly trained. The proposed network is applied to solve two linear inverse problems, deconvolution and the inverse Radon transform. The network successfully reconstructed original data following blurring and Radon transformation.

Single image rain streak removal via layer similarity prior

Single image rain streak removal is a significant and challenging task, which is widely applied in many artificial intelligence domains as preprocessing process. Most of existing rain streak removal works focus on designing various deraining unit (e.g., multi-stream dilation convolution) without considering the correlation between different convolution layers, which may lead to large model size. In this paper, we propose a simple and effective deep network architecture for single image rain streak removal based on deep Convolutional Neural Network (CNN). Benefit from the adjacent layers with different dilation factors have similar feature structures, we design a powerful rain streak representation network based on the Layer Similarity Prior Block (LSPB). To better cater to the property of layer similarity prior, the multi-dense-short-connection is developed and it regards every LSPB as a convolution layer, this connection style makes our overall framework to be a layer similarity prior network. To the best of our knowledge, this is the first paper to investigate the effectiveness of exploiting the layer similarity prior and the multi-dense-short-connection. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms other state-of-the-art methods with the least parameters.

Construction of a far-ultraviolet all-sky map from an incomplete survey: application of a deep learning algorithm

ABSTRACT We constructed a far-ultraviolet (FUV) all-sky map based on observations from the Far Ultraviolet Imaging Spectrograph (FIMS) aboard the Korean microsatellite Science and Technology SATellite-1. For the ${\sim}20{{\ \rm per\ cent}}$ of the sky not covered by FIMS observations, predictions from a deep artificial neural network were used. Seven data sets were chosen for input parameters, including five all-sky maps of H α, E(B − V), N(H i), and two X-ray bands, with Galactic longitudes and latitudes. 70 ${{\ \rm per\ cent}}$ of the pixels of the observed FIMS data set were randomly selected for training as target parameters and the remaining 30 ${{\ \rm per\ cent}}$ were used for validation. A simple four-layer neural network architecture, which consisted of three convolution layers and a dense layer at the end, was adopted, with an individual activation function for each convolution layer; each convolution layer was followed by a dropout layer. The predicted FUV intensities exhibited good agreement with Galaxy Evolution Explorer observations made in a similar FUV wavelength band for high Galactic latitudes. As a sample application of the constructed map, a dust scattering simulation was conducted with model optical parameters and a Galactic dust model for a region that included observed and predicted pixels. Overall, FUV intensities in the observed and predicted regions were reproduced well.

Image reconstruction through a multimode fiber with a simple neural network architecture

Multimode fibers (MMFs) have the potential to carry complex images for endoscopy and related applications, but decoding the complex speckle patterns produced by mode-mixing and modal dispersion in MMFs is a serious challenge. Several groups have recently shown that convolutional neural networks (CNNs) can be trained to perform high-fidelity MMF image reconstruction. We find that a considerably simpler neural network architecture, the single hidden layer dense neural network, performs at least as well as previously-used CNNs in terms of image reconstruction fidelity, and is superior in terms of training time and computing resources required. The trained networks can accurately reconstruct MMF images collected over a week after the cessation of the training set, with the dense network performing as well as the CNN over the entire period.

SNR-Based Features and Diverse Training Data for Robust DNN-Based Speech Enhancement

In this paper, we address the generalization of deep neural network (DNN) based speech enhancement to unseen noise conditions for the case that training data is limited in size and diversity. To gain more insights, we analyze the generalization with respect to (1) the size and diversity of the training data, (2) different network architectures, and (3) the chosen features. To address (1), we train networks on the Hu noise corpus (limited size), the CHiME 3 noise corpus (limited diversity) and also propose a large and diverse dataset collected based on freely available sounds. To address (2), we compare a fully-connected feed-forward and a long short-term memory (LSTM) architecture. To address (3), we compare three input features, namely logarithmized noisy periodograms, noise aware training (NAT) and the proposed signal-to-noise ratio (SNR) based noise aware training (SNR-NAT). We confirm that rich training data and improved network architectures help DNNs to generalize. Furthermore, we show via experimental results and an analysis using t-distributed stochastic neighbor embedding (t-SNE) that the proposed SNR-NAT features yield robust and level independent results in unseen noise even with simple network architectures and when trained on only small datasets, which is the key contribution of this paper.

HistoClean: Open-source software for histological image pre-processing and augmentation to improve development of robust convolutional neural networks

The growth of digital pathology over the past decade has opened new research pathways and insights in cancer prediction and prognosis. In particular, there has been a surge in deep learning and computer vision techniques to analyse digital images. Common practice in this area is to use image pre-processing and augmentation to prevent bias and overfitting, creating a more robust deep learning model. This generally requires consultation of documentation for multiple coding libraries, as well as trial and error to ensure that the techniques used on the images are appropriate. Herein we introduce HistoClean; a user-friendly, graphical user interface that brings together multiple image processing modules into one easy to use toolkit.HistoClean is an application that aims to help bridge the knowledge gap between pathologists, biomedical scientists and computer scientists by providing transparent image augmentation and pre-processing techniques which can be applied without prior coding knowledge.In this study, we utilise HistoClean to pre-process images for a simple convolutional neural network used to detect stromal maturity, improving the accuracy of the model at a tile, region of interest, and patient level. This study demonstrates how HistoClean can be used to improve a standard deep learning workflow via classical image augmentation and pre-processing techniques, even with a relatively simple convolutional neural network architecture. HistoClean is free and open-source and can be downloaded from the Github repository here: https://github.com/HistoCleanQUB/HistoClean.

Quasi-Periodic Parallel WaveGAN: A Non-Autoregressive Raw Waveform Generative Model With Pitch-Dependent Dilated Convolution Neural Network

In this paper, we propose a quasi-periodic parallel WaveGAN (QPPWG) waveform generative model, which applies a quasi-periodic (QP) structure to a parallel WaveGAN (PWG) model using pitch-dependent dilated convolution networks (PDCNNs). PWG is a small-footprint GAN-based raw waveform generative model, whose generation time is much faster than real time because of its compact model and non-autoregressive (non-AR) and non-causal mechanisms. Although PWG achieves high-fidelity speech generation, the generic and simple network architecture lacks pitch controllability for an unseen auxiliary fundamental frequency ($F_{0}$) feature such as a scaled $F_{0}$. To improve the pitch controllability and speech modeling capability, we apply a QP structure with PDCNNs to PWG, which introduces pitch information to the network by dynamically changing the network architecture corresponding to the auxiliary $F_{0}$ feature. Both objective and subjective experimental results show that QPPWG outperforms PWG when the auxiliary $F_{0}$ feature is scaled. Moreover, analyses of the intermediate outputs of QPPWG also show better tractability and interpretability of QPPWG, which respectively models spectral and excitation-like signals using the cascaded fixed and adaptive blocks of the QP structure.

Extending Attack Graphs to Represent Cyber-Attacks in Communication Protocols and Modern IT Networks

An attack graph is a method used to enumerate the possible paths that an attacker can take in the organizational network. MulVAL is a known open-source framework used to automatically generate attack graphs. MulVAL's default modeling has two main shortcomings. First, it lacks the ability to represent network protocol vulnerabilities, and thus it cannot be used to model common network attacks, such as ARP poisoning. Second, it does not support advanced types of communication, such as wireless and bus communication, and thus it cannot be used to model cyber-attacks on networks that include IoT devices or industrial components. In this article, we present an extended network security model for MulVAL that: (1) considers the physical network topology, (2) supports short-range communication protocols, (3) models vulnerabilities in the design of network protocols, and (4) models specific industrial communication architectures. Using the proposed extensions, we were able to model multiple attack techniques including: spoofing, man-in-the-middle, and denial of service attacks, as well as attacks on advanced types of communication. We demonstrate the proposed model in a testbed which implements a simplified network architecture comprised of both IT and industrial components.