Fully-connected Neural Network Research Articles

Hamiltonian learning with real-space impurity tomography in topological moiré superconductors

Extracting Hamiltonian parameters from available experimental data is a challenge in quantum materials. In particular, real-space spectroscopy methods such as scanning tunneling spectroscopy allow probing electronic states with atomic resolution, yet even in those instances extracting the effective Hamiltonian is an open challenge. Here we show that impurity states in modulated systems provide a promising approach to extracting non-trivial Hamiltonian parameters of a quantum material. We show that by combining the real-space spectroscopy of different impurity locations in a moiré topological superconductor, modulations of exchange and superconducting parameters can be inferred via machine learning. We demonstrate our strategy with a physically-inspired harmonic expansion combined with a fully-connected neural network that we benchmark against a conventional convolutional architecture. We show that while both approaches allow extracting exchange modulations, only the former approach allows inferring the features of the superconducting order. Our results demonstrate the potential of machine learning methods to extract Hamiltonian parameters by real-space impurity spectroscopy as local probes of a topological state.

A Deep Learning Model for Detecting Rhegmatogenous Retinal Detachment Using Ophthalmologic Ultrasound Images

Introduction: Rhegmatogenous retinal detachment (RRD) is one of the most common fundus diseases. Many rural areas of China have few ophthalmologists, and ophthalmologic ultrasound examination is of great significance for remote diagnosis of RRD. Therefore, this study aimed to develop and evaluate a deep learning (DL) model, to be used for automated RRD diagnosis based on ophthalmologic ultrasound images, in order to support timely diagnosis of RRD in rural and remote areas. Methods: A total of 6,000 ophthalmologic ultrasound images from 1,645 participants were used to train and verify the DL model. A total of 5,000 images were used for training and validating DL models, and an independent testing set of 1,000 images was used to test the performance of eight DL models trained using four different DL model architectures (fully connected neural network, LeNet5, AlexNet, and VGG16) and two preprocessing techniques (original, original image augmented). Receiver operating characteristic (ROC) curves were used to analyze their performance. Heatmaps were generated to visualize the process of the best DL model in the identification of RRD. Finally, five ophthalmologists were invited to diagnose RRD independently on the same test set of 1,000 images for performance comparison with the best DL model. Results: The best DL model for identifying RRD achieved an area under the ROC curve (AUC) of 0.998 with a sensitivity and specificity of 99.2% and 99.8%, respectively. The best preprocessing method in each model architecture was the application of original image augmentation (average AUC = 0.982). The best model architecture in each preprocessing method was VGG16 (average AUC = 0.998). Conclusion: The best DL model determined in this study has higher accuracy, sensitivity, and specificity than the ophthalmologists’ diagnosis in identifying RRD based on ophthalmologic ultrasound images. This model may provide support for timely diagnosis in locations without access to ophthalmologic care.

BioEGRE: a linguistic topology enhanced method for biomedical relation extraction based on BioELECTRA and graph pointer neural network

BackgroundAutomatic and accurate extraction of diverse biomedical relations from literature is a crucial component of bio-medical text mining. Currently, stacking various classification networks on pre-trained language models to perform fine-tuning is a common framework to end-to-end solve the biomedical relation extraction (BioRE) problem. However, the sequence-based pre-trained language models underutilize the graphical topology of language to some extent. In addition, sequence-oriented deep neural networks have limitations in processing graphical features.ResultsIn this paper, we propose a novel method for sentence-level BioRE task, BioEGRE (BioELECTRA and Graph pointer neural net-work for Relation Extraction), aimed at leveraging the linguistic topological features. First, the biomedical literature is preprocessed to retain sentences involving pre-defined entity pairs. Secondly, SciSpaCy is employed to conduct dependency parsing; sentences are modeled as graphs based on the parsing results; BioELECTRA is utilized to generate token-level representations, which are modeled as attributes of nodes in the sentence graphs; a graph pointer neural network layer is employed to select the most relevant multi-hop neighbors to optimize representations; a fully-connected neural network layer is employed to generate the sentence-level representation. Finally, the Softmax function is employed to calculate the probabilities. Our proposed method is evaluated on three BioRE tasks: a multi-class (CHEMPROT) and two binary tasks (GAD and EU-ADR). The results show that our method achieves F1-scores of 79.97% (CHEMPROT), 83.31% (GAD), and 83.51% (EU-ADR), surpassing the performance of existing state-of-the-art models.ConclusionThe experimental results on 3 biomedical benchmark datasets demonstrate the effectiveness and generalization of BioEGRE, which indicates that linguistic topology and a graph pointer neural network layer explicitly improve performance for BioRE tasks.

Quartet: A Query Aware Database Adaptive Compilation Decision System

The executor is an important component of a database. Typical executors that are applied in modern database systems follow either the VOLCANO model or Compiled model, each of which fits some scenarios but not all. Even the widely employed PostgreSQL (PGSQL) and CockroachDB (CRDB) have to rely on human experts to achieve the optimal execution mode. Nevertheless, the accuracy of these decisions is only 32.8% on average, even with an expert involved. Moreover, due to the exclusive use of rule-based strategies, it is not feasible to reasonably switch between two working modes when confronted with different queries. To solve this problem, we propose a QUery awARe daTabase adaptivE compilaTion decision system (Quartet), which can determine the most suitable execution mode with respect to the current workload at runtime. Quartet generates operator-cost and tree-based vectors by analysing the query execution plan (QEP) and then uses the fully connected neural network (FCNN) and tree-based convolutional neural network (TBCNN) to learn the relationship between the QEP and the optimal execution. Our evaluations show that Quartet can improve execution decision accuracy by 60% on average under TPC-H (under 3 GB) workloads.

Inverse design of optical cages by using a fully connected neural network and time-reversed dipole radiation

In the conventional scheme, the focal field is designed by selecting appropriate dipole parameters based on theory of the time-revered dipole radiation. The time-revered method relies on empirical judgment and iterative parameter testing to create only a few special optical cages, involving vast time-consuming global parameter scanning and prescreening. In this work, a combination of a time-reversed dipole radiation model and a fully connected neural networks (FCNN) is introduced as the inverse design of optical cages. Flexibility and little time consumption in the design of the light cage are the two pivotal advantages. The dual light needle focusing fields with varying lengths and the synthesis of light cages featuring bubble focusing fields with distinct characteristics are attained, which are prescribed focal structures with freedom. With the help of the FCNN, the required dipole parameters for generating the expected focal fields are obtained quickly. This approach opens up possibilities for the utilization of neural networks in the design and modulation of more structured focusing fields. The achieved optical cage holds great potential for diverse applications, such as multiparticle trapping, manipulation, assembly, and enhanced nanoscale optical operations.

Development of an improved deep network model as a general technique for thin film nanocomposite reverse osmosis membrane simulation

Due to the complexity of the inherent trade-off between water permeability and salt rejection in thin film nanocomposite reverse osmosis membranes (TFN-ROMs), conventional methods have not performed satisfactorily in predicting the performance of TFN-ROMs. In this study, to address the limitations of traditional machine learning in predicting the performance of TFN-ROMs, we developed a three-component residual artificial neural network (R-TNN) and learned from membrane properties, nanoparticle properties, and membrane performance to capture the complex trade-off between water permeability and salt rejection of TFN-ROMs. The coefficients of determination of the trained R-TNN model for relative salt passage (RSP) and relative water permeability (RWP) of TFN-ROMs were 0.910 ± 0.028 and 0.739 ± 0.028, respectively, which showed good performance compared to fully-connected neural network and conventional machine learning models even with small sample data. Meanwhile, subsequent ablation experiments conducted verified that the improvement of the R-TNN model is practical and effective. In addition, we conducted data scale experiments, which showed that the R-TNN model has the potential to become a generalized technical framework for the initial informationization era and the future big data era. Subsequently, we employed the partial dependence plot to mechanistically interpret the model. Specifically, we observed a positive correlation between nanoparticle size and both RWP and RSP, while the loading of nanoparticles initially promoted and later inhibited these performances. The validated R-TNN model can better learn the inherent trade-off between salt rejection and water permeability of TFN-ROMs, which implies that this study can provide decision support for the fabrication of membranes with more outstanding performances, which greatly contributes to the promising applications of TFN-ROMs.

Robust optimization of geoenergy production using data-driven deep recurrent auto-encoder and fully-connected neural network proxy

Robust and efficient optimization of post-history well production schedule under history-matched geomodel known as closed-loop production management is crucial to achieve reasonable and reliable decision-making for subsurface geoenergy resources (e.g., oil/gas resources, geothermal, natural gas hydrate, etc.) recovery in petroleum industry. The procedure generally consists of a cycling implementation of geomodel history matching and production schedule optimization through repeatedly simulating high-fidelity reservoir models and therefore needs almost infeasible computation cost particularly for field-scale oil production scenario. Through hybridizing deep recurrent neural network and particle-swarm optimization algorithm, we present a novel data-driven computation framework for robust post-history production optimization without explicitly performing history matching. In the proposed framework, we develop a hybrid neural network proxy model, which combines recurrent autoencoder (RAE) and fully-connected neural network (FCNN), referred to as RAE-FCNN. The RAE structure can extract latent information from time-series history data, and the FCNN structure can map the combination of extracted latent variables by RAE and post-history well-control sequence to net present value (NPV). The trained RAE-FCNN proxy model can predict NPV values corresponding to the history measurements and user-specific post-history well controls. In the robust optimization framework, the averaged NPV over an ensemble of noisy history measurements are regarded as the objective function while the post-history well-controls are chosen as the decision variables. The proposed surrogate-based optimization framework has been demonstrated on two waterflooding reservoir production cases, e.g., a 2D synthetic Gaussian model and a 3D benchmaRk channelized EGG model.The decision variables considered in optimization are the injection rates with linear bound constraints at the injectors. In comparison with the optimization results by high-fidelity reservoir models, solely 1000 high-fidelity reservoir models for both cases are sufficient to train accurate proxy model and achieve satisfactory post-history production optimization results. Our proposed deep-learning-based optimization method can effectively make full use of history data to predict post-history oil well production without the need of computationally expensive history matching step, which enables us to achieve quick decision-making and risk evaluation for close-loop waterflooding reservoir production management and optimization.

Cross-section geometry prediction for laser metal deposition layer-based on multi-mode convolutional neural network and multi-sensor data fusion

Laser metal deposition (LMD), a widely used additive manufacturing technology for materials, can produce complex parts with excellent mechanical properties. However, it suffers from geometric defects such as size drift and edge collapse. Real-time monitoring plays a vital role in improving the quality of deposited layers. This paper proposed an online monitoring method for LMD based on multi-sensor data fusion and a multi-mode convolutional neural network (M-CNN) for predicting the deposited layer size. To build the LMD dataset for training and validating the M-CNN, an experimental method was devised to acquire experimental data via a monitoring system. The M-CNN, consisting of Resnet18 and a fully connected neural network (FCNN), used deposition images, temperature and process parameters as inputs to predict the deposited layer sizes (including width, height and cross-sectional area). It established the relationship between the deposited size and process parameters. Batch analysis was performed to obtain the sensitivity of deposited size to the process parameters and the dependency between deposited sizes, facilitating the rapid selection of process windows. The learning process and prediction results of the neural network were visualized and analyzed. The model accuracy was verified by comparing with the baseline. This work can be used for developing tuning methods for LMD.

Emulation of Pavlovian conditioning and pattern recognition through fully connected neural networks using Holmium oxide (Ho2O3) based synaptic RRAM device.

In this manuscript, we report on the paramagnetic Ho2O3 -based synaptic resistive random-access memory (RRAM) device for the implementation of neuronal functionalities such as long-term potentiation (LTP), long-term depression (LTD) and spike timing dependent plasticity (STDP) respectively. The plasticity of the artificial synapse is also studied by varying pulse amplitude, pulse width, and pulse interval. In addition, we could classify handwritten Modified National Institute of Standards and Technology (MNIST) data set using a fully connected neural network (FCN). The device-based FCN records a high classification accuracy of 93.47% which is comparable to the software-based test accuracy of 97.97%. This indicates the highly optimized behavior of our synaptic device for hardware neuromorphic applications. Successful emulation of Pavlovian classical conditioning for associative learning of the biological brain is achieved. We believe that the present device consists the potential to utilize in neuromorphic applications.

Predicting the Outcome of Patients with Aneurysmal Subarachnoid Hemorrhage: A Machine-Learning-Guided Scorecard.

Aneurysmal subarachnoid hemorrhage (aSAH) frequently causes long-term disability, but predicting outcomes remains challenging. Routine parameters such as demographics, admission status, CT findings, and blood tests can be used to predict aSAH outcomes. The aim of this study was to compare the performance of traditional logistic regression with several machine learning algorithms using readily available indicators and to generate a practical prognostic scorecard based on machine learning. Eighteen routinely available indicators were collected as outcome predictors for individuals with aSAH. Logistic regression (LR), random forest (RF), support vector machines (SVMs), and fully connected neural networks (FCNNs) were compared. A scorecard system was established based on predictor weights. The results show that machine learning models and a scorecard achieved 0.75~0.8 area under the curve (AUC) predicting aSAH outcomes (LR 0.739, RF 0.749, SVM 0.762~0.793, scorecard 0.794). FCNNs performed best (~0.95) but lacked interpretability. The scorecard model used only five factors, generating a clinically useful tool with a total cutoff score of ≥5, indicating poor prognosis. We developed and validated machine learning models proven to predict outcomes more accurately in individuals with aSAH. The parameters found to be the most strongly predictive of outcomes were NLR, lymphocyte count, monocyte count, hypertension status, and SEBES. The scorecard system provides a simplified means of applying predictive analytics at the bedside using a few key indicators.

NIMG-39. BENCHMARKING MULTI-LABEL PREDICTION OF TARGET GENE MUTATION STATUS IN GLIOBLASTOMA USING RADIOGENOMICS AND MULTI-PARAMETRIC MR IMAGING

Abstract PURPOSE In vivo imaging-based prediction of gene mutation status can significantly contribute to the diagnosis and treatment of glioblastoma. Radiogenomics provides a non-invasive approach to capturing the phenotypical characteristics of molecular alterations. We build machine learning (ML) models on radiogenomics and MR imaging data for benchmarking the prediction of multiple target genes in a single institution dataset. METHODS Multi-parametric MRI scans (T1, T1-Gd, T2, T2-FLAIR, DSC, DTI) of 286 individuals with glioblastoma were used to extract radiomics features, including histograms, morphological and textural descriptors. Genetic markers were obtained through a targeted next generation sequencing panel. We built ML models to predict mutation status in two settings: (a) 4 frequently-occurring mutations (EGFR, PTEN, NF1, TP53), (b) 13 mutations including some that are infrequent (PDGFRA, MET, FGFR2, PIK3CA, PIK3R1, BRAF, MDM4, CDKN2A). The models include k-nearest neighbor classifiers, fully-connected neural networks w/o dimensionality reduction trained on radiomics data, convolutional (CNN) and self-attention-based (ViT) networks trained on MR imaging data. All models were trained with 5-fold cross validation on discovery cohort (n=257) and tested on replication cohort (n=29). RESULTS Multi-label prediction models achieved highest accuracies of 75.9% / 88.6% for 4 / 13 mutations using radiomics features. Models trained with radiomics features have a marginally better average F1 score than CNNs and ViTs trained on MR imaging data (by 4.1% / 4.6% for 4 / 13 mutations). CONCLUSION We had modest success in finding highly distinct MR signatures of multiple target mutations in glioblastoma using multi-label ML methods, presumably because of the small sample size and the molecular, the spatial heterogeneity of the tumor, and the difficulty of simultaneously learning multiple imaging signatures. This highlights the importance of pooling and harmonizing radiogenomics data from multiple institutions to increase the sample size, and of incorporating biological domain knowledge into ML models.

Estimating air methane and total hydrocarbon concentrations in Alberta, Canada using machine learning

Fugitive emission sources are significant contributors to methane emissions, and time series data on reported emissions from such sources remain underutilized. The Alberta Energy Regulator (AER) has been collecting air quality data since 1986, including methane and total hydrocarbons concentration data. However, this data has not been thoroughly analyzed to forecast air quality trends. Our analysis of the data shows that average methane concentrations measured at most Alberta airshed stations exceed the global average, and the data exhibits increasing and decreasing trends depending on the station. We compared the predictive performance of three machine learning methods: Long Short-Term Memory (LSTM) recurrent neural network, Fully-Connected Neural Network (FC-NN), and Autoregressive Integrated Moving Average (ARIMA), using the AER methane concentration data. Our results indicate that the LSTM neural network model outperforms the other two methods. Also, our findings suggest that the AER methane concentration data can be effectively analyzed and utilized to forecast air quality trends in the region.

Prediction of departure delays at original stations using deep learning approaches: A combination of route conflicts and rolling stock connections

Rolling stock connections and route conflicts of trains at terminal stations are critical to train delay propagation and prediction on the railway networks. Previous studies primarily focused on delay prediction/propagation from a macro perspective, without considering the two factors. To address this problem, a hybrid neural network architecture, called TLF-net, is proposed to predict departure delays at original stations (DDOSs), considering the rolling-stock connection and potential route conflicts in the railway network. TLF-net consists of a transformer, a long short-term memory (LSTM), and a fully-connected neural network (FCNN) block, to separately address variables with different characteristics. Based on the real-world data from two terminal stations in China, the experimental results show that the consideration of the potential route conflicts from the network can considerably improve TLF-net's prediction performance over the model that only considering train interactions on a single railway line. Also, it is proven that arrival/departure routes of consecutive trains are crucial for delay prediction, while using the transformer block can efficiently reveal the route conflict severity between arrival/departure routes. The sensitivity analysis to influence factors demonstrates the significance of considering the rolling stock connection and potential route conflicts. Finally, to support real-time railway traffic management, a train arrival delay prediction model from our previous study is integrated to predict the input of TLF-net (i.e., train arrival delays), enabling the proposed model to dynamically predict the DDOSs. This dynamic updating lengthens the prediction horizon (the prediction time ahead), making it better support real-time train traffic control and management.

Using Machine Learning to Predict Window Opening Position in a Naturally Ventilated Building

Advancements in machine learning have faciliated its use in many domains. In this work we apply it to building sector, where mechanical ventilation systems are prevalent. While natural ventilation still can be suitable in many situations, the difficulty in estimating airflows and long computational simulation times prevents its adoption. Since ventilation rate depends heavily on window opening angle, we employ a computer vision techniques to estimate the states. We train a Fully-Connected Neural Network on images of European-style tilt-and-turn windows set at discrete positions, achieving over 95% average F1-Score. We highlight potential drawbacks with the method and identify steps forward on the path to real-world implementation.

Predicting houses price by deep learning neural network

Deep learning is widely used in various fields, the article proposed a deep learning method to predict house prices through different characteristics of real estate, establish a prediction model, and carry out simulation experiments. First, extracting data from property transactions records, it is difficult to directly input the raw-data into the deep learning model, and there may be overfitting in the model training, so data will be pretreated. Second, Fully-Connected Neural Network is used to model different features influences to price. The sample data will be randomly divided into a training set and a validation set, of which 70% of the samples are used for building and training the model, and the remaining 30% are used for model accuracy verification. Experimental results show that the model can achieve a high accuracy in predicting houses price. The model can be used as a reference for the evaluation of housing prices.

Interpretation and characterization of rate of penetration intelligent prediction model

Accurate prediction of the rate of penetration (ROP) is significant for drilling optimization. While the intelligent ROP prediction model based on fully connected neural networks (FNN) outperforms traditional ROP equations and machine learning algorithms, its lack of interpretability undermines its credibility. This study proposes a novel interpretation and characterization method for the FNN ROP prediction model using the Rectified Linear Unit (ReLU) activation function. By leveraging the derivative of the ReLU function, the FNN function calculation process is transformed into vector operations. The FNN model is linearly characterized through further simplification, enabling its interpretation and analysis. The proposed method is applied in ROP prediction scenarios using drilling data from three vertical wells in the Tarim Oilfield. The results demonstrate that the FNN ROP prediction model with ReLU as the activation function performs exceptionally well. The relative activation frequency curve of hidden layer neurons aids in analyzing the overfitting of the FNN ROP model and determining drilling data similarity. In the well sections with similar drilling data, averaging the weight parameters enables linear characterization of the FNN ROP prediction model, leading to the establishment of a corresponding linear representation equation. Furthermore, the quantitative analysis of each feature's influence on ROP facilitates the proposal of drilling parameter optimization schemes for the current well section. The established linear characterization equation exhibits high precision, strong stability, and adaptability through the application and validation across multiple well sections.

Demonstration of sub-micron UCN position resolution using room-temperature CMOS sensor

High spatial resolution of ultracold neutron (UCN) measurement is of growing interest to UCN experiments such as UCN spectrometers, UCN polarimeters, quantum physics of UCNs, and quantum gravity. Here we utilize physics-informed deep learning to enhance the experimental position resolution and to demonstrate sub-micron spatial resolutions for UCN position measurements obtained using a room-temperature CMOS sensor, extending our previous work (Kuk et al., 2021; Yue et al., 2023) that demonstrated a position uncertainty of 1.5μm. We explore the use of the open-source software Allpix Squared to generate experiment-like synthetic hit images with ground-truth position labels. We use physics-informed deep learning by training a fully-connected neural network (FCNN) to learn a mapping from input hit images to output hit position. The automated analysis for sub-micron position resolution in UCN detection combined with the fast data rates of current and next generation UCN sources will enable improved precision for future UCN research and applications.

Detecting Cybersecurity Threats using AI Network

Finding a method to quickly and efficiently identify online hazards is one of the most crucial issues in the world of cybersecurity. This study demonstrates a novel approach to identifying internet threats using artificial intelligence and artificial neural networks. The suggested method significantly increases the ability to identify cyber threats by breaking down a huge number of security events into individual event profiles and using a deep learning-based detection mechanism. AI-SIEM (Artificial Intelligence Security Information and Event Management) has been developed as a means of achieving this. This system integrates multiple artificial neural network methods, such as Fully Connected Neural Networks (FCNN), Convolutional Neural Networks (CNN), and Long Short-Term Memory (LSTM) networks, with event profiling for data preparation.

A search strategy for publications in interdisciplinary research

To retrieve the right collection of publications in interdisciplinary research, we have developed a search strategy with four progressive steps and take the area of public affairs (PA) as a case study. A set of seed publications in PA is first identified, followed by the construction of a pool set of publications with wider coverage for refinement in the next step, which is critical and in which an expanded set of publications is established on the basis of the references and text semantic information, thus generating two respective subsets. One of these subsets is obtained on the basis of the number of references shared between each publication pair between the seed set and the pool set. To optimize the results, we construct two models, viz. a support vector machine (SVM) and a fully connected neural network (FCNN), and find that the FCNN model outperforms the SVM model. The second subset of publications are collected by selecting the publications with high topic similarity to the seed publications collected in the first step. The final step is to integrate the seed publications with the expanded publications collected in steps 1 and 3. The results show that PA research involves an extremely wide range of disciplines (n = 45), among which public administration, environmental sciences, economics, management, and health policy and services, among others, play the most significant roles.

Naive Prediction of Protein Backbone Phi and Psi Dihedral Angles Using Deep Learning.

Protein structure prediction represents a significant challenge in the field of bioinformatics, with the prediction of protein structures using backbone dihedral angles recently achieving significant progress due to the rise of deep neural network research. However, there is a trend in protein structure prediction research to employ increasingly complex neural networks and contributions from multiple models. This study, on the other hand, explores how a single model transparently behaves using sequence data only and what can be expected from the predicted angles. To this end, the current paper presents data acquisition, deep learning model definition, and training toward the final protein backbone angle prediction. The method applies a simple fully connected neural network (FCNN) model that takes only the primary structure of the protein with a sliding window of size 21 as input to predict protein backbone ϕ and ψ dihedral angles. Despite its simplicity, the model shows surprising accuracy for the ϕ angle prediction and somewhat lower accuracy for the ψ angle prediction. Moreover, this study demonstrates that protein secondary structure prediction is also possible with simple neural networks that take in only the protein amino-acid residue sequence, but more complex models are required for higher accuracies.