Fully-connected Deep Neural Network Research Articles

On the benefit of attention in inverse design of thin films filters

Abstract Attention layers are a crucial component in many modern deep learning models, particularly those used in natural language processing and computer vision. Attention layers have been shown to improve the accuracy and effectiveness of various tasks, such as machine translation, image captioning, etc. Here, the benefit of attention layers in designing optical filters based on a stack of thin film materials is investigated. The superiority of Attention layers over fully-connected Deep Neural Networks is demonstrated for this task.

Topology Optimization using Nonlinear Finite Element Analysis

Topology optimization is a structural-mechanical investigation that structures optimization function values in an optimization iteration. Topology optimization with nonlinear static behavior is difficult due to several design considerations. Due of significant integrate distortion, low-density finite elements provide significant numerical challenges in the prevailing element density focused topology planning taking nonlinear dynamics under factor. Iterative procedures are used in the Bi-directional Evolutionary Structural Optimization (BESO) technique for reducing waste from a structure while concurrently adding efficient material using a finite element-based topology optimization strategy. Integrating the fully-connected Deep Neural Network (DNN) with the norm-level-set techniques yields a powerful method for optimizing structural topology. Hence, BESO-DNN has been designed spatial optimization of material distribution inside a specified area is the focus of topology optimization is a mathematical approach for minimizing a certain cost function while meeting a set of predefined restrictions. Topology optimization of geometrically non-linear systems is advantageous because of the solutions' lack of intermediate-density components and their great processing efficiency of high-resolution barrier representation can be effective. As a result, topology optimization is ensuring effective and providing fresh and efficient designs, understanding that outcomes needs an integration of credible decision-making, domain knowledge, and numerical analytic skills. The outcome of finite element analysis has to understand and anticipate an object's performance under various physical conditions using mathematical models, and testing.

Missing well-log reconstruction using a sequence self-attention deep-learning framework

Well logging is a critical tool for reservoir evaluation and fluid identification. However, due to borehole conditions, instrument failure, economic constraints, etc., some types of well logs are occasionally missing or unreliable. Existing logging curve reconstruction methods based on empirical formulas and fully connected deep neural networks (FCDNN) can only consider point-to-point mapping relationships. Recurrently structured neural networks can consider a multipoint correlation, but it is difficult to compute in parallel. To take into account the correlation between log sequences and achieve computational parallelism, we develop a novel deep-learning framework for missing well-log reconstruction based on state-of-the-art transformer architecture. The missing well-log transformer (MWLT) uses a self-attention mechanism instead of a circular recursive structure to model the global dependencies of the inputs and outputs. To use different usage requirements, we design the MWLT in three scales: small, base, and large, by adjusting the parameters in the network. A total of 8609 samples from 209 wells in the Sichuan Basin, China, are used for training and validation, and two additional blind wells are used for testing. The data augmentation strategy with random starting points is implemented to increase the robustness of the model. The results show that our proposed MWLT achieves a significant improvement in accuracy over the conventional Gardner’s equation and data-driven approaches such as FCDNN and bidirectional long short-term memory, on the validation data set and blind test wells. The MWLT-large and MWLT-base have lower prediction errors than MWLT-small but require more training time. Two wells in the Songliao Basin, China, are used to evaluate the cross-regional generalized performance of our method. The generalizability test results demonstrate that density logs reconstructed by MWLT remain the best match to the observed data compared with other methods. The parallelizable MWLT automatically learns the global dependence of the parameters of the subsurface reservoir, enabling an efficient missing well-log reconstruction performance.

Improved Bathymetry in the South China Sea from Multisource Gravity Field Elements Using Fully Connected Neural Network

Traditional bathymetry inversion methods that rely on an altimetry-derived gravity anomaly (GA) and/or a vertical gravity gradient anomaly (VGG) have been widely used for bathymetry prediction in the South China Sea. However, few studies attempt new methods to combine multisource gravity data to improve the accuracy of the bathymetry. In this study, we introduce a fully connected deep neural network (FC-DNN) to merge GA, VGG, and the deflection of vertical (DOV) to predict the bathymetry in the South China Sea. Single beam sounding depths were used as sample data for neural network training. Independent shipboard depths and GEBCO2023, topo_25.1, and ETOPO2022 models were applied as validation data. The assessment results showed that the FC-DNN model reached a high precision level with an STD of 49.20 m. More than 70% of the differences between the FC-DNN bathymetric model and other depth models were less than 100 m. Furthermore, the spectral analysis results showed that the FC-DNN bathymetry model has stronger energy in medium and short wavelengths than other models, which indicates that additional gravity field element DOVs can recover richer topographic signals in those particular bands.

Determining peanut moisture content by scattering coefficient

Moisture content is a fundamental parameter in peanut production and storage. Moisture content information is essential for crop analysis. Crop analysis is significant for production planning and improving yield. Thus, developing a moisture content measurement system with high precision and low cost can benefit peanut production, processing, and storage. To achieve accurate and rapid nondestructive testing of shelled peanuts and peanut pods, this study proposes a new method for measuring moisture content based on the microwave scattering coefficient. The microwave free-space method was used to measure the scattering parameters of peanut samples with different moisture contents (8.11–45.59% wet basis) at different temperatures and sample thicknesses. Moreover, a peanut moisture measurement model using S-parameters was developed. Using the amplitude, phase, and real part and imaginary part combinations of the S11 and S21 parameters of the peanut samples as the input and peanut moisture content as the output, moisture content and S-parameter prediction models based on the gradient boost regression trees (GBRT), extreme gradient boost (XGBoost), and fully connected deep neural network (FC-DNN) were established. Test results verified that the FC-DNN-based model had the best prediction performance, with a coefficient of determination (R2) = 0.9998, mean absolute error (MAE) = 0.0891, mean square error (MSE) = 0.0254, and root mean square error (RMSE) = 0.1593. This study provides a new method for the nondestructive measurement of moisture content during peanut production and storage. Moreover, the proposed model can be applied for moisture content measurement of other agricultural products in the food processing industry.

An Integrated Deep Learning Framework for Classification of Mineral Thin Sections and Other Geo-Data, a Tutorial

Recent studies have demonstrated the potential of machine learning methods for fast and accurate mineral classification based on microscope thin sections. Such methods can be extremely useful to support geoscientists during the phases of operational geology, especially when mineralogical and petrological data are fully integrated with other geological and geophysical information. In order to be effective, these methods require robust machine learning models trained on pre-labeled data. Furthermore, it is mandatory to optimize the hyper-parameters of the machine learning techniques in order to guarantee optimal classification accuracy and reliability. Nowadays, deep learning algorithms are widely applied for image analysis and automatic classification in a large range of Earth disciplines, including mineralogy, petrography, paleontology, well-log analysis, geophysical imaging, and so forth. The main reason for the recognized effectiveness of deep learning algorithms for image analysis is that they are able to quickly learn complex representations of images and patterns within them. Differently from traditional image-processing techniques based on handcrafted features, deep learning models automatically learn and extract features from the data, capturing, in almost real-time, complex relationships and patterns that are difficult to manually define. Many different types of deep learning models can be used for image analysis and classification, including fully connected deep neural networks (FCNNs), convolutional neural networks (CNNs or ConvNet), and residual networks (ResNets). In this paper, we compare some of these techniques and verify their effectiveness on the same dataset of mineralogical thin sections. We show that the different deep learning methods are all effective techniques in recognizing and classifying mineral images directly in the field, with ResNets outperforming the other techniques in terms of accuracy and precision. In addition, we compare the performance of deep learning techniques with different machine learning algorithms, including random forest, naive Bayes, adaptive boosting, support vector machine, and decision tree. Using quantitative performance indexes as well as confusion matrixes, we demonstrate that deep neural networks show generally better classification performances than the other approaches. Furthermore, we briefly discuss how to expand the same workflow to other types of images and geo-data, showing how this deep learning approach can be generalized to a multiscale/multipurpose methodology addressed to the analysis and automatic classification of multidisciplinary information. This article has tutorial purposes, too. For that reason, we will explain, with a didactical level of detail, all the key steps of the workflow.

Evaluation of a High-Accuracy Indoor-Positioning System with Wi-Fi Time of Flight (ToF) and Deep Learning

This article proposes the use of Wi-Fi ToF and a deep learning approach to build a cheap, practical, and highly-accurate IPS. To complement that, rather than using the classic geometrical approach (such as multilateration), it uses a more data-driven approach, i.e., the location fingerprinting technique. The fingerprint of a location, in this case, is a set of Wi-Fi ToFs between the target device and an access point (AP). Therefore, the number of APs in the area dictates the set size. The location fingerprinting technique requires a collection of fingerprints of various locations in the area to build a reference database or map. This database or map contains the information used to carry out the main task of the location fingerprinting technique, namely, estimating the position of a device based on its location fingerprint. For that task, we propose using a fully connected deep neural network (FCDNN) model to act as a positioning engine. The model is given a location fingerprint as its input to produce the estimated location coordinates as its output. We conduct an experiment to analyze the impact of the available AP pair in the dataset, from 1 unique AP pair, 2 AP pairs, and more, using WKNN and FCDNN to compare their performance. Our experimental results show that our IPS, DeepIndoor, can achieve an average positioning error or mean square error of 0.1749 m, and root mean square error of 0.5740 m in scenario 3, where 1–10 AP pairs or the raw dataset is used.

Reconstructing Three-dimensional geological structures by the Multiple-point statistics method coupled with a deep neural network: A case study of a metro station in Guangzhou, China

A credible large-scale three-dimensional (3D) geological model constructed from a small amount of local high-density data provides a vital data infrastructure for the design and construction of metro lines. The multiple-point statistics (MPS) method has successfully generated 3D geological models in many fields. However, the global cognition of the spatial relations between geological objects is difficult to reconstruct using a local optimization strategy with moving templates in the MPS simulation. This study presented an MPS simulation approach for constructing 3D geological structures with two-dimensional (2D) cross-sections, where the MPS process was coupled with a fully connected deep neural network (FCDNN) simulation. The core idea of the proposed method was that the global features of each geological object were generated from the trained FCDNN, the input parameter of which was the subsurface depth. Then, the entire model was constructed on the initial model using the following iterative MPS process: the semantic relationship of geological objects was extracted from 2D cross-sections and used as a constraint in the modeling process, which ensured the order of the stratigraphic sequence and rationality of the geological structure in the final model. The joint objective function of the proposed method was composed of the loss function of the FCDNN and conventional MPS objective function constrained by the spatial semantics of geological objects. The concrete example in the Jiangtai Road Metro Station of Guangzhou Line 11, China illustrated that the details of the complex geological structures could be obtained accurately by the presented method. The global features from the trained FCDNN provided the basic geometry for the final model, which overcame the limitations of the conventional MPS method.

Feature extraction and prediction of fine particulate matter (PM2.5) chemical constituents using four machine learning models

The concentrations of fine particulate matter (PM2.5) constituents, which are very important and essential information for the identification of air pollution sources, were predicted at three sites (Seoul, Ulsan, Baengnyeong) in South Korea between 2016 and 2018 using four machine learning (ML) models: generative adversarial imputation network (GAIN), fully connected deep neural network (FCDNN), random forest (RF), and k-nearest neighbor (kNN). 3 PM2.5 constituent groups, namely 8 ions, 2 carbons, and 15 trace elements, were targeted for prediction. The latest hyperparameter optimization techniques were used to learn air pollution characteristics from ambient PM2.5-related information, such as time, meteorology, and air pollutant concentrations. We compared the feature extraction abilities of the four models. The prediction accuracy identified by the coefficient of determination (R2) between prediction and observation was highest in GAIN, followed by FCDNN and RF or kNN. On availability of data on the time, air pollutant concentrations, and/or meteorology, simultaneously missed 20 % data of all PM2.5 constituent groups were predicted, with R2 = 0.897, 0.861, 0.785, and 0.801 by the GAIN, FCDNN, RF, and kNN, respectively. As missing ratios (20 %, 40 %, 60 %, 80 %) of input data increased, prediction accuracy decreased in the four models and was predominantly more noticeable in GAIN and kNN. As the available period of data increased, prediction accuracy increased in GAIN and FCDNN. Trace elements were predicted with the lowest R2 in all models among the target constituent groups. Study sites with more emission sources showed lower prediction accuracy, resulting in the highest R2 in Baengnyeong island and the lowest in Ulsan. According to the current findings, ML models can be used to evaluate various air pollution issues for which data is missing.

Speaker identification and localization using shuffled MFCC features and deep learning

The use of machine learning in automatic speaker identification and localization systems has recently seen significant advances. However, this progress comes at the cost of using complex models, computations, and increasing the number of microphone arrays and training data. Therefore, in this work, we propose a new end-to-end identification and localization model based on a simple fully connected deep neural network (FC-DNN) and just two input microphones. This model can jointly or separately localize and identify an active speaker with high accuracy in single and multi-speaker scenarios by exploiting a new data augmentation approach. In this regard, we propose using a novel Mel Frequency Cepstral Coefficients (MFCC) based feature called Shuffled MFCC (SHMFCC) and its variant Difference Shuffled MFCC (DSHMFCC). In order to test our approach, we analyzed the performance of the identification and localization proposed model on the new features at different noise and reverberation conditions for single and multi-speaker scenarios. The results show that our approach achieves high accuracy in these scenarios, outperforms the baseline and conventional methods, and achieves robustness even with small-sized training data.

Residential load forecasting based on electricity consumption pattern clustering

In order to reduce the peak load on the power grid, various types of demand response (DR) programs have been developed rapidly, and an increasing number of residents have participated in the DR. The change in residential electricity consumption behavior increases the randomness of electricity load power, which makes residential load forecasting relatively difficult. Aiming at increasing the accuracy of residential load forecasting, an innovative electricity consumption pattern clustering is implemented in this paper. Six categories of residential load are clustered considering the power consumption characteristics of high-energy-consuming equipment, using the entropy method and criteria importance though intercrieria correlation (CRITIC) method. Next, based on the clustering results, the residential load is predicted by the fully-connected deep neural network (FDNN). Compared with the prediction result without clustering, the method proposed in this paper improves the accuracy of the prediction by 5.21%, which is demonstrated in the simulation.

Data-driven deep learning for OTFS detection

Recently, orthogonal time frequency space (OTFS) was presented to alleviate severe Doppler effects in high mobility scenarios. Most of the current OTFS detection schemes rely on perfect channel state information (CSI). However, in real-life systems, the parameters of channels will constantly change, which are often difficult to capture and describe. In this paper, we summarize the existing research on OTFS detection based on data-driven deep learning (DL) and propose three new network structures. The presented three networks include a residual network (ResNet), a dense network (DenseNet), and a residual dense network (RDN) for OTFS detection. The detection schemes based on data-driven paradigms do not require a model that is easy to handle mathematically. Meanwhile, compared with the existing fully connected-deep neural network (FC-DNN) and standard convolutional neural network (CNN), these three new networks can alleviate the problems of gradient explosion and gradient disappearance. Through simulation, it is proved that RDN has the best performance among the three proposed schemes due to the combination of shallow and deep features. RDN can solve the issue of performance loss caused by the traditional network not fully utilizing all the hierarchical information.

A contradiction solving method for complex product conceptual design based on deep learning and technological evolution patterns

Contradictions caused by the various design constraints present increasing challenges to efficiency and innovation in product development. TRIZ provides Inventive Principles (IPs) and Contradiction Matrix that are the most frequently applied in conflict resolution. However, the high-level abstraction and subjective selection of IPs inhibit achieving the transformation process from paradoxical states to physical structures. To fill this gap, a contradiction solving method by integrating deep learning and technological evolution patterns for product conceptual design is proposed, which illustrates the mechanism of contradiction transition from the perspective of system evolution and supplies a systematic and model-based design approach. Firstly, generic engineering parameters are extracted to define the underlying contradictions transformed from critical defects which are found out through function modeling and root-conflict analysis. Then, a fully-connected deep neural network with excellent performance is developed to uncover the non-linear relationships between engineering parameters and evolution patterns. Finally, an evolution tree based on the predicted patterns is constructed to visualize transformation potentials of a technical system and help designers generate innovative specific solutions. In addition, a case study concerning design conflict resolution for beat-up system of three-dimensional tubular weaving machine is used to validate the adaptability and reliability of the proposed approach.

Data-Driven Soft Demapping for Residual Impairments Channels

Deep learning based solutions are being integrated into the physical and link layers of wireless networks. They often effect an improvement in transmission reliability and/or efficiency when there is a model or an algorithm deficit. In this letter, we propose a deep learning-aided soft demapper, consisting of a fully-connected deep neural network (DNN), to alleviate a channel model deficit. We apply it in microwave backhaul transmissions affected by impairments generated by the local oscillator and power amplifier. The proposed DNN soft demapper learns the best approximation for the log-likelihood ratios (LLRs). The learned LLRs show gains over model-based impairment-aware LLRs, as they capture the actual channel as observed through data. We implement weight pruning and periodical retraining to adapt to statistical changes and make our proposed approach fit for practical cost-aware applications.

Neural distortion fields for spatial calibration of wide field-of-view near-eye displays.

We propose a spatial calibration method for wide field-of-view (FoV) near-eye displays (NEDs) with complex image distortions. Image distortions in NEDs can destroy the reality of the virtual object and cause sickness. To achieve distortion-free images in NEDs, it is necessary to establish a pixel-by-pixel correspondence between the viewpoint and the displayed image. Designing compact and wide-FoV NEDs requires complex optical designs. In such designs, the displayed images are subject to gaze-contingent, non-linear geometric distortions, which explicit geometric models can be difficult to represent or computationally intensive to optimize. To solve these problems, we propose neural distortion field (NDF), a fully-connected deep neural network that implicitly represents display surfaces complexly distorted in spaces. NDF takes spatial position and gaze direction as input and outputs the display pixel coordinate and its intensity as perceived in the input gaze direction. We synthesize the distortion map from a novel viewpoint by querying points on the ray from the viewpoint and computing a weighted sum to project output display coordinates into an image. Experiments showed that NDF calibrates an augmented reality NED with 90° FoV with about 3.23 pixel (5.8 arcmin) median error using only 8 training viewpoints. Additionally, we confirmed that NDF calibrates more accurately than the non-linear polynomial fitting, especially around the center of the FoV.

Perturbation of deep autoencoder weights for model compression and classification of tabular data

Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, in tabular data classification, DNNs are challenged by the often superior performance of traditional machine learning models. This paper proposes periodic perturbations (prune and regrow) of DNN weights, especially at the self-supervised pre-training stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning or weight regularization (L1 or L2) for four out of six tabular data sets in downstream classification tasks. Unlike dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40% sparsity across six tabular data sets, resulting in compressed pretrained models. The proposed pretrained model compression improves the accuracy of downstream classification, unlike traditional weight pruning methods that trade off performance for model compression. Our experiments reveal that a pretrained deep autoencoder with weight perturbation can outperform traditional machine learning in tabular data classification, whereas baseline fully-connected DNNs yield the worst classification accuracy. However, traditional machine learning models are superior to any deep model when a tabular data set contains uncorrelated variables. Therefore, the performance of deep models with tabular data is contingent on the types and statistics of constituent variables.

Optimal and Fully Connected Deep Neural Networks Based Classification Model for Unmanned Aerial Vehicle Using Hyperspectral Remote Sensing Images

Unmanned Aerial Vehicle (UAV) is treated as an effective technique for gathering high resolution aerial images. The UAV based aerial image collection is highly preferred due to its inexpensive and effective nature. However, automatic classification of aerial images poses a major challenging issue in the design of UAV, which could be handled by the deep learning (DL) models. This study designs a novel UAV assisted DL based image classification model (UAVDL-ICM) for Industry 4.0 environment. The proposed UAVDL-ICM technique involves an ensemble of voting based three DL models, namely Residual network (ResNet), Inception with ResNetv2, and Densely Connected Networks (DenseNet). Also, the hyperparameter tuning of these DL models takes place using a genetic programming (GP) approach. Finally, Oppositional Water Wave Optimization (OWWO) with Fully Connected Deep Neural networks (FCDNN) is employed for the classification of aerial images. A wide range of simulations takes place and the results are examined in terms of different parameters. A detailed comparative study highlighted the betterment of the UAVDL-ICM technique compared to other recent approaches.

The Impact of Speaker Diarization on DNN-based Autism Severity Estimation.

This paper presents a speech-based system for autism severity estimation combined with automatic speaker diarization. Speaker diarization was performed by two different methods. The first used acoustic features, which included Mel-Frequency Cepstral Coefficients (MFCC) and pitch, and the second used x-vectors - embeddings extracted from Deep Neural Networks (DNN). The speaker diarization was trained using a Fully Connected Deep Neural Network (FCDNN) in both methods. We then trained a Convolutional Neural Network (CNN) to estimate the severity of autism based on 48 acoustic and prosodic features of speech. One hundred thirty-two young children were recorded in the Autism Diagnostic Observation Schedule (ADOS) examination room, using a distant microphone. Between the two diarization methods, the MFCC and Pitch achieved a better Diarization Error Rate (DER) of 26.91%. Using this diarization method, the severity estimation system achieved a correlation of 0.606 (Pearson) between the predicted and the actual autism severity scores (i.e., ADOS scores). Clinical Relevance- The presented system identifies children's speech segments and estimates their autism severity sc30:310ore.

Deep Learning Based Detection for Communications Systems With Radar Interference

Due to the increasing demand for spectrum resources, the co-existence of communications and radar systems has been proposed that allows radar and communications systems to operate in the same frequency band. On the other hand, deep learning has shown great potential in revolutionizing communications systems. In this work, we investigate the use of deep learning in communications systems subject to interference from radar systems. Specifically, we consider a single-carrier communications system. Linear frequency-modulated (LFM) and frequency-modulated continuous-wave (FMCW) are considered for radar. Several important system parameters, including the level of noise and interference, the radar interference coverage, the symbol duration, feature extraction methods and the number of hidden layers are investigated for the performance of the detector. Fully connected deep neural network (FCDNN) and long short-term memory (LSTM) detectors are implemented, where principle component analysis (PCA) is applied to preprocess the observed signals for the FCDNN detector. Numerical results show that the learning-based detector achieves comparable performance in the radar-communication system to the traditional detector but without interference cancellation. Preprocessing the received signals with PCA can improve the performance of FCDNN when interference is strong. Also, LSTM shows more robust performance than FCDNN when the channel has time-related distortion.

Predicting communication quality in construction projects: A fully-connected deep neural network approach

Establishing high-quality communication in construction projects is essential to securing successful collaboration and maintaining understanding among project stakeholders. Indeed, poor communication results in low productivity, poor efficiency, and substandard deliverables. While high-quality communication is recognized as contingent on the interpersonal skills of workers, the impacts of communication quality on job performance remain unknown. This study addresses this deficiency by developing a method to evaluate construction workers' communication quality. A literature review is undertaken to capture salient interpersonal skills. Leadership style, listening, team building, and clarifying expectations are identified. A questionnaire survey is drafted to capture construction practitioners' perception of these skills' effects on communication quality, returning 180 responses. Next, an artificial neural network model, or communication quality predictor (CQP), is developed, able to predict the quality of workers' interpersonal communication. The model accuracy on training is 87%; for testing, 79%. Finally, CQP is deployed in a real-time context in order to validate the reliability, returning an 80% prediction accuracy. This study is the first of its kind in offering a quantified, predictive model associating interpersonal skills with quality of communications in the context of the construction sector. In practical terms, the CQP can flag interpersonal conflicts before they escalate, while also guiding construction managers in the design of interpersonal skills training