Incomplete Data Sets Research Articles

Iterative Image Reconstruction with Under-Sampled Data Assisted by a Neural Network.

Image reconstruction with under-sampled data is usually achieved by an iterative algorithm, which minimizes an objective function. The objective function commonly contains a data fidelity term and one or more Bayesian terms. A popular Bayesian term is the total variation (TV) norm of the image. This paper suggests an addition Bayesian term that is generated by a neural network. This neural network is essentially a classifier. This classifier can recognize the artifacts caused by the incomplete data. This classifier is trained by patient images reconstructed by complete and incomplete data sets. This newly introduced Bayesian term is referred to as the CNN score, which is a real number in (-∞, ∞). Patient studies show the good correlation between the CNN score and the severeness of the artifacts due to the incomplete measurements. A neural network can extract features from images that are suffering from incomplete measurements and convert the features into a CNN score. An iterative image reconstruction algorithm can be developed to minimize this CNN score to suppress the artifacts in the reconstructed image.

Fuzzy neuron modeling of incomplete data for missing value imputation

Missing values are a common problem found in many real-world datasets, and cannot be avoided. It is a challenging task to model incomplete data and reasonably impute missing values. This paper focuses on regression imputation and uses a tracking-removed autoencoder (TRAE) to construct the mutual fitting correlation on incomplete data. Considering the differences in regression relationships across different sample categories, we introduce Takagi-Sugeno (TS) fuzzy architecture and propose a category-based tracking-removed autoencoder (TS-TRAE) to model incomplete data for missing value imputation. The TS-TRAE model partitions the incomplete dataset into several subclusters using membership information obtained from fuzzy clustering, then establishes a TRAE-based submodel to mine relationships within each subcluster for precise modeling of incomplete data. During model training, in order to fully utilize all existing values, we treat missing values as variables and propose an iterative learning method that optimizes missing variables and network parameters collaboratively. This method allows incomplete samples to participate in model training while also enabling the imputation of missing values. The TS-TRAE model integrates the inner category structure of incomplete data and the attribute association features effectively. The experimental results verify the effectiveness of the proposed method.

Do we need a more sophisticated multilayer artificial neural network to compute roughness coefficient?

Artificial neural networks (ANNs) are one of the most rapidly growing fields of soft computing. Along with deep learning, they are currently the most widely used machine learning techniques. Artificial neural networks are especially suitable for problem-solving where a researcher deals with incomplete data sets and no algorithms or specific sets of rules to be followed.This article deals with a case of comparison of several modifications of neural networks that may be applied to compute Chézy’s roughness coefficient. Neural network modelling is often started with one hidden layer. Having even one hidden layer, a neural network presents a powerful computing system to give good results. If it is necessary, the number of hidden layers may increase. Usually, two or three hidden layers of neurons are used. Diverse activation functions may also apply. The article aims to explore the necessity of developing sophisticated multilayer artificial neural networks to compute Chézy’s roughness coefficient.Under the study, the following modifications of the neural network computing Chézy’s roughness coefficient were considered and analysed: (1) Application of two hidden layers of neurons; (2) Application of three hidden layers of neurons; (3) Use of a dropout algorithm for training neural networks by randomly dropping units during training to prevent their co-adaptation; (4) Apart from the sigmoid (logistic) activation function, the use of other artificial neuron transfer functions – hyperbolic tangent (tanh) and rectifying activation function (ReLU).The training and testing of the considered neural network options were carried out using the actual hydro-morphological and hydrological data related to the channel section on the Dnieper River (downstream of Kyiv), the Desna River section near Chernihiv, and the Pripyat River section near the town of Turiv. The Python object-oriented programming environment was applied to build and train the neural networks. The test results confirm the acceptability and sufficiency of computing the Chézy roughness coefficient using the ANN of direct propagation with one hidden layer and a sigmoid logistic activation function. The formation of a qualitative set of training data, as well as data arrangement and choosing a relevant computing model based on empirical knowledge, are, as concluded, among more actual issues than creating more sophisticated neural networks.

Neutrosophic soft set matrix and their applications based on multi criteria decision-making problems

Importance of decision-making problems in uncertain environments cannot be overstated. Soft set theory and neutrosophic soft set theory are advanced mathematical techniques that have recently been developed to better handle these uncertain, incomplete, and inconsistent data sets. The concepts of neutrosophic soft set, neutrosophic soft matrix, and its operations are presented in this work. To compare the neutrosophic soft sets, we also provide score, certainty, and accuracy functions. Afterward, we create the neutrosophic arithmetic operator (NA), neutrosophic weighted arithmetic operator neutrosophic geometric operator , neutrosophic weighted geometric operator , neutrosophic harmonic operator and neutrosophic weighted harmonic operator to aggregate the neutrosophic information. Furthermore, based on the above operators, we develop the neutrosophic multi criteria decision-making by using Laplace criterion, optimism criterion and savage criterion. Finally, a numerical example of this technique is presented to show how it can be applied and how effective it is.

Smart city design plan selection through single-valued neutrosophic probabilistic hesitant fuzzy rough aggregation information

One of the hottest areas for applying the solutions currently available is the internet of things-based smart housing society architecture and its uncertainty analysis. When intelligent parking, waste management, public transportation, public safety, and other automatic methods for housing society’s growth were implemented, it became even more crucial. An intelligent, smart system is necessary to manage these problems and provide smooth services. Additionally, it will be helpful in reducing issues with time waste and societal safety. However, the issue comes up when describing accurate, approximate, or questionable parking, transit, safety, and waste management areas. This paper discusses several mathematical solutions for the smart housing society that use fuzzy rough sets, probabilistic hesitant fuzzy sets, and their extensions with neutrosophic sets. For further growth, a few studies on the graphic display of the evolution of the smart housing society are also considered. The rough set theory can be useful when dealing with imprecise, incomplete, or indeterminate data sets. The core contribution of this work is the construction of a novel generalized notion of a single-valued neutrosophic probabilistic hesitant fuzzy rough set (SV-NPHFRS), which is a hybrid structure of the single-valued neutrosophic set, the probabilistic hesitant fuzzy set, and the rough set. In contrast to the present literature, the underlying idea of SV-NPHFRS is that it is a powerful mathematical tool for managing uncertainty and imperfect information. This method is particularly beneficial when there are a number of competing criteria to consider. The aggregation technique plays an important role in decision-making concerns, especially when more competing criteria are present. In the study’s comparison phase, the suggested decision support system is compared to relevant existing approaches. The results suggest that, in terms of choice flexibility, the suggested technique has the potential to outperform the drawbacks of the current decision-making tools. The proposed study is expected to be useful for a number of researchers conducting future work on housing societies, waste management, public safety diagnostics, and hybridization.

Maritime greenhouse gas emission estimation and forecasting through AIS data analytics: a case study of Tianjin port in the context of sustainable development

The escalating greenhouse gas (GHG) emissions from maritime trade present a serious environmental and biological threat. With increasing emission reduction initiatives, such as the European Union’s incorporation of the maritime sector into the emissions trading system, both challenges and opportunities emerge for maritime transport and associated industries. To address these concerns, this study presents a model specifically designed for estimating and projecting the spatiotemporal GHG emission inventory of ships, particularly when dealing with incomplete automatic identification system datasets. In the computational aspect of the model, various data processing techniques are employed to rectify inaccuracies arising from incomplete or erroneous AIS data, including big data cleaning, ship trajectory aggregation, multi-source spatiotemporal data fusion and missing data complementation. Utilizing a bottom-up ship dynamic approach, the model generates a high-resolution GHG emission inventory. This inventory contains key attributes such as the types of ships emitting GHGs, the locations of these emissions, the time periods during which emissions occur, and emissions. For predictive analytics, the model utilizes temporal fusion transformers equipped with the attention mechanism to accurately forecast the critical emission parameters, including emission locations, time frames, and quantities. Focusing on the sea area around Tianjin port—a region characterized by high shipping activity—this study achieves fine-grained emission source tracking via detailed emission inventory calculations. Moreover, the prediction model achieves a promising loss function of approximately 0.15 under the optimal parameter configuration, obtaining a better result than recurrent neural network (RNN) and long short-term memory network (LSTM) in the comparative experiments. The proposed method allows for a comprehensive understanding of emission patterns across diverse vessel types under various operational conditions. Coupled with the prediction results, the study offers valuable theoretical and data-driven support for formulating emission reduction strategies and optimizing resource allocation, thereby contributing to sustainable maritime transformation.

Combining data discretization and missing value imputation for incomplete medical datasets.

Data discretization aims to transform a set of continuous features into discrete features, thus simplifying the representation of information and making it easier to understand, use, and explain. In practice, users can take advantage of the discretization process to improve knowledge discovery and data analysis on medical domain problem datasets containing continuous features. However, certain feature values were frequently missing. Many data-mining algorithms cannot handle incomplete datasets. In this study, we considered the use of both discretization and missing-value imputation to process incomplete medical datasets, examining how the order of discretization and missing-value imputation combined influenced performance. The experimental results were obtained using seven different medical domain problem datasets: two discretizers, including the minimum description length principle (MDLP) and ChiMerge; three imputation methods, including the mean/mode, classification and regression tree (CART), and k-nearest neighbor (KNN) methods; and two classifiers, including support vector machines (SVM) and the C4.5 decision tree. The results show that a better performance can be obtained by first performing discretization followed by imputation, rather than vice versa. Furthermore, the highest classification accuracy rate was achieved by combining ChiMerge and KNN with SVM.

A Neighborhood-Similarity-Based Imputation Algorithm for Healthcare Data Sets: A Comparative Study

The increasing computerisation of medical services has highlighted inconsistencies in the way in which patients’ historic medical data were recorded. Differences in process and practice between medical services and facilities have led to many incomplete and inaccurate medical histories being recorded. To create a single point of truth going forward, it is necessary to correct these inconsistencies. A common way to do this has been to use imputation techniques to predict missing data values based on the known values in the data set. In this paper, we propose a neighborhood similarity measure-based imputation technique and analyze its achieved prediction accuracy in comparison with a number of traditional imputation methods using both an incomplete anonymized diabetes medical data set and a number of simulated data sets as the sources of our data. The aim is to determine whether any improvement could be made in the accuracy of predicting a diabetes diagnosis using the known outcomes of the diabetes patients’ data set. The obtained results have proven the effectiveness of our proposed approach compared to other state-of-the-art single-pass imputation techniques.

A sustainable approach for exploiting cross-border nonrenewable resources using hybrid Game Theory and Ordinal Priority Approach

Throughout history, cross-border resources have been a major cause of conflicts and tensions among countries. Typically, involved countries put all their capacities into exploiting nonrenewable resources as much as possible to gain maximum economic benefits, compared to their neighbors. Such attitude will affect the future of Earth and human life in the shared-border areas in the mid and long term. The Game Theory, despite its broad applications and popularity, cannot dissuade free-riders from their self-centered strategies toward shared nonrenewable resources due to the higher payoff. Moreover, Game Theory cannot encourage countries to employ sustainable approaches and propel them toward renewable resources. In this global-scale case, the intermediation of objective and powerful “International Organizations” is highly needed to supervise competitive games between countries and promote sustainability in players' strategies and behaviors. This study proposes a “hybrid Game Theory and Ordinal Priority Approach” as a new solution for exploiting cross-border nonrenewable resources with the view of sustainable development. The proposed framework enables international organizations to enforce the “sustainability criteria for protecting natural resources and shifting to renewables” and control “Countries’ strategies in extracting shared nonrenewable resources.” Consequently, neighbouring countries see their desired pay-offs in sustainable strategies that regenerate the environment, decarbonize the economy, and protect societies and local communities. This study continues with an illustrative case study to discuss the applicability of hybrid Game-OPA in real world under complete and incomplete datasets. Ultimately, the performance and reliability of the hybrid Game-OPA model have been checked through a sensitivity analysis, which showed how small changes in inputs could impact final decisions.

A-MKMC: An effective adaptive-based multilevel K-means clustering with optimal centroid selection using hybrid heuristic approach for handling the incomplete data

In general, clustering is defined as partitioning similar and dissimilar objects into several groups. It has been widely used in applications like pattern recognition, image processing, and data analysis. When the dataset contains some missing data or value, it is termed incomplete data. In such implications, the incomplete dataset issue is untreatable while validating the data. Due to these flaws, the quality or standard level of the data gets an impact. Hence, the handling of missing values is done by influencing the clustering mechanisms for sorting out the missing data. Yet, the traditional clustering algorithms fail to combat the issues as it is not supposed to maintain large dimensional data. It is also caused by errors of human intervention or inaccurate outcomes. To alleviate the challenging issue of incomplete data, a novel clustering algorithm is proposed. Initially, incomplete or mixed data is garnered from the five different standard data sources. Once the data is to be collected, it is undergone the pre-processing phase, which is accomplished using data normalization. Subsequently, the final step is processed by the new clustering algorithm that is termed Adaptive centroid based Multilevel K-Means Clustering (A-MKMC), in which the cluster centroid is optimized by integrating the two conventional algorithms such as Border Collie Optimization (BCO) and Whale Optimization Algorithm (WOA) named as Hybrid Border Collie Whale Optimization (HBCWO). Therefore, the validation of the novel clustering model is estimated using various measures and compared against traditional mechanisms. From the overall result analysis, the accuracy and precision of the designed HBCWO-A-MKMC method attain 93 % and 95 %. Hence, the adaptive clustering process exploits the higher performance that aids in sorting out the missing data issuecompared to the other conventional methods.

MREDNet: A Multi-Residual Encoder-Decoder Network for recovering LFM interfered by intra-pulse retransmission interferences

Retransmission interference in radar and communication signal processing significantly threatens the received linear frequency modulation signal. With the advancement in the hardware capabilities of interference devices, they can intercept and retransmit signals within the duration of a single pulse. Therefore, we refer to this type of interference as intra-pulse retransmission interference, which includes retransmission modes smeared spectrum, interrupted sampling repeater jamming, and interrupted nonuniform sampling repeater jamming. Although traditional interference cancellation techniques offer optimal solutions under the assumption of fully known interference parameters, practical scenarios often involve unknown parameters, estimation errors, random parameters, and multiple interferences, leading to performance degradation. To address these problems, we propose a Multi-Residual Encoder-Decoder Network (MRDENet), a novel approach for effectively mitigating intra-pulse retransmission interferences and recovering the signal of linear frequency modulation. In contrast to parameter estimation and interference cancellation, MREDNet extracts crucial features of both the target and interference and performs reconstruction. More specifically, MREDNet acquires information about the retransmission pattern of interference and the modulation frequency through its multi-level residual shortcut connections. To address the issue of network parameters being real while the signals are complex, we devised decomplexification—a data preprocessing approach enabling neural networks for direct utilization in front-end complex signal processing. Besides, we evaluate the performance of MREDNet comprehensively. The experimental conditions encompassed single and multiple interferences, incomplete data sets, varying signal-to-noise ratios, and jamming-to-signal ratios. The computational simulations demonstrate that MREDNet successfully solves the problem of recovering linear frequency modulation signals interfered with by intra-pulse retransmission interference. According to the boundary condition of MREDNet, the proposed method exhibits valuable applications in real-world scenarios.

A novel information fusion method using improved entropy measure in multi-source incomplete interval-valued datasets

Multi-source data is a comprehensive data type that combines multiple sources of information or datasets. Compared to point-valued data, interval-valued data provides a more accurate representation of the uncertainty and variability associated with objects. In practical situations, data obtained from multiple sources may contain missing values for various reasons. Therefore, it is essential to develop multi-source information fusion technology in order to achieve information fusion or information extraction from multi-source incomplete data. This paper aims to explore the information fusion problem of multi-source incomplete interval-valued datasets. The primary contributions of this study involve utilizing the principle of statistical distribution and KL divergence to establish a metric for measuring the similarity between intervals. Firstly, this approach helps to reduce the problem of disregarding internal information within interval values, which can result in the loss of valuable information. Secondly, we establish an interval fuzzy similarity relation based on the mentioned concept of similarity among interval values. Moreover, we investigate the uncertainty measurement of incomplete interval-valued decision datasets and design an emerging information entropy fusion method. Finally, we comprehensively evaluate the effectiveness of the proposed method. Experimental results indicate that the proposed approach has advantage over the maximum, minimum, mean, and information entropy fusion method based on tolerance relationship. In addition, the distance metric used in this article can improve the fusion classification effect compared to several common interval-valued distance measures.

Research on the Bearing Lifespan Prediction Method for Ship Propulsion Shaft Systems Based on an Enhanced Domain Adversarial Neural Network

To address the challenge of accurate lifespan prediction for bearings in different operating conditions within ship propulsion shaft systems, a two-stage prediction model based on an enhanced domain adversarial neural network (DANN) is proposed. Firstly, pre-training features containing comprehensive degradation information are extracted from the entire source domain dataset encompassing all operational conditions. Subsequently, DANN is employed to extract domain-invariant features that are difficult to distinguish. Following this, a feature alignment process is utilized to align high-dimensional features with pre-training features, thereby mitigating the adverse effects caused by missing data in the incomplete target operational condition dataset. Finally, the effectiveness of this approach is validated using operational data from bearings under multiple operating conditions. The experimental results demonstrate that the method presented in this paper achieves an average error reduction of 0.0626 and 0.0845 compared to the MK-MMD transfer learning method and self-attention ConvLSTM algorithms, respectively, and exhibits higher predictive reliability. This method can provide valuable insights for lifespan prediction challenges concerning bearings in ship propulsion shaft systems under various operational conditions, as well as similar cross-domain lifespan prediction problems.

Application of neural network modeling methods in solving initial boundary value problems for partial differential equations

Machine learning allows you to solve a variety of data analysis problems, but its use for solving differential equations has appeared relatively recently. The approximation of the solution of the boundary value problem for differential equations (ordinary and partial derivatives) is constructed using neural network functions. The selection of weighting coefficients is carried out during the training of the neural network. The criteria for the quality of training in this case are inconsistencies in the equation and boundary-initial conditions. This approach makes it possible, instead of grid solutions, to find solutions defined on the entire feasible region of the boundary value problem. Specific examples show the features of the application of physics-informed neural networks to the solution of boundary value problems for differential equations of various types. Physics-informed neural networks training methods can be used in the tasks of retraining intelligent control systems on incomplete sets of input data.

Multi-class glioma segmentation on real-world data with missing MRI sequences: comparison of three deep learning algorithms

This study tests the generalisability of three Brain Tumor Segmentation (BraTS) challenge models using a multi-center dataset of varying image quality and incomplete MRI datasets. In this retrospective study, DeepMedic, no-new-Unet (nn-Unet), and NVIDIA-net (nv-Net) were trained and tested using manual segmentations from preoperative MRI of glioblastoma (GBM) and low-grade gliomas (LGG) from the BraTS 2021 dataset (1251 in total), in addition to 275 GBM and 205 LGG acquired clinically across 12 hospitals worldwide. Data was split into 80% training, 5% validation, and 15% internal test data. An additional external test-set of 158 GBM and 69 LGG was used to assess generalisability to other hospitals’ data. All models’ median Dice similarity coefficient (DSC) for both test sets were within, or higher than, previously reported human inter-rater agreement (range of 0.74–0.85). For both test sets, nn-Unet achieved the highest DSC (internal = 0.86, external = 0.93) and the lowest Hausdorff distances (10.07, 13.87 mm, respectively) for all tumor classes (p < 0.001). By applying Sparsified training, missing MRI sequences did not statistically affect the performance. nn-Unet achieves accurate segmentations in clinical settings even in the presence of incomplete MRI datasets. This facilitates future clinical adoption of automated glioma segmentation, which could help inform treatment planning and glioma monitoring.

GMRLNet: A graph-based manifold regularization learning framework for placental insufficiency diagnosis on incomplete multimodal ultrasound data.

Multimodal analysis of placental ultrasound (US) and microflow imaging (MFI) could greatly aid in the early diagnosis and interventional treatment of placental insufficiency (PI), ensuring a normal pregnancy. Existing multimodal analysis methods have weaknesses in multimodal feature representation and modal knowledge definitions and fail on incomplete datasets with unpaired multimodal samples. To address these challenges and efficiently leverage the incomplete multimodal dataset for accurate PI diagnosis, we propose a novel graph-based manifold regularization learning (MRL) framework named GMRLNet. It takes US and MFI images as input and exploits their modality-shared and modality-specific information for optimal multimodal feature representation. Specifically, a graph convolutional-based shared and specific transfer network (GSSTN) is designed to explore intra-modal feature associations, thus decoupling each modal input into interpretable shared and specific spaces. For unimodal knowledge definitions, graph-based manifold knowledge is introduced to describe the sample-level feature representation, local inter-sample relations, and global data distribution of each modality. Then, an MRL paradigm is designed for inter-modal manifold knowledge transfer to obtain effective cross-modal feature representations. Furthermore, MRL transfers the knowledge between both paired and unpaired data for robust learning on incomplete datasets. Experiments were conducted on two clinical datasets to validate the PI classification performance and generalization of GMRLNet. State-of-the-art comparisons show the higher accuracy of GMRLNet on incomplete datasets. Our method achieves 0.913 AUC and 0.904 balanced accuracy (bACC) for paired US and MFI images, as well as 0.906 AUC and 0.888 bACC for unimodal US images, illustrating its application potential in PI CAD systems.

Leveraging electronic device outputs for service quality assessment of public transportation systems: An initial look at the Philippines’ service contracting program through data analytic lens

Electronic devices such as CCTVs and GPS trackers are becoming more prevalent in public utility vehicles. These electronic devices produce data that can be aggregated and processed to generate insights regarding the service quality provided by public transportation systems. We describe a methodology for processing CCTV-produced videos and GPS logs, and utilize it to perform a first order, limited assessment of the Philippines’ Service Contracting Program, which ran during the COVID-19 pandemic. Our analyses uncovered vast discrepancies between data-supported figures and manually-generated estimates provided by the transport operators, especially with regards to vehicle occupancy and total number of commuters served. Our study also uncovered imperfections in the data generated by the electronic devices, which place caveats or limitations on the insights generated by the data analyses. We provide and suggest ways on how the analyses can be modified to take into account such missing data, thus enabling us to come up with educated, reasonable insights despite incomplete input datasets.

Poster] Identifying Maintenance, Repair and Overhaul Events based on ADS-B Data

Maintenance, Repair and Overhaul (MRO) plays a big role in aviation schedules and operational costs. In this work service difficulty reports are combined with aircraft ADS-B data to identify MRO activity based on an aircraft's layover information. The ``Inspection/Maintenance'' stage of operation and date of service difficulty reports allow to label layovers from ADS-B with the presence of MRO activity. Using a support vector machine this work classifies layovers based on their duration and time of day, achieving an 86% overall accuracy and 78% balanced accuracy. These results reflect the difficulty of working with this sparse and incomplete dataset, however, they allow us to study MRO schedules. The combination of these data sources offers new opportunities to find relations between aircraft trajectory information and MRO, which will be further investigated in future work.

On a probabilistic evolutionary approach to ocean modelling: From Lorenz-63 to idealized ocean models

In this study we develop an alternative way to model the ocean reflecting the chaotic nature of ocean flows and uncertainty of ocean models — instead of making use of classical deterministic or stochastic differential equations we offer a probabilistic evolutionary approach (PEA) that capitalizes on the use of probabilistic dynamics in phase space. The main feature of the data-driven version of PEA proposed in this work is that it does not require to know the physics behind the flow dynamics to model it. Within the PEA framework we develop two probabilistic evolutionary methods, which are based on probabilistic evolutionary models using quasi time-invariant structures in phase space.The methods have been tested on complete and incomplete reference data sets generated by the Lorenz 63 system and by an idealized two-layer quasi-geostrophic model. The results show that both methods reproduce large- and small-scale features of the reference flow by keeping the probabilistic dynamics within the phase space of the reference flow. The proposed approach offers appealing benefits and a great flexibility to ocean modellers working with mathematical models and measurements. The most remarkable one is that it provides an alternative to the mainstream ocean parameterizations, requires no modification of existing ocean models, and is easy to implement. Moreover, it does not depend on the nature of input data, and therefore could work with both numerically-computed flows and real measurements from different sources (drifters, weather stations, etc.).

Transfer Learning Prediction Performance of Chillers for Neural Network Models

Building automation systems installed in large commercial buildings record sub-hourly measurements from hundreds of sensors. The use of such large datasets are challenging because of missing and erroneous data, which can prevent the development of accurate prediction models of the performance of heating, ventilation, and air-conditioning equipment. The use of the transfer learning (TL) method for building applications attracted researchers to solve the problems created by small and incomplete datasets. This paper verifies the hypothesis that the deep neural network models that are pre-trained for one chiller (called the source chiller) with a small dataset of measurements from July 2013 could be applied successfully, by using TL strategies, for the prediction of the operation performance of another chiller (called the target chiller) with different datasets that were recorded during the cooling season of 2016. Measurements from a university campus are used as a case study. The results show that the initial hypothesis of this paper is confirmed.