Uneven Distribution Of Data Research Articles

Towards application of various machine learning techniques in agriculture

Since the invention of the computer, all available information in every field has been digitized and made available to people who use computer resources. As a result, massive amounts of data are being generated in every domain at an alarming rate. Agriculture is one such area of interest for researchers. Machine learning is the process of extracting useful information from various types of data. The classification of objects is an important area within the field of data mining, and its application extends to a variety of areas, whether or not in the field of science. Although k-Nearest Neighbor classification is a simple and effective technique, it slows down the classification of each object. Furthermore, the classification’s effectiveness suffers as a result of the uneven distribution of training data. The purpose of this paper is to look into the applicability of various machine learning techniques in agriculture.

Bayesian Estimation of Surface Strain Rates From Global Navigation Satellite System Measurements: Application to the Southwestern United States

AbstractSeismic hazard assessment in active fault zones can benefit of strain rate measurements derived from geodetic data. Producing a continuous strain rate map from discrete data is an inverse problem traditionally tackled with standard interpolation schemes. Most algorithms require user‐defined regression parameters that determine the smoothness of the recovered velocity field and the amplitude of its spatial derivatives. This may lead to biases in the strain rates estimation which could eventually impact studies on earthquake hazard. Here we propose a transdimensional Bayesian method to estimate surface strain rates from Global Navigation Satellite System (GNSS) velocities. We parameterize the velocity field with a variable number of Delaunay triangles and use a reversible‐jump Monte‐Carlo Markov Chain algorithm to sample the probability distribution of surface velocities and spatial derivatives. The solution is a complete probability distribution function for each component of the strain rate field. We conduct synthetic tests and compare our approach to a standard b‐spline interpolation scheme. Our method is more resilient to data errors and uneven data distribution, while providing uncertainties associated with recovered velocities and strain rates. We apply our method to the Southwestern United States, an extensively studied and monitored area and infer probabilistic strain rates along the main fault systems, including the San Andreas one, from the inversion of interseismic GNSS velocities. Our approach provides a full description of the strain rate tensor for zones where strain rates are highly contrasted, with no need to manually tune user‐defined parameters. We recover sharp velocity gradients, without systematic artifacts.

DV-DVFS: merging data variety and DVFS technique to manage the energy consumption of big data processing

Data variety is one of the most important features of Big Data. Data variety is the result of aggregating data from multiple sources and uneven distribution of data. This feature of Big Data causes high variation in the consumption of processing resources such as CPU consumption. This issue has been overlooked in previous works. To overcome the mentioned problem, in the present work, we used Dynamic Voltage and Frequency Scaling (DVFS) to reduce the energy consumption of computation. To this goal, we consider two types of deadlines as our constraint. Before applying the DVFS technique to computer nodes, we estimate the processing time and the frequency needed to meet the deadline. In the evaluation phase, we have used a set of data sets and applications. The experimental results show that our proposed approach surpasses the other scenarios in processing real datasets. Based on the experimental results in this paper, DV-DVFS can achieve up to 15% improvement in energy consumption.

An Intestinal Centerline Extraction Algorithm Based on Federated Framework

The intestine is an important organ of the human body, and its internal structure always needs to be observed in clinical applications so as to provide a basis for accurate diagnosis. However, due to the limited intestinal data obtained by a single institution, deep learning cannot effectively train the intestines, and the effect is not satisfied. For this reason, we propose a distributed training method to carry out federated learning to alleviate the situation of patient sample data shortage, not shared and uneven data distribution. And the blockchain is introduced to enhance the interaction between networks, to solve the problem of a single point of failure of the federated learning server. Fully excavate the multiscale features of samples, to construct a fusion enhancement model and intestinal segmentation module for accurate positioning. At the local end, the centerline extraction algorithm is optimized, with the edge as the main and the source as the auxiliary to realize centerline extraction.

Generating Road Networks for Old Downtown Areas Based on Crowd-Sourced Vehicle Trajectories.

With the popularity of portable positioning devices, crowd-sourced trajectory data have attracted widespread attention, and led to many research breakthroughs in the field of road network extraction. However, it is still a challenging task to detect the road networks of old downtown areas with complex network layouts from high noise, low frequency, and uneven distribution trajectories. Therefore, this paper focuses on the old downtown area and provides a novel intersection-first approach to generate road networks based on low quality, crowd-sourced vehicle trajectories. For intersection detection, virtual representative points with distance constraints are detected, and the clustering by fast search and find of density peaks (CFDP) algorithm is introduced to overcome low frequency features of trajectories, and improve the positioning accuracy of intersections. For link extraction, an identification strategy based on the Delaunay triangulation network is developed to quickly filter out false links between large-scale intersections. In order to alleviate the curse of sparse and uneven data distribution, an adaptive link-fitting scheme, considering feature differences, is further designed to derive link centerlines. The experiment results show that the method proposed in this paper preforms remarkably better in both intersection detection and road network generation for old downtown areas.

Research on Hybrid Index Based on 3D Multi-Level Adaptive Grid and R+ Tree

With the widespread application of Geographic Information System (GIS), three-dimensional spatial data, as the reflection of the real world entity, has an increasing amount of data, and the phenomenon of uneven data distribution appears. If a single spatial index structure is used to store and manage these data, there will be a waste of storage space and low query efficiency. A hybrid index structure based on 3D multi-level adaptive grid and R+ tree was proposed to solve these problems. The index structure was mainly composed of two structures, multi-level grid and R+ tree. Firstly, the data set was processed by the multi-level automatic grid algorithm based on normal distribution, and the length, width and height of the grid were obtained. Secondly, a multi-level adaptive grid structure was used to partition the data space quickly and effectively, and the advantage of zero overlap of the intermediate nodes of the R+ tree was used for efficient indexing. Finally, the maintenance and query algorithms of the index structure were given in detail, which solved the problem of low index establishment and retrieval efficiency under the condition of uneven distribution of massive data sets. In this paper, a data set subject to Gauss distribution was used to simulate the distribution of three-dimensional data. Through a large number of experimental comparison tests, it was proved that the hybrid index structure based on 3D multi-level adaptive grid-R+ tree proposed in this paper had good performance in both index structure construction and query in the case of massive data sets or uneven data distribution.

Recent Research at San Isidro, El Salvador, in the Context of Southeastern Mesoamerican Archaeology

Archaeology of the extreme southeast of Mesoamerica, despite receiving a fair amount of scholarly attention, still remains a relatively poorly developed field. In this article I identify main factors that hamper our understanding of the ancient past of this region, including population density, volcanism, multiple historical reasons, and exceptionally uneven distribution of data from different periods. The second half of the Preclassic period (ca. 1000 BC - AD 250) seems to be the most understudied, and likely the most crucial time for the southeastern boundary of Mesoamerica from the perspective of reconstructing processes of cultural dynamics and emergence of identities. I offer a probable, if only partial solution to the problem by presenting recent advances, and future directions of my ongoing research at a large Preclassic site of San Isidro, Sonsonate, El Salvador. I argue that even at the early stage of investigation San Isidro shows great potential for providing the missing data.

New Online Streaming Feature Selection Based on Neighborhood Rough Set for Medical Data

Not all features in many real-world applications, such as medical diagnosis and fraud detection, are available from the start. They are formed and individually flow over time. Online streaming feature selection (OSFS) has recently attracted much attention due to its ability to select the best feature subset with growing features. Rough set theory is widely used as an effective tool for feature selection, specifically the neighborhood rough set. However, the two main neighborhood relations, namely k-neighborhood and neighborhood, cannot efficiently deal with the uneven distribution of data. The traditional method of dependency calculation does not take into account the structure of neighborhood covering. In this study, a novel neighborhood relation combined with k-neighborhood and neighborhood relations is initially defined. Then, we propose a weighted dependency degree computation method considering the structure of the neighborhood relation. In addition, we propose a new OSFS approach named OSFS-KW considering the challenge of learning class imbalanced data. OSFS-KW has no adjustable parameters and pretraining requirements. The experimental results on 19 datasets demonstrate that OSFS-KW not only outperforms traditional methods but, also, exceeds the state-of-the-art OSFS approaches.

Automated glaucoma screening method based on image segmentation and feature extraction.

Glaucoma is a chronic disease that threatens eye health and can cause permanent blindness. Since there is no cure for glaucoma, early screening and detection are crucial for the prevention of glaucoma. Therefore, a novel method for automatic glaucoma screening that combines clinical measurement features with image-based features is proposed in this paper. To accurately extract clinical measurement features, an improved UNet++ neural network is proposed to segment the optic disc and optic cup based on region of interest (ROI) simultaneously. Some important clinical measurement features, such as optic cup to disc ratio, are extracted from the segmentation results. Then, the increasing field of view (IFOV) feature model is proposed to fully extract texture features, statistical features, and other hidden image-based features. Next, we select the best feature combination from all the features and use the adaptive synthetic sampling approach to alleviate the uneven distribution of training data. Finally, a gradient boosting decision tree (GBDT) classifier for glaucoma screening is trained. Experimental results based on the ORIGA dataset show that the proposed algorithm achieves excellent glaucoma screening performance with sensitivity of 0.894, accuracy of 0.843, and AUC of 0.901, which is superior to other existing methods.Graphical abstract Framework of the proposed glaucoma classification method.

Optimization of Skewed Data Using Sampling-Based Preprocessing Approach.

In the past few years, classification has undergone some major evolution. With a constant surge of the amount of data gathered from different sources, efficient processing and analysis of data is becoming difficult. Due to the uneven distribution of data among classes, data classification with machine-learning techniques has become more tedious. While most algorithms focus on major data samples, they ignore the minor class data. Thus, the data-skewing issue is one of the critical problems that need attention of researchers. The paper stresses upon data preprocessing using sampling techniques to overcome the data-skewing problem. Here, three different sampling techniques such as Resampling, SpreadSubSampling, and SMOTE are implemented to reduce this uneven data distribution issue and classified with the K-nearest neighbor algorithm. The performance of classification is evaluated with various performance metrics to determine the efficiency of classification.

An Optimization Technique of the 3D Indoor Map Data Based on an Improved Octree Structure

The construction and retrieval of indoor maps are important for indoor positioning and navigation. It is necessary to ensure a good user experience while meeting real-time requirements. Unlike outdoor maps, indoor space is limited, and the relationship between indoor objects is complex which would result in an uneven indoor data distribution and close relationship between the data. A data storage model based on the octree scene segmentation structure was proposed in this paper initially. The traditional octree structure data storage model has been improved so that the data could be backtracked. The proposed method will solve the problem of partition lines within the range of the object data and improve the overall storage efficiency. Moreover, a data retrieval algorithm based on octree storage structure was proposed. The algorithm adopts the idea of “searching for a point, points around the searched point are within the searching range.” Combined with the octree neighbor retrieval methods, the closure constraints are added. Experimental results show that using the improved octree storage structure, the retrieval cost is 1/8 of R-tree. However, by using the neighbor retrieval, it improved the search efficiency by about 27% on average. After adding the closure constraint, the retrieval efficiency increases by 25% on average.

Two-dimensional discrete feature based spatial attention CapsNet For sEMG signal recognition

Deep learning frameworks(such as deep convolutional networks) require data to have a regular shape. However, discrete features extracted from heterogeneous data cannot be collected in a regular shape to convolute. In this article, a Two-Dimensional Discrete Feature Based Spatial Attention CapsNet(TDACAPS) is proposed to convert one-dimensional discrete features into two-dimensional structured data through Cartesian Product for surface electromyogram(sEMG) signal recognition. sEMG signal varies from person to person is the main signal source of prosthetic control. Our model transforms multi-angle discrete features into structured data to find the inherent law of sEMG signal. Due to uneven information distribution of structured data, this model combines capsule network with attention mechanism to place emphasis on abundant information regions and reduce ancillary information loss. Extensive experiments show our model yields an improvement for sEMG signal recognition of almost 3% than capsule network and other neural networks under different conditions. Our attention mechanism that employs overlapping pooling to search feature map weight is preferable to the squeeze-and-excitation module, convolutional block attention module and others. Moreover, we validate that our model has great expansibility on Wine Quality Dataset and Breast Cancer Wisconsin.

450-P: Development and Validation of a Predictive Major Adverse Cardiac Events (MACE) Risk Model for Diabetes Patients with Acute Coronary Syndrome (ACS)

Background: Studies have shown that acute coronary syndrome (ACS) has a higher incidence of MACE in diabetic patients than in nondiabetic patients by 39%. The complexity of the MACE limits clinical predictions, the success rate is shallow. Several existing MACE risk prediction models, including some developed specifically for ACS patients, also stuck with low accuracy and are not suitable for complications scenarios. The objective of this study is to develop and validate a model based on EHR data to predict MACE risk in patients with type 2 diabetes mellitus (T2DM) complicated with ACS. Methods: This study recruited 16,807 in-hospital T2DM patients with ACS from 38 urban and rural hospitals. Patients with 41 variables, including age, demographics, living habits, ACS type, and Killip class. The criteria for the diagnosis of MACE developed for this study consider cardiovascular death, myocardial infarction, or ischemic stroke. There are 711 MACE records diagnosed by doctors during patients’ hospitalization and recorded in medical records, only 4.2% of the total cohort. Very low morbidity inspires us to introduce a model called Support Vector Data Description (SVDD) to combat this uneven distribution of data. SVDD only maps the features of MACE patients to the high-dimensional space and constructs a minimum sphere to contain all the target data, taking the sphere as the boundary to classify the risk of newly admitted patients. Models were developed and validated on 80% and 20% of the sample, respectively. The model performance was assessed by C-statistics and the F1. Results: The model performed with C-statistics of 0.873 and F1 of 0.833. We found 5 variables had a significant univariate association with MACE included Age, Heart rate, Killip class, white blood cell count and Creatinine at p<10-4. Conclusions: This novel study suggests that the well trained SVDD model would be possible to predict the harmful MACE among T2DM patients with ACS. Disclosure P. Hu: None. X. Di: None. Y. Zhang: None. Z. Tang: None. S. Li: None. J. Mei: None. C. Ma: None.

A Fast Tile-Pyramid Construction Algorithm Based on Multilevel Task Parallelism

With the rapid development of satellite sensor technology and aerial photography technology, spatial and temporal resolution of the remote sensing image has been greatly improved. To speed up the visualization of remote sensing images, pyramid building technology is the best choice at present. However, the performance of existing methods is not as good as expected. Considering the load imbalance problem caused by the uneven data distribution problem, this paper proposes a fast tile-pyramid construction algorithm based on multilevel task parallelism. Our method changed the traditional parallel tile generation algorithm which is generated layer by layer and implemented a new algorithm that generated multiple layers at the same time by using the entire tile pyramid as a whole. Experimental results on real-life datasets show that our new algorithm can improve the construction speed of tile-pyramids.

Privacy-Preserving Federated Learning in Fog Computing

Federated learning can combine a large number of scattered user groups and train models collaboratively without uploading data sets, so as to avoid the server collecting user sensitive data. However, the model of federated learning will expose the training set information of users, and the uneven amount of data owned by users in multiple users' scenarios will lead to the inefficiency of training. In this article, we propose a privacy-preserving federated learning scheme in fog computing. Acting as a participant, each fog node is enabled to collect Internet-of-Things (IoT) device data and complete the learning task in our scheme. Such design effectively improves the low training efficiency and model accuracy caused by the uneven distribution of data and the large gap of computing power. We enable IoT device data to satisfy ε -differential privacy to resist data attacks and leverage the combination of blinding and Paillier homomorphic encryption against model attacks, which realize the security aggregation of model parameters. In addition, we formally verified our scheme can not only guarantee both data security and model security but completely resist collusion attacks launched by multiple malicious entities. Our experiments based on the Fashion-MNIST data set prove that our scheme is highly efficient in practice.

A review on classification of imbalanced data for wireless sensor networks

Classification of imbalanced data is a vastly explored issue of the last and present decade and still keeps the same importance because data are an essential term today and it becomes crucial when data are distributed into several classes. The term imbalance refers to uneven distribution of data into classes that severely affects the performance of traditional classifiers, that is, classifiers become biased toward the class having larger amount of data. The data generated from wireless sensor networks will have several imbalances. This review article is a decent analysis of imbalance issue for wireless sensor networks and other application domains, which will help the community to understand WHAT, WHY, and WHEN of imbalance in data and its remedies.

Recent Advances in Real-Time Pluvial Flash Flood Forecasting

Recent years have witnessed considerable developments in multiple fields with the potential to enhance our capability of forecasting pluvial flash floods, one of the most costly environmental hazards in terms of both property damage and loss of life. This work provides a summary and description of recent advances related to insights on atmospheric conditions that precede extreme rainfall events, to the development of monitoring systems of relevant hydrometeorological parameters, and to the operational adoption of weather and hydrological models towards the prediction of flash floods. With the exponential increase of available data and computational power, most of the efforts are being directed towards the improvement of multi-source data blending and assimilation techniques, as well as assembling approaches for uncertainty estimation. For urban environments, in which the need for high-resolution simulations demands computationally expensive systems, query-based approaches have been explored for the timely retrieval of pre-simulated flood inundation forecasts. Within the concept of the Internet of Things, the extensive deployment of low-cost sensors opens opportunities from the perspective of denser monitoring capabilities. However, different environmental conditions and uneven distribution of data and resources usually leads to the adoption of site-specific solutions for flash flood forecasting in the context of early warning systems.

Improvement of job completion time in data-intensive cloud computing applications

Task stragglers in MapReduce jobs dramatically impede job execution of data-intensive computing in cloud data centers. This impedance is due to the uneven distribution of input data, heterogeneous data nodes, resource contention situations, and network configurations. Data skew of intermediate data in MapReduce job causes delay failures due to the violation of job completion time. Data-intensive computing frameworks, such as MapReduce or Hadoop YARN, employ HashPartitioner. This partitioner may cause intermediate data skew, which results in straggler reducers. In this paper, we strive to make Hadoop YARN more efficient in cloud environments. We present, a new partitioning scheme, called balanced data clusters partitioner (BDCP), to handle straggler Reduce tasks based on sampling of input data and feedback information about the current processing task. Our extensive experimental results show that BDCP can outperform the default Hadoop HashPartitioner and Range partitioner. BDCP can assist in straggler mitigation during reduce phase and minimize the job completion time in MapReduce jobs within data-intensive cloud computing.

Learning from Imbalanced Data in Classification

Imbalanced data learning is a research area and day by day development is going on. Due to these researchers are motivated to pay attention to find efficient and adaptive methods for real-world problems. Machine learning, as well as data mining, is a field where researchers are finding different methods to solve problems related to imbalanced datasets and also the challenges faced in day to day life. The uneven class distribution in the dataset is the reason behind the degradation of performance in approaches used by data mining as well as machine learning. Continuous advancements of machine learning as well as mining data combining it with big data, a deep insight is required to understand the nature of learning imbalanced data. New challenges are emerging due to this development. Among the two approaches algorithm level and data level, the most popular approach compared to this is the hybrid approach. It is found that there is a bias for the majority class which affects the decision making task and overall accuracy of classification. The ensemble method is an efficient technique to deal with the uneven distribution of data. The aim of the paper is to presents the overview of class imbalance problems, solutions to handle it, open issues and challenges in learning imbalanced datasets. Based on the experiment conducted on one dataset it is found that ensemble technique along with other data-level methods gives good results. This hybrid method can be applied in many real-life applications like software defect prediction, behavior analysis, intrusion detection, medical diagnosis, etc. The paper further provides research directions in learning from the imbalanced dataset.

Distribution Consistency Loss for Large-Scale Remote Sensing Image Retrieval

Remote sensing images are featured by massiveness, diversity and complexity. These features put forward higher requirements for the speed and accuracy of remote sensing image retrieval. The extraction method plays a key role in retrieving remote sensing images. Deep metric learning (DML) captures the semantic similarity information between data points by learning embedding in vector space. However, due to the uneven distribution of sample data in remote sensing image datasets, the pair-based loss currently used in DML is not suitable. To improve this, we propose a novel distribution consistency loss to solve this problem. First, we define a new way to mine samples by selecting five in-class hard samples and five inter-class hard samples to form an informative set. This method can make the network extract more useful information in a short time. Secondly, in order to avoid inaccurate feature extraction due to sample imbalance, we assign dynamic weight to the positive samples according to the ratio of the number of hard samples and easy samples in the class, and name the loss caused by the positive sample as the sample balance loss. We combine the sample balance of the positive samples with the ranking consistency of the negative samples to form our distribution consistency loss. Finally, we built an end-to-end fine-tuning network suitable for remote sensing image retrieval. We display comprehensive experimental results drawing on three remote sensing image datasets that are publicly available and show that our method achieves the state-of-the-art performance.