Adaptive Data Sampling Research Articles

Adaptive Data Sampling and Feature Optimization with LSH-IS and Pearson Correlation in Big Data Contexts

A huge amount of data is produced with the evolution of modern technologies. This high-throughput data generation results in Big Data, which consist of many features (attributes). Storing and processing large and varied datasets (known as big data) is challenging to do in real time. In machine learning, streaming feature selection has always been considered a superior technique for selecting the relevant subset features from highly dimensional data and thus reducing learning complexity. Instance and Feature selection has been a key research area in data mining, which chooses a subset of relevant features for use in model building. This paper aims to provide an overview of instance and feature selection methods for big data mining. It first discusses the current challenges and difficulties faced when mining valuable information from big data.Instance and feature selection has become an effective approach due to enormous data which is continuously being produced in the field of research. It is difficult to process such large datasets by many systems. Though the traditional techniques are useful for large datasets, the numbers when in hundreds, thousands or millions face scaling problems. The proposed work focuses on, scalable instance and feature selection in big data environment. Locality-sensitive hashing instance selection (LSH-IS) is a two pass method used to find similar instances along with Pearson correlation coefficient for feature selection. Hash function family is used which is a general method of reducing the size of a set; this is achieved by re- indexing the elements into buckets. This process find similar instance in same bucket, hence instance can be reduced. The work aims at improving the performance of locality sensitive hashing by storing additional information of the instances and features assigned of each class in the bucket and also to improve accuracy of instance and feature selection algorithm.

Extended nnU-Net for Brain Metastasis Detection and Segmentation in Contrast-Enhanced Magnetic Resonance Imaging With a Large Multi-Institutional Data Set

PurposeTo investigate an extended self-adapting nnU-Net framework for detecting and segmenting brain metastases (BM) on MRI. ApproachSix different nnU-Net systems with adaptive data sampling, adaptive Dice loss (ADL) or different patch/batch sizes were trained and tested for detecting and segmenting intraparenchymal BM with a size ≥ 2 mm on 3D post-Gd T1-weighted MRI volumes using 2092 patients from seven institutions (1712, 195 and 185 patients for training, validation and testing, respectively). Gross tumor volumes (GTVs) of BM delineated by physicians for stereotactic radiosurgery (SRS) were collected retrospectively and curated at each institute. Additional centralized data curation was carried out to create GTVs of uncontoured BM by two radiologists to improve accuracy of ground truth. The training dataset was augmented with synthetic BMs of 1025 MRI volumes using a 3D generative pipeline. BM detection was evaluated by lesion-level sensitivity and false-positive (FP) rate. BM segmentation was assessed by lesion-level Dice similarity coefficient (DSC), 95-percentile Hausdorff distance (HD95) and average HD. The performances were assessed across different BM sizes. Additional testing was performed using a second dataset of 206 patients. ResultsOf the six nnU-Net systems, the nnU-Net with ADL achieved the best detection and segmentation performance on the first testing dataset. At an FP rate of 0.65±1.17, overall sensitivity was 0.904 for all sizes of BM, 0.966 for BM ≥ 0.1 cm3 and 0.824 for BM < 0.1 cm3. Mean values of DSC, HD95 and average HD of all detected BMs were 0.758, 1.45 mm and 0.23 mm, respectively. Performances on the second testing dataset achieved sensitivity of 0.907 at an FP rate of 0.57±0.85 for all BM sizes, and average HD of 0.33 mm for all detected BM. ConclusionsOur proposed extension of the self-configuring nnU-Net framework substantially improved small BM detection sensitivity while maintaining a controlled FP rate. Clinical utility of the extended nnU-Net model for assisting early BM detection and SRS planning will be investigated.

FAST: Enhancing Federated Learning Through Adaptive Data Sampling and Local Training

FAST: Enhancing Federated Learning Through Adaptive Data Sampling and Local Training

Multibiosensor Data Sampling and Transmission Reduction With Decision-Making for Remote Patient Monitoring in IoMT Networks

The rise in chronic diseases and the aging of the population led to an increase in the demand for remote healthcare systems that employ biosensors to monitor people’s health status. The increasing need for these automated systems has led to the emergence of the Internet of Medical Things (IoMT) networks. In the IoMT networks, the biosensor devices collect vital signs and transmit them to the gateway for further analysis and fusion. In light of the limited biosensor device resources (power, storage, and computation) and the periodical transmission of a large amount of data, it is necessary to optimize the transmission of data in order to conserve power while maintaining data quality at the gateway. Also, it became important to have a decision-making-based machine learning model at the gateway to evaluate a patient’s health and make a quick, accurate decision in case of an emergency. This paper proposes Multibiosensor Data Sampling and Transmission Reduction with Decision-making (MuDaSaTReD) for Remote Patient Monitoring in the IoMTs Networks. The MuDaSaTReD achieves this goal on two levels: biosensors and a fog gateway. It uses an Energy-saving Lightweight Data Transmission (ELiDaT) algorithm to get rid of the repeated data and then adapts the sampling rate of each biosensor using an Adaptive Data Sampling (ADaS) algorithm. The MuDaFuDeC implements the machine learning model at the fog gateway to learn and decide the situation of the patient according to the received data from the biosensors. The performance evaluation shows that the MuDaFuDeC outperforms other approaches in terms of the data reduction percentage and energy consumption. It keeps a good representation of all the scores at the fog gateway and makes automated, fast, and accurate decisions based on the patient’s condition.

The causality analysis of incipient fault in industrial processes using dynamic data stream transfer entropy

The propagation of the developing incipient fault embeds potential risks to the safety management of industrial processes. The transfer entropy (TE) based causality analysis method is an effective solution for early detection and timely intervention of fault propagation. However, the slow-varying incipient fault could not be analyzed timely since the conventional TE is restricted by the limited data processing capacity and fixed data selection mode. For real-time causality analysis of developing industrial fault, we proposed the dynamic data stream transfer entropy (DDSTE) algorithm and presented the DDSTE-based causality analysis method in this paper. Firstly, the conventional static data model is replaced by dynamic data stream model, which could be updated and expanded with continues import of new data and would not be restricted by the limited storage capacity. Secondly, concerning the varying timing feature of incipient fault, the adaptive data sampling window is designed to match with incipient fault evolution stages. Thirdly, the long-lasting fault sequence is coarse-grained to improve the calculation efficiency for on-line using. Compared to conventional methods, DDSTE-based causality analysis outperforms in rapid tracking of incipient fault propagation and accurate estimation of fault location. The data collected from a real coal gasification process is used to verify the reliability and superiority of the proposed method.

Adaptive goal-oriented data sampling in Data-Driven Computational Mechanics

Data-Driven (DD) computing is an emerging field of Computational Mechanics, motivated by recent technological advances in experimental measurements, the development of highly predictive computational models, advances in data storage and data processing, which enable the transition from a material data-scarce to a material data-rich era. The predictive capability of DD simulations is contingent on the quality of the material data set, i.e. its ability to closely sample all the strain–stress states in the phase space of a given mechanical problem. In this study, we develop a methodology for increasing the quality of an existing material data set through iterative expansions. Leveraging the formulation of the problems treated with the DD paradigm as distance minimization problems, we identify regions in phase space with poor data coverage, and target them with additional experiments or lower-scale simulations. The DD solution informs the additional experiments so that they can provide better coverage of the phase space of a given application.We first illustrate the convergence properties of the approach through a DD finite element simulation of a linear elastic cylinder under triaxial compression. The same numerical experiment is then performed on a specimen of Hostun sand, a material with complex history-dependent behavior. Data sampling is performed with Level-Set Discrete Element Method (LS-DEM) calculations of unit cells representative of this granular material, subjected to loading paths determined by the proposed method. It is shown that this adaptive expansion of the data set, tailored for a particular application, leads to convergent and accurate DD predictions, without the computational cost of using large databases with potentially redundant or low-quality data.

Efficient Global Optimization of Advanced Drop-Hinged Flap

This paper proposes an improved design methodology and global optimization method for the design of an advanced dropped-hinge flap (ADHF) using an integrated approach. The approach implemented an actuation-driven parametric model for a hinged-flap concept. The optimization methodology incorporates an adaptive data sampling method in the update of a series of surrogate models. Inclusion of an actuation mechanism in the definition of the model eliminated a large number of gap and overlap settings that are not practically possible, leading to further savings in the number of computational fluid dynamic calculations required. Parameters for the actuation definition, flap geometries, and spoiler settings are included simultaneously in the optimization in order to consider the interactions among these factors and to obtain near-global-optimal solutions. Compared with a traditional sequential shape/mechanism optimization methodology, which can only produce one design point meeting the landing requirement, a series of design points can be obtained from the proposed methodology, meeting the requirements for both takeoff and landing configurations. The optimal result not only significantly improves at an angle of attack of eight by 17.8% but also decreases the distance between the hinge and the lower surface of the airfoil by 22.4%, which can reduce the fairing size. Comparisons are given between the optimized ADHF and the traditional Fowler flap, along with detailed analysis on the potential benefits by supplementing the design with spoiler deflections.

Toward fast and accurate emergency cases detection in BSNs

In body sensor networks (BSNs), medical sensors capture physiological data from the human body and send them to the coordinator who act as a gateway to health care. The main aim of BSNs is to save peoples’ lives. Therefore, fast and correct detection of emergencies while maintaining low-energy consumption of sensors is essential requirement of BSNs. In this study, the authors propose a new adaptive data sampling approach, where the sampling ratio is adapted based on the sensed data variation. The idea is to use the modified version of the cumulative sum (CUSUM) algorithm (modified CUSUM) that they previously proposed for wireless sensor networks to monitor the data variability, and adapt the sampling rate accordingly. Modified CUSUM is then applied to the adaptively sampled data to detect anomalies, and the correlation property between physiological parameters will be used to identify emergency cases from false alarms. Several experiments are performed and compared to evaluate the efficiency of their approach, and different parameters are considered.

Automated model generation for electronic devices using neural network approaches

This paper describes artificial neural network approaches to convert the conventionally expensive process of electronic modelling into an automated model generation process. Artificial neural networks are trained using machine learning algorithm to learn the electronic behaviour, and the trained neural network becomes a model to help predict the electronic device behaviour. The automated model generation algorithm performs adaptive data sampling to determine the amount and the distribution of training data needed to train neural networks. The algorithm also determines the number of hidden neurons needed to achieve a compact and accurate model. Also incorporated into the automated model generation method is an efficient interpolation approach to make the process much faster. The objective of the described method is to generate a compact neural-network based model with better accuracy and in less time than conventional approach. Examples of automated modeling of radio-frequency and microwave filters used in wireless electronic systems are described showing the advantage of this technique.

Stream-Based Data Sampling Mechanism for Process Object

Process object is the instance of process. Vertexes and edges are in the graph of process object. There are different types of the object itself and the associations between object. For the large-scale data, there are many changes reflected. Recently, how to find appropriate real-time data for process object becomes a hot research topic. Data sampling is a kind of finding c hanges o f p rocess o bjects. There i s r equirements f or s ampling to be adaptive to underlying distribution of data stream. In this paper, we have proposed a adaptive data sampling mechanism to find a ppropriate d ata t o m odeling. F irst o f all, we use concept drift to make the partition of the life cycle of process object. Then, entity community detection is proposed to find changes. Finally, we propose stream-based real-time optimization of data sampling. Contributions of this paper are concept drift, community detection, and stream-based real-time computing. Experiments show the effectiveness and feasibility of our proposed adaptive data sampling mechanism for process object.

Multi‐objective robust dynamic VAR planning in power transmission girds for improving short‐term voltage stability under uncertainties

Modern power transmission grids are facing more and more critical short-term voltage stability problems. This study proposes a novel robust dynamic VAR planning approach for improving the short-term voltage stability level under uncertainties including the peak load level, the maximum proportion of dynamic load, fault clearing time and deviation of the actual capacity of dynamic VAR compensators from rated capacity when contingencies occur. The robust dynamic VAR planning problem is formulated as a multi-objective optimisation model with objectives including the investment cost, the expectation and robustness of the short-term voltage stability level. The complexity of the planning model is firstly reduced by selecting severe contingencies and potential buses, leading to a simplified multi-objective optimisation model. Latin hypercube sampling is then used for the uncertainty quantification. The simplified multi-objective optimisation model is then solved by the combination of a multi-objective evolutionary algorithm called ${\rm \epsilon }$e-NSGAII and extreme learning machine-based surrogate modelling with adaptive training data sampling. This combination significantly reduces the unaffordable computing burden. Simulations are carried on the IEEE 39-bus system, illustrating that the authors proposed methodology is reliable with high computational efficiency, and offering decision-makers planning solutions with high mean performance and strong robustness with respect to the short-term voltage stability level.

AEGEUS++: an energy-aware online partition skew mitigation algorithm for mapreduce in cloud

This paper investigates the partition skew problem at reduce phase in the mapreduce jobs. Our study summarize the skew problem in both offline and online manner. Offline is a heuristics based approach waits for the completion of map tasks and it involves computation overhead to estimate the partition size. In online approach, the overloaded tasks are distributed across other nodes that needs extra split and merge operations. These extra operations and ineffective utilization of resources in turn hamper the performance of the entire system. In this paper, we propose Aegeus++, to address the skew mitigation and adaptive data sampling problems for mapreduce jobs which enables to build an online prediction model with improved accuracy in minimal waiting time. In addition, we propose near linear skew detection and fine-grained Resource Allocation algorithms for identifying the skewed partition and allocating appropriate resources to reducers based on the partition size. Finally, our energy-aware opportunistic frequency tuning algorithm improves the performance of the reducer container on-fly, that can process the skewed data faster with minimal energy consumption. We evaluated Aegeus++ in the cloud setup by using benchmark datasets, compared its performance with native Hadoop and its other approaches. Based on our observation, Aegeus++ outperforms native Hadoop by 44% by maximizing its overall performance of the application and decreases the energy consumption by 37.67% when compared with existing approaches.

Scale-adaptive supervoxel-based random forests for liver tumor segmentation in dynamic contrast-enhanced CT scans.

Toward an efficient clinical management of hepatocellular carcinoma (HCC), we propose a classification framework dedicated to tumor necrosis rate estimation from dynamic contrast-enhanced CT scans. Based on machine learning, it requires weak interaction efforts to segment healthy, active and necrotic liver tissues. Our contributions are two-fold. First, we apply random forest (RF) on supervoxels using multi-phase supervoxel-based features that discriminate tissues based on their dynamic in response to contrast agent injection. Second, we extend this technique in a hierarchical multi-scale fashion to deal with multiple spatial extents and appearance heterogeneity. It translates in an adaptive data sampling scheme combining RF and hierarchical multi-scale tree resulting from recursive supervoxel decomposition. By concatenating multi-phase features across the hierarchical multi-scale tree to describe leaf supervoxels, we enable RF to automatically infer the most informative scales without defining any explicit rules on how to combine them. Assessment on clinical data confirms the benefits of multi-phase information embedded in a multi-scale supervoxel representation for HCC tumor segmentation. Dedicated but not limited only to HCC management, both contributions reach further steps toward more accurate multi-label tissue classification.

Marginal Space Deep Learning: Efficient Architecture for Volumetric Image Parsing.

Robust and fast solutions for anatomical object detection and segmentation support the entire clinical workflow from diagnosis, patient stratification, therapy planning, intervention and follow-up. Current state-of-the-art techniques for parsing volumetric medical image data are typically based on machine learning methods that exploit large annotated image databases. Two main challenges need to be addressed, these are the efficiency in scanning high-dimensional parametric spaces and the need for representative image features which require significant efforts of manual engineering. We propose a pipeline for object detection and segmentation in the context of volumetric image parsing, solving a two-step learning problem: anatomical pose estimation and boundary delineation. For this task we introduce Marginal Space Deep Learning (MSDL), a novel framework exploiting both the strengths of efficient object parametrization in hierarchical marginal spaces and the automated feature design of Deep Learning (DL) network architectures. In the 3D context, the application of deep learning systems is limited by the very high complexity of the parametrization. More specifically 9 parameters are necessary to describe a restricted affine transformation in 3D, resulting in a prohibitive amount of billions of scanning hypotheses. The mechanism of marginal space learning provides excellent run-time performance by learning classifiers in clustered, high-probability regions in spaces of gradually increasing dimensionality. To further increase computational efficiency and robustness, in our system we learn sparse adaptive data sampling patterns that automatically capture the structure of the input. Given the object localization, we propose a DL-based active shape model to estimate the non-rigid object boundary. Experimental results are presented on the aortic valve in ultrasound using an extensive dataset of 2891 volumes from 869 patients, showing significant improvements of up to 45.2% over the state-of-the-art. To our knowledge, this is the first successful demonstration of the DL potential to detection and segmentation in full 3D data with parametrized representations.

Adaptive sampling of potential-field data: A direct approach to compressive inversion

Compressive inversion uses computational algorithms that decrease the time and storage needs of a traditional inverse problem. Most compression approaches focus on the model domain, and very few, other than traditional downsampling focus on the data domain for potential-field applications. To further the compression in the data domain, a direct and practical approach to the adaptive downsampling of potential-field data for large inversion problems has been developed. The approach is formulated to significantly reduce the quantity of data in relatively smooth or quiet regions of the data set, while preserving the signal anomalies that contain the relevant target information. Two major benefits arise from this form of compressive inversion. First, because the approach compresses the problem in the data domain, it can be applied immediately without the addition of, or modification to, existing inversion software. Second, as most industry software use some form of model or sensitivity compression, the addition of this adaptive data sampling creates a complete compressive inversion methodology whereby the reduction of computational cost is achieved simultaneously in the model and data domains. We applied the method to a synthetic magnetic data set and two large field magnetic data sets; however, the method is also applicable to other data types. Our results showed that the relevant model information is maintained after inversion despite using 1%–5% of the data.

Multiplatform Ocean Exploration: Insights From the NEEMO Space Analog Mission

AbstractSince the beginning of space exploration, methods and protocols of exploration have been developed using space analogs on Earth to reduce research costs, develop safe deployment/retrieval protocols, and ready astronauts for hostile environments in less threatening settings. Space analogs are required as much as ever today as astronauts and scientists develop new tools and techniques for exploration, while working to address evolving mission objectives from low-earth orbit to deep-space exploration. This study examines coordinated human and robotic exploration at the Aquarius Underwater Habitat off of the coast of Key Largo, Florida, in support of the NEEMO 15 (NASA Extreme Environment Mission Operations) program. The exploration scheme presented in this work fuses (1) robotic precursor missions as a means of remote sensing data collection; (2) crowdsourcing to process immense amounts of data to identify key targets of interest that might be missed in the tight cycle of mission operations; and (3) human exploration to examine locations directly up close and collect physical samples that require involved sampling techniques. Autonomous underwater vehicles (AUVs) and single-person submersibles, called DeepWorkers™, were used as underwater analogs of robotic systems currently being used and human-operated vehicles (HOVs) proposed for use on a Near Earth Asteroid (NEA), the Moon, or Mars. In addition to operational lessons learned for space exploration that are directly applicable to ocean exploration, ocean floor mapping provides new levels of detail of benthic habitat critical for coral reef monitoring and management. Opportunistic (onsite adaptive) data sampling also took place by placing self-recording instrumentation onto each of the DeepWorkers, increasing the collection of scientific information during the submersible missions and contributing to mission planning for optimal and efficient use of expensive assets.

Consistency-driven data quality management of networked sensor systems

With more and more real deployments of wireless sensor network applications, we envision that their success is nonetheless determined by whether the sensor networks can provide a high quality stream of data over a long period. In this paper, we propose a consistency-driven data quality management framework called Orchis that integrates the quality of data into an energy efficient sensor system design. Orchis consists of four components, data consistency models, adaptive data sampling and process protocols, consistency-driven cross-layer protocols and flexible APIs to manage the data quality, to support the goals of high data quality and energy efficiency. We first formally define a consistency model, which not only includes temporal consistency and numerical consistency, but also considers the application-specific requirements of data and data dynamics in the sensing field. Next, we propose an adaptive lazy energy efficient data collection protocol, which adapts the data sampling rate to the data dynamics in the sensing field and keeps lazy when the data consistency is maintained. Finally, we conduct a comprehensive evaluation to the proposed protocol based on both a TOSSIM-based simulation and a real prototype implementation using MICA2 motes. The results from both simulation and prototype show that our protocol reduces the number of delivered messages, improves the quality of collected data, and in turn extends the lifetime of the whole network. Our analysis also implies that a tradeoff should be carefully set between data consistency requirements and energy saving based on the specific requirements of different applications.

Four‐dimensional cone beam CT with adaptive gantry rotation and adaptive data sampling

We have developed a new four-dimensional cone beam CT (4D-CBCT) on a Varian image-guided radiation therapy system, which has radiation therapy treatment and cone beam CT imaging capabilities. We adapted the speed of gantry rotation time of the CBCT to the average breath cycle of the patient to maintain the same level of image quality and adjusted the data sampling frequency to keep a similar level of radiation exposure to the patient. Our design utilized the real-time positioning and monitoring system to record the respiratory signal of the patient during the acquisition of the CBCT data. We used the full-fan bowtie filter during data acquisition, acquired the projection data over 200 deg of gantry rotation, and reconstructed the images with a half-scan cone beam reconstruction. The scan time for a 200-deg gantry rotation per patient ranged from 3.3 to 6.6 min for the average breath cycle of 3-6 s. The radiation dose of the 4D-CBCT was about 1-2 times the radiation dose of the 4D-CT on a multislice CT scanner. We evaluated the 4D-CBCT in scanning, data processing and image quality with phantom studies. We demonstrated the clinical applicability of the 4D-CBCT and compared the 4D-CBCT and the 4D-CT scans in four patient studies. The contrast-to-noise ratio of the 4D-CT was 2.8-3.5 times of the contrast-to-noise ratio of the 4D-CBCT in the four patient studies.

Robust classification of animal tracking data

This paper describes an application of the K-means classification algorithm to categorize animal tracking data into various classes of behavior. It was found that, even without explicit consideration of biological factors, the clustering algorithm repeatably resolved tracking data from cows into two groups corresponding to active and inactive periods. Furthermore, it is shown that this classification is robust to a large range of data sampling intervals. An adaptive data sampling algorithm is suggested for improving the efficiency of both energy and memory usage in animal tracking equipment.

Enhancing Flow Field Measurements Through Adaptive Multidimensional Data Sampling

A way to gain insight into the flow field conditions in turbomachinery is by carrying out a series of point measurements in a cross section of the flow, for example, with a miniature multihole pressure probe. A problem commonly encountered in situations like these is the selection of a suitable measurement grid layout and density for obtaining all essential information in a cost-effective and timely manner. In order to achieve the latter, a novel adaptive multidimensional data sampling technique has been developed at Cranfield University. This paper describes the underlying principles of this technique, the algorithms utilized, and the results obtained during its successful application to data sets of two different flow fields in a high-speed research compressor.