Big Data Volume Research Articles

MapReduce-Based D_ELT Framework to Address the Challenges of Geospatial Big Data

The conventional extracting–transforming–loading (ETL) system is typically operated on a single machine not capable of handling huge volumes of geospatial big data. To deal with the considerable amount of big data in the ETL process, we propose D_ELT (delayed extracting–loading –transforming) by utilizing MapReduce-based parallelization. Among various kinds of big data, we concentrate on geospatial big data generated via sensors using Internet of Things (IoT) technology. In the IoT environment, update latency for sensor big data is typically short and old data are not worth further analysis, so the speed of data preparation is even more significant. We conducted several experiments measuring the overall performance of D_ELT and compared it with both traditional ETL and extracting–loading– transforming (ELT) systems, using different sizes of data and complexity levels for analysis. The experimental results show that D_ELT outperforms the other two approaches, ETL and ELT. In addition, the larger the amount of data or the higher the complexity of the analysis, the greater the parallelization effect of transform in D_ELT, leading to better performance over the traditional ETL and ELT approaches.

Decomposition approach to design of microwave devices

It is considered a decomposition approach being the basic part during simulation of microwave engineering devices used for automatic designing systems. Actuality of minimal autonomous blocks method is confirmed by the fact that using direct formalization it is impossible to perform effective calculations because of complexity of end tasks formalization and big data volume. In this case the end task corresponds to the simulated object in a whole. Decomposition approach means that the complete system is divided in autonomous blocks. The parts of microwave devices such as waveguides and integrated circuits are considered as examples. As well it is considered the discretization method for end tasks of electrodynamics. This method is called as the method of minimum autonomous blocks that principally differs from use of difference schemebi and end elements. Types of autonomous blocks and use of the method of minimum autonomous blocks to solve non-electrodynamic and non-wave tasks are considered.

Ferroelectric Memristive Networks for Dimensionality Reduction: A Process for Effectively Classifying Cancer Datasets

In this work, a copper-doped (5%) zinc oxide (Cu:ZnO) ferroelectric materials-based memristor model was realized and it was employed to develop principal component analysis (PCA), a data dimension reduction technique. The developed PCA was utilized to efficaciously classify breast cancer datasets, which are considered as complex and big volumes of data. It was found that the controllable memristance variations were analogous to the weight modulations in the implemented neural network-based learning systems. Sanger’s rule was utilized to achieve unsupervised online learning in order to generate the principal components. On one side, the developed memristor-based PCA network was found to be effective to isolate distinct breast cancer classes with a high classification accuracy of 97.77% and the error in the classification of malignant cases as benign of 0.529%, a significantly low value. On the other side, the power dissipation was found to be 0.27 µW, which suggests the proposed memristive network is suitable for low-power applications. Further, a comparison was established with other existing non-memristor and non-PCA-based data classification systems. Furthermore, the devised less complex equations to implement PCA on this memristive crossbar array could be employed to implement any neural network algorithm.

DECISION MAKING ON DISASTER MANAGEMENT IN AGRICULTURE WITH SENTINEL APPLICATIONS

Abstract. Climate change and increase of extreme weather events, besides the numerous consequences, affect significantly and put in risk the agriculture sectors. Natural disasters, such as floods and wildfires, are responsible for a great loss in agriculture production. National governments together with international bodies make an important effort to cooperate towards the response and resilience when a disaster occurs. In this frame the European Earth Observation Programme - Copernicus provides a series of observation data, in-situ measurements and services related, amongst others, to different types of disasters. Concerning the availability of this big volume of observation data, the aim of DiAS (Disaster and Agriculture Sentinel Applications) project is to revise the existing knowledge on remote sensing methods for mapping the extent of natural and/or man-made disaster over agricultural areas and propose improvements. The developed methodology will be implemented in a Decision Support System (DSS), which will be freely available and easy-to-use by non-experts. In this paper, the developed methodology focuses on mapping floods over agricultural areas. Sentinel-1 and Sentinel-2 imagery are used as input information for the comparison analysis before and after the event. The reference for results’ evaluation is the corresponding information delivered by Copernicus Emergency Management Service (EMS). Although, the evaluation results are in good agreement when they could be used, a reference of higher accuracy is needed in order to estimate accurately the quality of the output products.

The geographical region of origin determines the phagocytic vulnerability of Lichtheimia strains.

Mucormycoses are life-threatening infections that affect patients suffering from immune deficiencies. We performed phagocytosis assays confronting various strains of Lichtheimia species with alveolar macrophages, which form the first line of defence of the innate immune system. To investigate 17 strains from four different continents in a comparative fashion, transmitted light and confocal fluorescence microscopy was applied in combination with automated image analysis. This interdisciplinary approach enabled the objective and quantitative processing of the big volume of image data. Applying machine-learning supported methods, a spontaneous clustering of the strains was revealed in the space of phagocytic measures. This clustering was not driven by measures of fungal morphology but rather by the geographical origin of the fungal strains. Our study illustrates the crucial contribution of machine-learning supported automated image analysis to the qualitative discovery and quantitative comparison of major factors affecting host-pathogen interactions. We found that the phagocytic vulnerability of Lichtheimia species depends on their geographical origin, where strains within each geographic region behaved similarly, but strongly differed amongst the regions. Based on this clustering, we were able to also classify clinical isolates with regard to their potential geographical origin.

RETRACTED: Erratum to “Smart health monitoring and management system: Toward autonomous wearable sensing for Internet of Things using big data analytics [Future Gener. Comput. Syst. 91 (2019) 611–619

Abstract The growing gap between users and the Big Data analytics requires innovative tools that address the challenges faced by big data volume, variety, and velocity. Therefore, it becomes computationally inefficient to analyze such massive volume of data. Recent advances and development in the field of Internet of Things (IoT) providing a unique way of exploiting the role of healthcare systems. Also, the role of healthcare in the IoT is studied widely since it plays a major role in the advances of human life that deals with the health regulations. The continuous involvement of heterogeneous devices in the IoT poses many challenges, i.e., empowering the IoT devices used for the healthcare system, aggregation and processing of real-time data. Therefore, based on such constraint, in this paper, we propose a novel architecture for a healthcare system based on energy harvesting technique that extends the device lifetime. Moreover, the healthcare system is supported by an architecture that welcomes both real-time and offline data. To handle such data, the architecture provides a novel decision model that process big data being generated by IoT devices. The data is considered as heterogeneous processed by the proposed layered architecture for the healthcare system. Furthermore, the feasibility and efficiency of the proposed system are implemented on Hadoop single node setup on UBUNTU 14.04 LTS core TM i5 machine with 3.2 GHz processor and 4 GB memory. Sample medical, sensory datasets are tested on the proposed system. Finally, the results show that the proposed system architecture efficiently process, analyze, and integrates different datasets efficiently and triggers an alarm to provide safety to the community.

Application of EDM to Understand the Online Students' Behavioral Pattern

In distance learning, the professor cannot see that the students are having trouble with a subject, and can fail to perceive the problem in time to intervene. However, in learning management systems (LMS's) a large volume of data regarding online access, participation and progress can be registered and collected allowing analysis based on students' behavioral patterns. As traditional methods have a limited capacity to extract knowledge from big volumes of data, educational data mining (EDM) arises as a tool to help teachers interpreting the behavior of students. The objective of the present article is to describe the application of educational data mining technics aiming to obtain relevant knowledge of students' behavioral patterns in an LMS for an online course, with 1,113 students enrolled. This paper applies two algorithms on educational context, decision tree and clustering, unveiling unknown relevant aspects to professors and managers, such as the most important examinations that contribute to students' approval as well as the most significant attributes to their success.

Machine Learning in Medical Imaging

The papers in this special issue focus on machine learning for use in medical image processing applications. The use of machine learning in this area has become indispensable in diagnosis and treatment of many diseases. With advances in new imaging techniques, the need to take full advantage of abundant images draws more and more attention. Machine learning, including deep learning particularly, provides us a new paradigm to learn and to utilize the overwhelming volume of big imaging data smartly. Nowadays, machine learning in medical imaging has become one of the most promising and growing fields of research. The main aim of this special issue is to help advance the scientific research within the broad field of machine learning in medical imaging. The special issue was planned in conjunction with the International Workshop on Machine Learning in Medical Imaging (MLMI) 2017.

Ship Spatiotemporal Key Feature Point Online Extraction Based on AIS Multi-Sensor Data Using an Improved Sliding Window Algorithm.

Large volumes of automatic identification system (AIS) data provide new ideas and methods for ship data mining and navigation behavior pattern analysis. However, large volumes of big data have low unit values, resulting in the need for large-scale computing, storage, and display. Learning efficiency is low and learning direction is blind and untargeted. Therefore, key feature point (KFP) extraction from the ship trajectory plays an important role in fields such as ship navigation behavior analysis and big data mining. In this paper, we propose a ship spatiotemporal KFP online extraction algorithm that is applied to AIS trajectory data. The sliding window algorithm is modified for application to ship navigation angle deviation, position deviation, and the spatiotemporal characteristics of AIS data. Next, in order to facilitate the subsequent use of the algorithm, a recommended threshold range for the corresponding two parameters is discussed. Finally, the performance of the proposed method is compared with that of the Douglas–Peucker (DP) algorithm to assess its feature extraction accuracy and operational efficiency. The results show that the proposed improved sliding window algorithm can be applied to rapidly and easily extract the KFPs from AIS trajectory data. This ability provides significant benefits for ship traffic flow and navigational behavior learning.

ITISS: an efficient framework for querying big temporal data

In the real word, temporal data can be found in many applications, and it is rapidly increasing nowadays. It is urgently important and challenging to manage and operate big temporal data efficiently and effectively, due to the large volume of big temporal data and the real-time response requirement. Processing big temporal data using a distributed system is a desired choice, since a single-machine based system usually has the limited computing ability. Nevertheless, existing distributed systems or methods either are disk-based solutions, or cannot support native queries, which may not well meet the demands of low latency and high throughput. To attack these issues, this article suggests a new approach to handle big temporal data. Our approach is an In-memory based Two-level Index Solution in Spark, dubbed as ITISS. The proposed framework of our solution is easily understood and implemented, but without loss of effectiveness and efficiency. Based on the proposed framework, this article develops targeted algorithms for handling time travel, temporal aggregation, and temporal join queries, respectively. We have implemented our framework in Apache Spark, extended the Apache Spark SQL to support declarative SQL interface that enables users to perform temporal queries with a few lines of SQL statements, and conducted extensive experiments to verify the performance of our solution. The experimental results, based on both real and synthetic datasets, consistently demonstrate that our proposed solution is efficient and competitive for processing big temporal data.

DATALET: An approach to manage big volume of data in cyber foraged environment

In the new era of cloud computing, the users get various services from the cloud. In recent years, the increasing volume of data specially from pervasive devices has become a great matter of concern. In mobile computing, cloudlets act as a shadow image of the data centers and provide low latency cloud environment. In this work, the authors have proposed an approach called DATALET that deals with distribution of data by utilizing the processing and storage resources of big intermittent networks. In DATALET, the cloudlets act as central managers for data management. DATALET provides a robust architecture which is fault-tolerant and also has a cloudlet job scheduler. The cloudlets maintain the information based on the availability of the user at a particular location and also utilize their computational resources. The model is simulated using NS-3 and will also provide the service in an environment where the network does not exist. The results indicate that DATALET approach has higher performance in terms of latency, Internet outgoing bandwidth for the users and resource utilization of the user devices.

A Marine Environment Early Warning Algorithm Based on Marine Data Sampled by Multiple Underwater Gliders

This study analyzes and summarizes seven main characteristics of the marine data sampled by multiple underwater gliders. These characteristics such as the big data volume and data sparseness make it extremely difficult to do some meaningful applications like early warning of marine environment. In order to make full use of the sea trial data, this paper gives the definition of two types of marine data cube which can integrate the big marine data sampled by multiple underwater gliders along saw-tooth paths, and proposes a data fitting algorithm based on time extraction and space compression (DFTS) to construct the temperature and conductivity data cubes. This research also presents an early warning algorithm based on data cube (EWDC) to realize the early warning of a new sampled data file. Experiments results show that the proposed methods are reasonable and effective. Our work is the first study to do some realistic applications on the data sampled by multiple underwater vehicles, and it provides a research framework for processing and analyzing the big marine data oriented to the applications of underwater gliders.

Evaluating the Impact of Temperature on Dynamic Fault Behaviour of FinFET-Based SRAMs with Resistive Defects

In recent years, FinFET-based Static Random Access Memories (SRAMs) have become a viable solution to provide the storage of big data volume in Systems-on-Chip (SoCs) as well as to assure high performance deep-scaled devices. As consequence, FinFET-based SRAMs are an extremely viable solution to guarantee the high-performance requirements of deep-scaled devices. However, FinFET-based SRAMs can also be affected by resistive defects that may lead to dynamic faults, which are considered one of the most important causes of manufacturing test escape in deep-submicron technologies. Hence, this paper proposes to evaluate the impact of temperature on the dynamic faulty behaviour of 20 nm FinFET-based SRAM cells affected by weak resistive defects. In more details, the critical resistances and the number of consecutive operations necessary to sensitize faults at the logic level are investigated. Additionally, the concept of Dynamic Behaviour Window (DBW) is defined. Results showed that temperature plays a major part in the sensitization of dynamic faults. Thus, the occurrence of dynamic faults has been mapped combining temperature as well as defect size, and the DBW has been defined considering resistive defects. The proposed evaluation helps to understand the behaviour of dynamic faults in memory cells even better and therefore can be used to improve the test procedures for deep-scaled FinFET memory devices. The proposed analysis has been performed using HSPICE simulations adopting a 20 nm Predictive Technology Model (PTM) of multi-gate transistors based on bulk FinFET.

Transfer learning to localise a continental soil vis-NIR calibration model

The rapid development in NIR and information technologies saw the development of various initiatives that have generated large scale databases of soil spectroscopy globally. Models generated within a specific spectral or geographical domain should be carefully used in other contexts since they may lose their validity. This includes the application of a global, continental or national spectral libraries to local areas or regions. Both, global and local models are valuable and, ideally, we would like to transfer some of the rules learnt by the more general global models to a local domain. In machine learning, the process of sharing intra-domain information is known as transfer learning. This paper aims to describe and evaluate the effectiveness of transfer learning to “localise” a general soil spectral model. The transfer process consists in, first, training a model with a big volume of data covering a diverse group of cases. Second, some layers of the trained neural network are used to build a local model, which is fine-tuned by using a smaller amount of local data. We demonstrated this method using the LUCAS database, an European dataset, comprising spectral data from 21 countries. For each country, we generated three models: a) Global, with data from all except the country of interest; b) Local, with data from the country; and c) Transfer, pre-trained as the Global model and fine-tuned with data from the country. The results showed that the Transfer model can lower the error (expressed as RMSE) 91% of the cases, with a mean reduction of RMSE: 10.5, 11.8, 12.0 and 11.5% for organic carbon, cation exchange capacity, clay content and pH, respectively. This paper demonstrates the usefulness of transfer learning for soil spectroscopy, which will enhance the use of global spectral libraries for local application.

Optimizing Partition Granularity, Membership Function Parameters, and Rule Bases of Fuzzy Classifiers for Big Data by a Multi-objective Evolutionary Approach

Classical data mining algorithms are considered inadequate to manage the volume, variety, velocity, and veracity aspects of big data. The advent of a number of open-source cluster-computing frameworks has opened new interesting perspectives for handling the volume and velocity features. In this context, thanks to their capability of coping with vague and imprecise information, distributed fuzzy models appear to be particularly suitable for handling the variety and veracity features of big data. Moreover, the interpretability of fuzzy models may assume a particular relevance in the context of big data mining. In this work, we propose a novel approach for generating, out of big data, a set of fuzzy rule–based classifiers characterized by different optimal trade-offs between accuracy and interpretability. We extend a state-of-the-art distributed multi-objective evolutionary learning scheme, implemented under the Apache Spark environment. In particular, we exploit a recently proposed distributed fuzzy decision tree learning approach for generating an initial rule base that serves as input to the evolutionary process. Furthermore, we integrate the evolutionary learning scheme with an ad hoc strategy for the granularity learning of the fuzzy partitions, along with the optimization of both the rule base and the fuzzy set parameters. Experimental investigations show that the proposed approach is able to generate fuzzy rule–based classifiers that are significantly less complex than the ones generated by the original multi-objective evolutionary learning scheme, while keeping the same accuracy levels.

Embedding of Genes Using Cancer Gene Expression Data: Biological Relevance and Potential Application on Biomarker Discovery.

Artificial neural networks (ANNs) have been utilized for classification and prediction task with remarkable accuracy. However, its implications for unsupervised data mining using molecular data is under-explored. We found that embedding can extract biologically relevant information from The Cancer Genome Atlas (TCGA) gene expression dataset by learning a vector representation through gene co-occurrence. Ground truth relationship, such as cancer types of the input sample and semantic meaning of genes, were showed to retain in the resulting entity matrices. We also demonstrated the interpretability and usage of these matrices in shortlisting candidates from a long gene list as in the case of immunotherapy response. 73 related genes are singled out while the relatedness of 55 genes with immune checkpoint proteins (PD-1, PD-L1, and CTLA-4) are supported by literature. 16 novel genes (ACAP1, C11orf45, CD79B, CFP, CLIC2, CMPK2, CXCR2P1, CYTIP, FER, MCTO1, MMP25, RASGEF1B, SLFN12, TBC1D10C, TRAF3IP3, TTC39B) related to immune checkpoint proteins were identified. Thus, this method is feasible to mine big volume of biological data, and embedding would be a valuable tool to discover novel knowledge from omics data. The resulting embedding matrices mined from TCGA gene expression data are interactively explorable online (http://bit.ly/tcga-embedding-cancer) and could serve as an informative reference for gene relatedness in the context of cancer and is readily applicable to biomarker discovery of any molecular targeted therapy.

Parallel K Nearest Neighbor Matching for 3D Reconstruction

In recent years, a 3D reconstruction based on structure from motion (SFM) has attracted much attention from the communities of computer vision and graphics. It is well known that the speed and quality of SFM systems largely depend on the technique of feature tracking. If a big volume of image data is inputted for SFM, the speed of this SFM system would become very slow. And, this problem becomes severer for large-scale scenes, which typically needs to capture several thousands of images to recover the point-cloud model of the scene. However, none of the existing methods fully addresses the problem of fast feature tracking. Brute force matching is capable of producing correspondences for small-scale scenes but often getting stuck in repeated features. Hashing matching can only deal with middle-scale scenes and is not capable of large-scale scenes. In this paper, we propose a new feature tacking method working in a parallel manner rather than in a single thread scheme. Our method consists of steps of keypoint detection, descriptor computing, descriptor matching by parallel $k$ -nearest neighbor (Parallel-KNN) search, and outlier rejecting. This method is able to rapidly match a big volume of keypoints and avoids to consume high computation time, then yielding a set of correct correspondences. We demonstrate and evaluate the proposed method on several challenging benchmark datasets, including those with highly repeated features, and compare to the state-of-the-art methods. The experimental results indicate that our method outperforms the compared methods in both efficiency and effectiveness.

Limitations of information extraction methods and techniques for heterogeneous unstructured big data

During the recent era of big data, a huge volume of unstructured data are being produced in various forms of audio, video, images, text, and animation. Effective use of these unstructured big data is a laborious and tedious task. Information extraction (IE) systems help to extract useful information from this large variety of unstructured data. Several techniques and methods have been presented for IE from unstructured data. However, numerous studies conducted on IE from a variety of unstructured data are limited to single data types such as text, image, audio, or video. This article reviews the existing IE techniques along with its subtasks, limitations, and challenges for the variety of unstructured data highlighting the impact of unstructured big data on IE techniques. To the best of our knowledge, there is no comprehensive study conducted to investigate the limitations of existing IE techniques for the variety of unstructured big data. The objective of the structured review presented in this article is twofold. First, it presents the overview of IE techniques from a variety of unstructured data such as text, image, audio, and video at one platform. Second, it investigates the limitations of these existing IE techniques due to the heterogeneity, dimensionality, and volume of unstructured big data. The review finds that advanced techniques for IE, particularly for multifaceted unstructured big data sets, are the utmost requirement of the organizations to manage big data and derive strategic information. Further, potential solutions are also presented to improve the unstructured big data IE systems for future research. These solutions will help to increase the efficiency and effectiveness of the data analytics process in terms of context-aware analytics systems, data-driven decision-making, and knowledge management.

Analysis of NoSQL Schema Design approaches using HBase for GIS Data

In this digital era, geospatial data plays vital role in developing various citizen centric services for sustainable development of the society.There are various portals available where people can tag the locations for further use but they are proprietary based. This data is satisfying all three Vs, Volume, velocity and variety of Big data hence to manage such a voluminous data No SQL data bases should be used rather than RDBMS. Main contribution of this paper is to discuss possible approaches of data base design using HBase No SQL data base and analyzing these approaches specifically for crowdsourcing enabled open source Geographical Information portal.

How Physical Exercise Level Affects Sleep Quality? Analyzing Big Data Collected from Wearables

Physical exercise and sleep have independent, yet synergistic, impacts on the health. However, the effects of acute exercise level on sleep quality have not been well investigated. We utilize statistical methods to investigate the differences of exercise level between the good and bad sleep nights. Our results present a complex interrelation between physical exercise and sleep quality with analyzing large personal data sets collected from wearables. As far as we know, this is the first study to investigate insights of interrelation of physical exercise and sleep quality based on a big volume of data collected from wearable devices of real users.