Fast Data Processing Research Articles

Mass data processing and multidimensional database management based on deep learning

Abstract With the rapid development of the Internet of Things, the requirements for massive data processing technology are getting higher and higher. Traditional computer data processing capabilities can no longer deliver fast, simple, and efficient data analysis and processing for today’s massive data processing due to the real-time, massive, polymorphic, and heterogeneous characteristics of Internet of Things data. Mass heterogeneous data of different types of subsystems in the Internet of Things need to be processed and stored uniformly, so the mass data processing method is required to be able to integrate multiple different networks, multiple data sources, and heterogeneous mass data and be able to perform processing on these data. Therefore, this article proposes massive data processing and multidimensional database management based on deep learning to meet the needs of contemporary society for massive data processing. This article has deeply studied the basic technical methods of massive data processing, including MapReduce technology, parallel data technology, database technology based on distributed memory databases, and distributed real-time database technology based on cloud computing technology, and constructed a massive data fusion algorithm based on deep learning. The model and the multidimensional online analytical processing model of the multidimensional database based on deep learning analyze the performance, scalability, load balancing, data query, and other aspects of the multidimensional database based on deep learning. It is concluded that the accuracy of multidimensional database query data is as high as 100%, and the accuracy of the average data query time is only 0.0053 s, which is much lower than the general database query time.

Millisecond autofocusing microscopy using neuromorphic event sensing

Rapid autofocusing is essential for many microscopic imaging applications. Existing methods either require complicated hardware implementations or slow z-stack image acquisition. In this work, we develop a new approach to achieve fast autofocusing using the neuromorphic event sensing technology, which detects the sparse brightness changes asynchronously and responds quickly to the specimen movement. A simple yet efficient autofocusing system is tailored for fast acquisition and processing of the non-redundant event data, allowing for autofocusing in only tens of milliseconds, which is thousands of times faster than current technologies. Experimental results show a substantial performance improvement and capability for biopsy specimen inspections. To the best of our knowledge, this is the first reported work on a neuromorphic methodology for autofocusing microscopy.

Leveraging R (LevR) for fast processing of mass spectrometry data and machine learning: Applications analyzing fingerprints and glycopeptides

Applying machine learning strategies to interpret mass spectrometry data has the potential to revolutionize the way in which disease is diagnosed, prognosed, and treated. A persistent and tedious obstacle, however, is relaying mass spectrometry data to the machine learning algorithm. Given the native format and large size of mass spectrometry data files, preprocessing is a critical step. To ameliorate this challenge, we sought to create an easy-to-use, continuous pipeline that runs from data acquisition to the machine learning algorithm. Here, we present a start-to-finish pipeline designed to facilitate supervised and unsupervised classification of mass spectrometry data. The input can be any ESI data set collected by LC-MS or flow injection, and the output is a machine learning ready matrix, in which each row is a feature (an abundance of a particular m/z), and each column is a sample. This workflow provides automated handling of large mass spectrometry data sets for researchers seeking to implement machine learning strategies but who lack expertise in programming/coding to rapidly format the data. We demonstrate how the pipeline can be used on two different mass spectrometry data sets: 1) ESI-MS of fingerprint lipid compositions acquired by direct infusion and, 2) LC-MS of IgG glycopeptides. This workflow is uncomplicated and provides value via its simplicity and effectiveness.

Analysis of fast data processing methods

Analysis of fast data processing methods

Smartphone-Based Quantitative Analysis of Protein Array Signals for Biomarker Detection in Lupus.

Fluorescence-based microarray offers great potential in clinical diagnostics due to its high-throughput capability, multiplex capabilities, and requirement for a minimal volume of precious clinical samples. However, the technique relies on expensive and complex imaging systems for the analysis of signals. In the present study, we developed a smartphone-based application to analyze signals from protein microarrays to quantify disease biomarkers. The application adopted Android Studio open platform for its wide access to smartphones, and Python was used to design a graphical user interface with fast data processing. The application provides multiple user functions such as “Read”, “Analyze”, “Calculate” and “Report”. For rapid and accurate results, we used ImageJ, Otsu thresholding, and local thresholding to quantify the fluorescent intensity of spots on the microarray. To verify the efficacy of the application, three antigens each with over 110 fluorescent spots were tested. Particularly, a positive correlation of over 0.97 was achieved when using this analytical tool compared to a standard test for detecting a potential biomarker in lupus nephritis. Collectively, this smartphone application tool shows promise for cheap, efficient, and portable on-site detection in point-of-care diagnostics.

Survey on Service Placement, Provisioning, and Composition for Fog-Based IoT Systems

In the modern IoT era, time and resource critical applications are more prevalent. Though cloud computing enables fast processing of data and producing results, the time and network overhead are huge. Fog based solutions have proved to eliminate this overhead. However, establishing a scalable fog based architecture for IoT applications involves a number of challenges such as guaranteeing QoS, reducing operational costs etc. Hence a viable fog based architecture must involve optimal service placement facilitated by suitable schemes for resource estimation and reservation, QoS monitoring, data interchange and integration. This paper surveys various service placement and optimization approaches proposed in the literature and compares them. This paper also explores various components that are required for composing a viable fog based architecture and presents a representative fog provisioning framework. Relevance of modern technologies and tools such as RESTful architecture, SDN, Kafka, RabbitMQ, Zookeeper in establishing an efficient framework for fog based IoT applications is discussed

Development of a sterile Interaction Device during Image guided minimal-invasive Interventions.

For correct setting and control of imaging during image-guided Magnetic Resonance (MR) procedures (e.g. changing the slice orientation), the radiologist currently depends heavily on the assistants in the control room. The communication between the surgeon and the assistant is difficult due to acoustic noise generated by the MR gradients. The development for a dedicated interventional interaction device is addressed in this work, which includes the development and evaluation of a sterile, MR-compatible capacitive gesture controller. The gesture controller consists of a surgical drape with four capacitive sensors, that are made of conductive fabric. These sensors are based on the principle of electrical capacitance between isolated conductive structures. Measurements of these capacitances are exploited to calculate the distance of a hand to one of the sensors. By comparing the distances of the hand to each sensor, simple swipe gestures can be detected. The sensor unit is connected to a micro-controller, which allows direct and fast processing of the incoming data. To evaluate the prototype, a user study with 25 participants took place. Each user had to perform the four swiping gestures and the hold gesture in a specific order. The gesture controller is capable of registering five different gestures (left-right, right-left, up-down, down-up, hold). A survey of the second prototype showed, that in 92,4% of cases, the correct gesture was registered. Clinical relevance This interaction device allows the practicing physician to manipulate the imaging independently and in a sterile manner, thus enabling a significant improvement in work-flow.

Grouping the Distribution of Diarrhea Based on the Public Health Centers in Binjai City Using the Clustering Method

Diarrhea is one of the endemic diseases that occurs throughout the year and is one of the highest causes of death for everyone, especially children under five in Indonesia. The purpose of this thesis is to analyze the spread of diarrheal disease by forming a grouping of the spread of diarrheal disease and to analyze the characteristics of the spread of diarrheal disease. With the development of technology that is increasingly sophisticated and easy to use, such technological developments motivate us to know the importance of using computers in fast and practical data processing. Clustering is a data analysis method, which is often included as one of the Data Mining methods, whose purpose is to group data with the same characteristics and data with different characteristics into others.

A neural network with competitive layers for character recognition

A structure and functioning mechanisms of a neural network with competitive layers are described. The network is intended to solve the character recognition task. The network consists of several competitive layers of neurons. Each layer is a neural network consisting of a number of neurons represented as a layer. The number of neural layers is equal to the number of recognized classes. All neural layers have one-to-one correspondence with one another and with the input raster. The neurons of every layer have mutual lateral learning connections, which weights are modified during the learning process. There is a competitive (inhibitory) relationship between all neural layers. This competitive interaction is realized by means of a “winner-take-all” (WTA) procedure which aim is to select the layer with the highest level of neural activity.Validation of the network has been done in experiments on recognition of handwritten digits of the MNIST database. The experiments have demonstrated that its error rate is few less than 2%, which is not a high result, but it is compensated by rather fast data processing and a very simple structure and functioning mechanisms.

Internet of Things data management: A systematic literature review, vision, and future trends

SummaryThe Internet of Things (IoT) is expected to connect devices with unique identifiers over a network to create an equilibrium system with high speeds and volumes of data while presenting an interoperability challenge. The IoT data management system is indispensable for attaining effective and efficient performance because IoT sensors generate and collect large amounts of data used to express large data sets. IoT data management has been analyzed from various perspectives in numerous studies. In this study, a Systematic Literature Review (SLR) method was used to investigate the various topics and key areas that have recently emerged in IoT data management. This study aims to classify and evaluate studies published between 2015 and 2021 in IoT data management. Therefore, the classification of studies includes five categories, data processing, data smartness application, data collection, data security, and data storage. Then, studies in each field are compared based on the proposed classification. Each study investigates novel findings, simulation/implementation, data set, application domain, experimental results, advantages, and disadvantages. In addition, the criteria for evaluating selected articles for each domain of IoT data management are examined. Big data accounts for the highest percentage of data processing fields in IoT data management, at 34%. In addition, fast data processing, distributed data, artificial intelligence data with 22%, and data uncertainty analysis account for 11% of the data processing field. Finally, studies highlight the challenges of IoT data management and its future directions.

Strain induced electromagnetic coupling in tri-phase multiferroic composites 0.8[(1-x)Fe2O3–xPbZrO3]+0.2Cr2O3 for multistate devices

Multiferroic composites are highly desirable for next-generation multifunctional smart devices because of their low power consumption, fast data processing, and nonvolatile nature in memory devices. Here we synthesized a series of tri-phase composite of Fe2O3, PbZrO3, and Cr2O3 with general formula 0.8[(1-x)Fe2O3–xPbZrO3]+0.2Cr2O3 (x = 0.0, 0.33, 0.66 and 1.0) via a chemical approach. Individual oxides are synthesized by the sol-gel auto-combustion method and the solid-state route is used to mix them. Through structural analysis, the formation of tri-phase composites is confirmed by the presence of peaks belonging to each phase. The morphological study suggested that the composites have a homogeneous distribution of grain size over the surface with small voids. To investigate the ferroelectric nature of these composites the saturation polarization, remanent polarization, and coercivity are calculated from their P-E loops. Using these values, the switching charge density and recoverable energy density are evaluated which highlighted the importance of these composites for fast switching and energy storage devices. The response of variation in polarization by the magnetic field is mapped by magneto-electric coupling measurement. Our results show that these tri-phase composites with ferroelectric and magnetoelectric response are potential candidates for high-density data storage and ME multifunctional devices.

RESEARCH IMPLEMENTATION OF BEIDOU DATA FAST PROCESSING OF LARGE-SCALE GNSS REFERENCE STATION NETWORK

Abstract. GNSS reference station network is the core infrastructure for the establishment ,maintenance, renewal and service of national geodetic reference framework. At present ,nearly 3000 reference stations have been constructed in the national resources system ,of which about 2500 stations have the continuous observation capacity of BDS signal. In the face of daily massive observation data, the traditional sub-network and fully combined baselines can not meet the needs of rapid data processing for the maintenance of geodetic coordinate framework.In this paper, the high-precision and fast processing of BDS observation data of large-scale GNSS reference station network (≥ 300 stations) is realized through modular design, non difference model, parameter management to be estimated and flexible ambiguity processing. Through the whole network calculation of the 7-day BDS observation data of about 500 MGEX stations and national reference stations that selected all around the world from 211 to 217 days in 2017, the positioning results show that the average RMS values of the coordinates in the three directions of station N,E and U are ±2.3mm, ±2.8mm and ±4.6mm respectively. The orbit determination results show that the average accuracy of GPS satellite orbit in three directions is better than ±2cm; BDS MEO / IGSO track accuracy is better than ±20cm and GEO track accuracy is better than ±200cm.

Distributed big data analysis using spark parallel data processing

Nowadays, the big data marketplace is rising rapidly. The big challenge is finding a system that can store and handle a huge size of data and then processing that huge data for mining the hidden knowledge. This paper proposed a comprehensive system that is used for improving big data analysis performance. It contains a fast big data processing engine using Apache Spark and a big data storage environment using Apache Hadoop. The system tests about 11 Gigabytes of text data which are collected from multiple sources for sentiment analysis. Three different machine learning (ML) algorithms are used in this system which is already supported by the Spark ML package. The system programs were written in Java and Scala programming languages and the constructed model consists of the classification algorithms as well as the pre-processing steps in a figure of ML pipeline. The proposed system was implemented in both central and distributed data processing. Moreover, some datasets manipulation manners have been applied in the system tests to check which manner provides the best accuracy and time performance. The results showed that the system works efficiently for treating big data, it gains excellent accuracy with fast execution time especially in the distributed data nodes.

Photonic (computational) memories: tunable nanophotonics for data storage and computing

Abstract The exponential growth of information stored in data centers and computational power required for various data-intensive applications, such as deep learning and AI, call for new strategies to improve or move beyond the traditional von Neumann architecture. Recent achievements in information storage and computation in the optical domain, enabling energy-efficient, fast, and high-bandwidth data processing, show great potential for photonics to overcome the von Neumann bottleneck and reduce the energy wasted to Joule heating. Optically readable memories are fundamental in this process, and while light-based storage has traditionally (and commercially) employed free-space optics, recent developments in photonic integrated circuits (PICs) and optical nano-materials have opened the doors to new opportunities on-chip. Photonic memories have yet to rival their electronic digital counterparts in storage density; however, their inherent analog nature and ultrahigh bandwidth make them ideal for unconventional computing strategies. Here, we review emerging nanophotonic devices that possess memory capabilities by elaborating on their tunable mechanisms and evaluating them in terms of scalability and device performance. Moreover, we discuss the progress on large-scale architectures for photonic memory arrays and optical computing primarily based on memory performance.

A systematic review of task scheduling approaches in fog computing

AbstractDue to increased use of IoT devices and data sensors a huge amount of data is being produced for processing in real‐time. Fog computing has evolved as a solution for fast processing of data. To complete the data processing as per requirement of user, it is processed at fog nodes which are near the user. To complete work in a specified time with limited resources, task scheduling is performed. With the increased amount of data to be processed, the completion of task within given time in fog computing is a major challenge. So, scheduling of tasks and resources is a very important issue. A lot of research has been undertaken in recent years. In this survey, authors have reviewed various task scheduling algorithms suggested by researchers to meet the user requirements. The focus of this article is on diverse scheduling techniques being deployed in fog computing. An effort has been made to classify existing approaches, research issues and determine significant issues existing in this field. There are four major scheduling categories that are used by researchers namely, static, dynamic, heuristic, and hybrid. As per this study, 17% researchers deployed static, 23% dynamic, 47% heuristic, and 13% hybrid approaches, respectively. Analysis shows that for QoS (Quality of Service) parameters 19% researchers are focused on response time, 18% on cost and energy consumption, and 16% on makespan. For QoS parameter other factors have much smaller contribution in comparison to the above factors. In the tools used, researchers observed that 40% of the researches have used iFogSim, according to literature. Besides, this research article also discusses open issues and future work in the field of fog computing. This article also underlines various open issues and future directions in the field of fog computing.

Real-time X-ray phase-contrast imaging using SPINNet—a speckle-based phase-contrast imaging neural network

X-ray phase-contrast imaging has become indispensable for visualizing samples with low absorption contrast. In this regard, speckle-based techniques have shown significant advantages in spatial resolution, phase sensitivity, and implementation flexibility compared with traditional methods. However, the computational cost associated with data inversion has hindered their wider adoption. By exploiting the power of deep learning, we developed a speckle-based phase-contrast imaging neural network (SPINNet) that significantly improves the imaging quality and boosts the phase retrieval speed by at least 2 orders of magnitude compared to existing methods. To achieve this performance, we combined SPINNet with a coded-mask-based technique, an enhanced version of the speckle-based method. Using this scheme, we demonstrate the simultaneous reconstruction of absorption and phase images on the order of 100 ms, where a traditional correlation-based analysis would take several minutes even with a cluster. In addition to significant improvement in speed, our experimental results show that the imaging and phase retrieval quality of SPINNet outperform existing single-shot speckle-based methods. Furthermore, we successfully demonstrate SPINNet application in x-ray optics metrology and 3D x-ray phase-contrast tomography. Our result shows that SPINNet could enable many applications requiring high-resolution and fast data acquisition and processing, such as in situ and in operando 2D and 3D phase-contrast imaging and real-time at-wavelength metrology and wavefront sensing.

Fast CBCT Reconstruction using Convolutional Neural Networks for Arbitrary Robotic C-arm Orbits.

Cone-beam CT (CBCT) with non-circular acquisition orbits has the potential to improve image quality, increase the field-of view, and facilitate minimal interference within an interventional imaging setting. Because time is of the essence in interventional imaging scenarios, rapid reconstruction methods are advantageous. Model-Based Iterative Reconstruction (MBIR) techniques implicitly handle arbitrary geometries; however, the computational burden for these approaches is particularly high. The aim of this work is to extend a previously proposed framework for fast reconstruction of non-circular CBCT trajectories. The pipeline combines a deconvolution operation on the backprojected measurements using an approximate, shift-invariant system response prior to processing with a Convolutional Neural Network (CNN). We trained and evaluated the CNN for this approach using 1800 randomized arbitrary orbits. Noisy projection data were formed from 1000 procedurally generated tetrahedral phantoms as well as anthropomorphic data in the form of 800 CT and CBCT images from the Lung Image Database Consortium Image Collection (LIDC). Using this proposed reconstruction pipeline, computation time was reduced by 90% as compared to MBIR with only minor differences in performance. Quantitative comparisons of nRMSE, FSIM and SSIM are reported. Performance was consistent for projection data simulated with acquisition orbits the network has not previously been trained on. These results suggest the potential for fast processing of arbitrary CBCT trajectory data with reconstruction times that are clinically relevant and applicable - facilitating the application of non-circular orbits in CT image-guided interventions and intraoperative imaging.

High-speed wavelength-swept femtosecond source from 1055 to 1300 nm using a GHz femtosecond fiber laser.

In this Letter, we demonstrate a high-speed broadband wavelength-swept femtosecond source (WFS) that leverages the soliton self-frequency shift (SSFS) and intensity-wavelength encoding technologies. The optical wavelength of the high-speed WFS can be continuously swept from 1055 nm to nearly 1300 nm at a sweeping rate of 100 kHz. This WFS is especially seeded by a femtosecond mode-locked all-fiber laser at 1055 nm that has a fundamental repetition rate of ∼1.0 GHz, a maximum output power of 7 W, and a compressed pulse width of 220 fs. It is anticipated that this high-speed broadband WFS can be a promising source for applications that require fast wavelength scanning and high-speed data processing.

A joint computational and resource allocation model for fast parallel data processing in fog computing

A joint computational and resource allocation model for fast parallel data processing in fog computing

GPU accelerated Cartesian GRAPPA reconstruction using CUDA

Background and ObjectiveGRAPPA (Generalized Auto-calibrating Partially Parallel Acquisition) is an advanced parallel MRI reconstruction method (pMRI) that enables under-sampled data acquisition with multiple receiver coils to reduce the MRI scan time and reconstructs artifact free image from the acquired under-sampled data. However, the reduction in MRI scan time comes at the expense of long reconstruction time. It is because the GRAPPA reconstruction time shows exponential growth with increasing number of receiver coils. Consequently, the conventional CPU platforms may not adhere to the requirements of fast data processing for MR image reconstruction. MethodsGraphics Processing Units (GPUs) have recently emerged as a viable commodity hardware to reduce the reconstruction time of pMRI methods. This paper presents a novel GPU based implementation of GRAPPA using custom built CUDA kernels, to meet the rising demands of fast MRI processing. The proposed framework exploits intrinsic parallelism in the calibration and synthesis phases of GRAPPA reconstruction process, aiming to achieve high speed MR image reconstruction for various GRAPPA configuration settings using different number of receiver coils, auto-calibration signals (ACS), sizes of GRAPPA kernel and acceleration factors. In-vivo experiments (using 8, 12 and 30 receiver coils) are performed to compare the performance of the proposed GPU accelerated GRAPPA with the CPU based GRAPPA extensions and GPU counterpart. ResultsThe results indicate that the proposed method achieves up to ≈47.8× , ≈17× and ≈3.8× speed up gains over multicore CPU (single thread), multicore CPU (8 thread) and Gadgetron (GPU based GRAPPA) respectively, without compromising the reconstruction accuracy. ConclusionsThe proposed method reduces the GRAPPA reconstruction time by employing the calibration phase (GRAPPA weights estimation) and synthesis phase (interpolation) on GPU. Our study shows that the proposed GPU based parallel framework for GRAPPA reconstruction provides a solution for high-speed image reconstruction while maintaining the quality of the reconstructed images.