Huge Volume Of Data Research Articles

Detection of Novel Classes in Dynamic Data Streams

Every second, a huge volume of multi-dimensional data is generated in fields such as Social Networking, Industrial Internet of Things, Stock market and E-commerce applications. Knowledge and pattern extraction are a challenging task in the evolving nature of data stream. Major issues are (i) ‘concept drift’ occurs as a result of pattern changes in the data distribution and (ii) ‘concept evolution’ occurs when a new class evolves in the data stream. These issues degrade the performance of learning models. In this paper, we focus on detection of concept evolution and enhance the performance of classifiers. For this, we propose a new model to identify novel classes, namely, Detection of Novel Classes (DNC). The proposed method adopts long short term memory to continuously observe the streaming data in order to detect emerging classes. The continuous monitoring allows the model to distinguish between existing classes and the novel classes which save time and memory. Also, the proposed method is demonstrated for identifying more than one novel class. The experiments are performed over seven different datasets. The results confirm the efficiency is increased ranging from 6% to 34% by the proposed method in identifying new concepts in the evolving data stream than the existing methods available in the literature.

Multimodal In-Sensor Computing System Using Integrated Silicon Photonic Convolutional Processor.

Photonic integrated circuits offer miniaturized solutions for multimodal spectroscopic sensory systems by leveraging the simultaneous interaction of light with temperature, chemicals, and biomolecules, among others. The multimodal spectroscopic sensory data is complex and has huge data volume with high redundancy, thus requiring high communication bandwidth associated with high communication power consumption to transfer the sensory data. To circumvent this high communication cost, the photonic sensor and processor are brought into intimacy and propose a photonic multimodal in-sensor computing system using an integrated silicon photonic convolutional processor. A microring resonator crossbar array is used as the photonic processor to implement convolutional operation with 5-bit accuracy, validated through image edge detection tasks. Further integrating the processor with a photonic spectroscopic sensor, the in situ processing of multimodal spectroscopic sensory data is demonstrated, achieving the classification of protein species of different types and concentrations at various temperatures. A classification accuracy of 97.58% across 45 different classes is achieved. The multimodal in-sensor computing system demonstrates the feasibility of integrating photonic processors and photonic sensors to enhance the data processing capability of photonic devices at the edge.

Cyber-attack Detection and Mitigation Process under Big Data Consideration: Improved Recursive Feature Elimination-based Feature Selection

Due to the rapid growth of network technology, huge volume and distinct data sent via networks is expanding constantly. The situation shows how complex and dense cyber attacks and hazards are developing. Due to the rapid advancement in network density, cyber security specialists find it difficult to monitor all network activity. Due to frequent and sophisticated cyber attacks, it is becoming more challenging to detect and identify abnormalities in network events. The use of deep learning provides a variety of tools and strategies for automated cyber-attack detection as well as quick attack-type prediction as well as evaluation. This work introduces a novel cyber-attack detection and mitigation process under the following phases including preprocessing, feature extraction via the Map Reduce framework that handles the big data, feature selection, attack detection and mitigation. The Improved Normalisation process is achieved on the preprocessing phase. The work is examined from a big data perspective; hence Map Reduce framework is utilised for this. As a result, the framework will manage the feature extraction process, where features including statistical features, raw features, improved correlation-based features, and info gain-based features will be extracted. Following feature extraction, the Improved Recursive Feature Elimination procedure is processed that selects the relevant features. The hybrid detection model, which combines Recurrent Neural Networks (RNN) Deep and Belief Networks (DBN) is used to detect the attacks. Once an attack has been detected, the attacker must be mitigated. To accomplish this, an improved BAIT-based mitigation procedure is used. The two datasets used in this work are, namely, Intrusion Detection Systems (IDS) 2018 Intrusion CSVs (CSE-CIC-IDS2018) and UNSW_NB15. Finally, the suggested model and the alternative methods are contrasted using a variety of measures such as accuracy, sensitivity, specificity, precision, FDR, FNR and FPR.

An in-depth analysis of data reduction methods for sustainable deep learning.

In recent years, deep learning has gained popularity for its ability to solve complex classification tasks. It provides increasingly better results thanks to the development of more accurate models, the availability of huge volumes of data and the improved computational capabilities of modern computers. However, these improvements in performance also bring efficiency problems, related to the storage of datasets and models, and to the waste of energy and time involved in both the training and inference processes. In this context, data reduction can help reduce energy consumption when training a deep learning model. In this paper, we present up to eight different methods to reduce the size of a tabular training dataset, and we develop a Python package to apply them. We also introduce a representativeness metric based on topology to measure the similarity between the reduced datasets and the full training dataset. Additionally, we develop a methodology to apply these data reduction methods to image datasets for object detection tasks. Finally, we experimentally compare how these data reduction methods affect the representativeness of the reduced dataset, the energy consumption and the predictive performance of the model.

MATHEMATICAL MODELS IN MEDICINE: MODERN APPROACHES TO DIAGNOSTIC PROCESS AUTOMATION

Technology is becoming vital in improving diagnostic accuracy in modern medical field. The diagnostic process, like a production chain, consists of databases (resourc-es), the diagnostic activities (production), and the ultimate diagnosis (outcome). However, the intricacy of medical diagnostics has evolved due to the swift proliferation of medical knowledge and immense amount of patient data. Physicians encounter the complexity of handling huge volumes of clinical data,. To address these issues, the work examines the use of automated diagnostic systems supported by sophisticated mathematical models. The aim of the project is to investigate novel mathematical methods for disease detection and predic-tion, with a specific emphasis on fuzzy set theory. Particularly when dealing with imprecise or confusing initial data, such as medical histories and laboratory results, these techniques are indispensable. The authors investigate the function of expert systems and machine learning in the diagnosis of diseases such as gallstone formation and the detection of the diseases like sclerosis using medical imaging algorithms. These systems are specifically developed to as-sist medical professionals in making sound judgments, thereby presenting a very promising prospect for medical diagnostics through the integration of knowledge from several academic fields. The use of such technology will enhance clinical procedures, optimizing both the pre-cision and effectiveness of diagnoses while increasing healthcare accessibility .

An ML-Based Solution in the Transformation towards a Sustainable Smart City

The rapid development of modern information technology (IT), power supply, communication and traffic information systems and so on is resulting in progress in the area of distributed and energy-efficient (if possible, powered by renewable energy sources) smart grid components securely connected to entire smart city management systems. This enables a wide range of applications such as distributed energy management, system health forecasting and cybersecurity based on huge volumes of data that automate and improve the performance of the smart grid, but also require analysis, inference and prediction using artificial intelligence. Data management strategies, but also the sharing of data by consumers, institutions, organisations and industries, can be supported by edge clouds, thus protecting privacy and improving performance. This article presents and develops the authors’ own concept in this area, which is planned for research in the coming years. The paper aims to develop and initially test a conceptual framework that takes into account the aspects discussed above, emphasising the practical aspects and use cases of the Social Internet of Things (SIoT) and artificial intelligence (AI) in the everyday lives of smart sustainable city (SSC) residents. We present an approach consisting of seven algorithms for the integration of large data sets for machine learning processing to be applied in optimisation in the context of smart cities.

USE OF IIoT TECHNOLOGIES IN ENGINE MONITORING AND CONTROL SYSTEMS

Determining the operating parameters of engine systems is very important to ensure its long-term reliability. Modern power plants use a large number of modern electronic components (sensors, controllers, etc.). All systems generate a huge volume of data that must be tracked, processed and analyzed, which in turn creates calls to gateways and servers. However, the current state of development of microcontrollers and their power allow the use of fog computing, which can lead to more efficient use of both the power plant and data transmission systems. Also, the use of modern microcontrollers and fog computing allows modernizing old power plants with minimal effort and creating new information nodes for monitoring the state of nodes and engine systems and easily combining them with systems already existing on the engine. The selection of a microcontroller and justification of its type for the modernization of the fuel system of a diesel engine are considered. Rapid processes occur in the high-pressure fuel system of diesel engines, which can be determined by changes in fuel pressure. For detailed measurement of processes in the fuel system, measurements should be made with a frequency of at least 24 kHz, i.e.: if the crankshaft rotation frequency is 4000 min-1, then 24,000 measurements should be made every second (when registering data after 1 degree of crankshaft rotation). Another indicator that imposes its requirements is the bit rate of the analog-to-digital converter (ADC), which depends on the range of recorded values. For example, an ADC with a resolution of 8 bits is capable of outputting 256 discrete values (0…255). That is, the higher the bit rate, the more sensitive the converter is to signal changes, and 8 bits may not be enough. Thus, even a small number of sensors can generate a huge amount of data. Transmitting large amounts of data is irrational and can lead to data loss and "clogging" of transmission channels, which challenges the computational performance of the microprocessor.

Stock Price Prediction Using Artificial Intelligence and Neural Networks

Predicting the stock prices is a very complex task, and to predict an almost accurate stock price,we need a robust and accurate algorithm which can analyze and compute the longer-term share prices.Several researcher’s equally in the world and different industries have been very interested in the stockmarket. Stock processes are correlated within the nature of the market and that is why it is difficult topredict the share price. This project aims at processing and analyzing huge volumes of data (live data)and running comprehensive algorithms on the dataset. The purpose of the paper is to understand theshortcomings of the current prediction algorithms and to provide a method using neural networks andartificial intelligence through which we can predict the shared values with accuracy.By using the proposed method, anyone can monitor the preferred stock in real-time and can invest in thestock to make the most money by buying a large number of shares at the cheapest price and sellingthem at the highest price..

In‐Sensor Touch Analysis for Intent Recognition

AbstractTactile intent recognition systems, which are highly desired to satisfy human's needs and humanized services, shall be accurately understanding and identifying human's intent. They generally utilize time‐driven sensor arrays to achieve high spatiotemporal resolution, however, which encounter inevitable challenges of low scalability, huge data volumes, and complex processing. Here, an event‐driven intent recognition touch sensor (IR touch sensor) with in‐sensor computing capability is presented. The merit of event‐driven and in‐sensor computing enables the IR touch sensor to achieve ultrahigh resolution and obtain complete intent information with intrinsic concise data. It achieves critical signal extraction of action trajectories with a rapid response time of 0.4 ms and excellent durability of >10 000 cycles, bringing an important breakthrough of tactile intent recognition. Versatile applications prove the integrated functions of the IR touch sensor for great interactive potential in all‐weather environments regardless of shading, dynamics, darkness, and noise. Unconscious and even hidden action features can be perfectly extracted with the ultrahigh recognition accuracy of 98.4% for intent recognition. The further auxiliary diagnostic test demonstrates the practicability of the IR touch sensor in telemedicine palpation and therapy. This groundbreaking integration of sensing, data reduction, and ultrahigh‐accuracy recognition will propel the leapfrog development for conscious machine intelligence.

Burst-classifier: Automated classification of solar radio burst type II, III and IV for CALLISTO spectra using physical properties during maximum of solar cycle 24

Continuous observation of solar radio bursts (SRBs) throughout the year using the CALLISTO spectrometer generates a huge volume of spectral data. This study introduces a burst-classifier algorithm, which is an automated algorithm, to classify the SRB spectrum into three solar radio bursts, namely Type II (SRBT II), Type III (SRBT III) and Type IV (SRBT IV). The proposed algorithm was designed using four characteristic parameters derived from a collection of training dataset files. The characteristic parameters were derived from the intensity bursts observed on frequency channels and timesteps of the spectrum. This dataset consisted of 50 spectra of SRBT II and SRBT III, along with 40 spectra for SRBT IV, collected during the solar maximum of 2014 (Solar Cycle 24). After observations and analysis of the training dataset, each burst type was set up with a threshold. A training dataset of 80 data spectra from 2013 to 2016 was used to test the algorithm. Accuracy of the proposed algorithm was calculated using the percentage of true positives (TP) and false positives (FP). Findings demonstrate an accuracy of ∼74 % with 57 out of 80 spectra classified as TP and 23 spectra as FP.

An efficient data compression and storage technique with key management authentication in cloud space

Cloud computing is one of the promising technologies that offers cost-effective choices for processing and storing the huge volumes of data. In today’s world, data is the most important asset that one can have but it needs to be handled and protected properly. Portability of data can be increased by reducing the size of the data to be stored because of the limited storage space. As a result, data compression has arisen significantly. Data compression is a useful technique for reducing data size and increasing the effectiveness of data transit and storage. Data compression reduces the size of a data file while using lossy or lossless compression. One of the newest techniques for data compression is data duplication, which can reduce the amount of data saved while removing unnecessary data and maintaining an exact copy of the data. This analysis presents an Efficient data compression and storage technique with key management authentication in cloud space. This approach uses Regressive probabilistic key encryption (RPKE) to encrypt the cloud data and Lempel-Ziv-77-Huffman coding (LZ77-HM) is used to compress the huge amounts of cloud data. The Performance of presented approach is evaluated in terms of compression ratio and compression rate.

Functionalized Morpholine-thiazole Scaffold: Synthetic Strategies and Applications.

The thiazole derivatives as important members of heterocyclic compounds have attracted much synthetic interest due to their different biological properties. In recent years, studies on the synthesis of morpholine compounds have increased because of the properties of this core. In particular, the hybrid structures in which the thiazole ring is linked to morpholine nuclei in one molecular frame have gained popularity. The presented review is an attempt to summarize a huge volume of data on morpholinothiazoles being a widely studied class of these molecules used in modern organic and medicinal chemistry. The manuscript covers the approaches to the synthesis of the morpholinothiazoles derivatives. The synthetic strategies of the target compounds depend on one-pot or multistage reactions or the transformation of other related heterocycles. Additionally, we covered the biological activities and other applications of certain morpholinothiazoles. The information on these compounds made special consideration of medicinal chemists to yield a combinatorial library and carry out thorough efforts in the search of morpholinothiazoles.

Physics-guided deep learning-based inversion for airborne electromagnetic data

SUMMARY The Earth's subsurface structure provides critical insights into sustainable resource management and geologic evolution. The airborne electromagnetic (AEM) method is an efficient data acquisition technique and can be used to image the underground resistivity structure with high spatial resolution. However, inversion of the increasingly huge volume of AEM data poses a heavy computational burden. In this study, we develop a hybrid deep learning-based approach by using the physics-guided neural network (PGNN) which incorporates the governing physical laws into the loss function to solve the AEM inverse problem. The PGNN integrates the strength of data-driven method for representation learning with electromagnetic laws and allows for the underlying physical constraints to be strictly satisfied. We validate the effectiveness of our approach using both synthetic and field datasets. Compared with the classic Gauss–Newton method, our PGNN inversion system shows strong robustness against multiple noise sources and reduces the risk of being trapped in local extrema. Moreover, the PGNN-inverted results are physically more consistent with the AEM observations compared to the purely data-driven approach. Application to the field AEM data from Northern Australia demonstrates that the PGNN-based inversion framework effectively estimates the subsurface electrical properties with considerable lateral continuity and significantly higher efficiency, completing the inversion of more than 2734000 AEM soundings taking only minutes on a common PC. Our proposed PGNN-based method shows great promise for large-scale underground resistivity imaging, and the well-identified subsurface resistivity structure can effectively improve our understanding of resource distributions and geological hazards.

Utilising InSAR for monitoring infrastructure at national scales

Interferometric SAR (InSAR) is now a widely used tool for the monitoring of infrastructure, in particular mines, dams, and oil and gas facilities. The technology is also widely used for the monitoring of construction activities such as tunnelling. With the growth of globally available and continuously updated SAR datasets, the monitoring of entire regional and national infrastructure networks is now feasible. Here we show how continuously updated, network-wide, InSAR datasets can be used to proactively manage large infrastructure networks such as roads, railways, and water pipelines. In the UK continuously updated InSAR data (derived from Sentinel-1 SAR data) is being effectively used across the entire railway network to proactively monitor potential embankment failures, large-scale landslides, and potential hazards outside of the boundary fence. A challenge of using InSAR data in this way is the huge volume of data that is available, as such custom risk metrics and machine learning algorithms are applied to highlight some of the areas at heightened risk. InSAR is also being applied to tackle the issue of water pipe leakage caused by natural ground movements, such as soil shrink-swell. In this case, regularly updated InSAR data has been used in the UK and Australia to detect areas at elevated risk of pipeline failures based on ground movement patterns, and by combining historical movement data and known failure events it is possible to refine such models. With expanding global SAR constellations proactive, nationwide, monitoring of critical assets is now possible, with the challenge being how we derive insight from these very large datasets.

Improved Video-Based Point Cloud Compression via Segmentation.

A point cloud is a representation of objects or scenes utilising unordered points comprising 3D positions and attributes. The ability of point clouds to mimic natural forms has gained significant attention from diverse applied fields, such as virtual reality and augmented reality. However, the point cloud, especially those representing dynamic scenes or objects in motion, must be compressed efficiently due to its huge data volume. The latest video-based point cloud compression (V-PCC) standard for dynamic point clouds divides the 3D point cloud into many patches using computationally expensive normal estimation, segmentation, and refinement. The patches are projected onto a 2D plane to apply existing video coding techniques. This process often results in losing proximity information and some original points. This loss induces artefacts that adversely affect user perception. The proposed method segments dynamic point clouds based on shape similarity and occlusion before patch generation. This segmentation strategy helps maintain the points' proximity and retain more original points by exploiting the density and occlusion of the points. The experimental results establish that the proposed method significantly outperforms the V-PCC standard and other relevant methods regarding rate-distortion performance and subjective quality testing for both geometric and texture data of several benchmark video sequences.

Solving Hadoop Drawbacks by Using Spark Framework

Abstract: In Today’s world, peta byte to exa byte data is producing every minute by social media, digital marketing, social sites. This is all generating a big data. Big data is a collection of huge and complex data sets and data volume that include the huge amount of data, data management services, social media analytics and real-time data. Big data analytics is the process of determining large volume of data. Because of generating large volume of data every second, shortage of the processing of that data techniques. Hadoop is the latest data processing framework which processes large volume of data through its HDFS and MapReduce.

Brain Tumor Classification Using DenseNet CNN Model

Abstract: In recent years, Due to its powerful soft tissue comparison and non-invasive nature, magnetic resonance imaging,or MRI, has attracted a lot of attention recently. MRI is a frequently utilized imaging modality for the purpose of locating brain cancers in infants. The MRI generates a huge volume of data. Tumor heterogeneity, isointense and hypointense characteristics limit manual segmentation within a reasonable time frame, which in turn limits the application of trustworthy quantitative metrics in clinical practice. Manual segmentation tasks in clinical practice take a lot of time, and the operator's experience has a big impact on how well they execute. Tumor segmentation methods must be accurate and automated, however due to the extreme spatial and structural variety of brain tumors, this is a challenging task. We have suggested completely automatic brain categorization in our project. Using the DenseNet model, we have presented a fully automatic way to classify brain cancers. Tests conducted on the datasets demonstrate the effectiveness of our suggested approach for identifying baby brain cancers from MRI (MRI).

Enhanced Network Intrusion Detection System for Internet of Things Security Using Multimodal Big Data Representation with Transfer Learning and Game Theory.

Internet of Things (IoT) applications and resources are highly vulnerable to flood attacks, including Distributed Denial of Service (DDoS) attacks. These attacks overwhelm the targeted device with numerous network packets, making its resources inaccessible to authorized users. Such attacks may comprise attack references, attack types, sub-categories, host information, malicious scripts, etc. These details assist security professionals in identifying weaknesses, tailoring defense measures, and responding rapidly to possible threats, thereby improving the overall security posture of IoT devices. Developing an intelligent Intrusion Detection System (IDS) is highly complex due to its numerous network features. This study presents an improved IDS for IoT security that employs multimodal big data representation and transfer learning. First, the Packet Capture (PCAP) files are crawled to retrieve the necessary attacks and bytes. Second, Spark-based big data optimization algorithms handle huge volumes of data. Second, a transfer learning approach such as word2vec retrieves semantically-based observed features. Third, an algorithm is developed to convert network bytes into images, and texture features are extracted by configuring an attention-based Residual Network (ResNet). Finally, the trained text and texture features are combined and used as multimodal features to classify various attacks. The proposed method is thoroughly evaluated on three widely used IoT-based datasets: CIC-IoT 2022, CIC-IoT 2023, and Edge-IIoT. The proposed method achieves excellent classification performance, with an accuracy of 98.2%. In addition, we present a game theory-based process to validate the proposed approach formally.

Artificial Intelligence Could be the Personalized Treatment Strategy for Cancer

Cancer is one of the world’s most serious medical challenges. Because of its great heterogeneity, persons with similar tumors could react in a different way to the same drugs or surgical methods, prompting the development of more accurate tumor treatment approaches as well as patient-specific tailored treatments. To establish targeted therapy choices for patients, it is critical to have a full acceptance of the changes that tumors endure, counting modifications in their genetic factors, proteins, and cancer cell behaviors. Tumor treatment requires precise targeting. Big data-driven artificial intelligence (AI) may reveal patterns, insights, and related information hidden inside huge volumes of data. To identify exact data from transcriptomics, radiomics, genomes, proteomics, digital pathological images, etc., subsets of AI’s machine learning capability may be explored. This may help clinicians get a better and more comprehensive knowledge of malignancies. In addition, to provides the optimal therapy for each patient and improves clinical outcomes.

Data hazards in synthetic biology.

Data science is playing an increasingly important role in the design and analysis of engineered biology. This has been fueled by the development of high-throughput methods like massively parallel reporter assays, data-rich microscopy techniques, computational protein structure prediction and design, and the development of whole-cell models able to generate huge volumes of data. Although the ability to apply data-centric analyses in these contexts is appealing and increasingly simple to do, it comes with potential risks. For example, how might biases in the underlying data affect the validity of a result and what might the environmental impact of large-scale data analyses be? Here, we present a community-developed framework for assessing data hazards to help address these concerns and demonstrate its application to two synthetic biology case studies. We show the diversity of considerations that arise in common types of bioengineering projects and provide some guidelines and mitigating steps. Understanding potential issues and dangers when working with data and proactively addressing them will be essential for ensuring the appropriate use of emerging data-intensive AI methods and help increase the trustworthiness of their applications in synthetic biology.