Massive Datasets Research Articles

Role of AI, Automation & Robotics in Pharmaceutical Industry

The pharmaceutical industry is undergoing a massive transformation driven by technological advancements and leapfrogging further due to the integration of Artificial intelligence (AI), automation, and robotics. These technologies are being deployed to cover various aspects, including drug discovery, manufacturing, supply chain, and patient care. AI's ability to process and analyze massive data sets allows researchers to identify new drug candidates faster or improve on current ones through various strategies. Automation transforms repetitive tasks and increases accuracy, but most importantly, it frees people up from work that they need not do and concentrates on the jobs that still require human involvement. Conversely, when integrated with AI, robots enhance the production process by enabling speedy, accurate, and scalable manufacturing. Robotics are now being used in pharmacies for medication dispensing and are very efficient. Furthermore, all these innovations driven by AI, Automation, and robotics also contribute to personalized medicine by extending tailored treatments based on individual patient data. This paper explores the role of AI, Automation, and Robotics in the Pharmaceutical industry and its benefits. The ongoing evolution of such technologies holds immense potential to address the growing needs and handle industry challenges such as increasing demand, regulatory compliance, and global health needs to ultimately lead the pharmaceutical industry towards a more efficient, adaptive, and patient-centered approach.

End to End Leaf Detection Using Convolutional Neural Network

Global food security is seriously threatened by crop diseases, yet in many places, it is still difficult to identify them quickly because of a lack of infrastructure. In tackling this problem, recent developments in leaf-based image categorization methods have produced encouraging outcomes. This study investigates how to differentiate between healthy and unhealthy leaves using curated datasets using the Random Forest algorithm. The development of the dataset, feature extraction, classifier training, and classification are the various stages of the implementation process. To efficiently distinguish between diseased and healthy samples, the Random Forest classifier is used to train the dataset, which includes pictures of both diseased and healthy leaves. A significant obstacle to agricultural production, plant diseases have an impact on crop yield and quality. Precision farming and sustainable agriculture depend on the timely and precise identification of leaf diseases. In this work, we present a complete system that automatically detects and classifies leaf diseases using Convolutional Neural Networks (CNN), starting with picture capture. The technology offers a quicker method of identifying leaf illnesses and does away with the requirement for manual feature engineering. Our tests, carried out on several public datasets, show the model's efficacy and achieve high accuracy for a range of illness categories. These results underline the practical uses that might improve crop management and early intervention techniques. Using machine learning on massive datasets that are accessible to the public, this approach offers a scalable solution for detecting plant diseases on a large scale.

Inferring redshift and galaxy properties via a multi-task neural net with probabilistic outputs. An application to simulated MOONS spectra

The era of large-scale astronomical surveys demands innovative approaches for rapid and accurate analysis of extensive spectral data, and a promising direction in which to address this challenge is offered by machine learning. Here, we introduce a new pipeline M-TOPnet (Multi-Task network Outputting Probabilities), which employs a convolutional neural network with residual learning to simultaneously derive redshift and other key physical properties of galaxies from their spectra. Our tool efficiently encodes spectral information into a latent space, employing distinct downstream branches for each physical quantity, thereby benefiting from multi-task learning. Notably, our method handles the redshift output as a probability distribution, allowing for a more refined and robust estimation of this critical parameter. We demonstrate preliminary results using simulated data from the MOONS instrument, which will soon be operating at the ESO/VLT. We highlight the effectiveness of our tool in accurately predicting the redshift, stellar mass, and star formation rate of galaxies at $z 1-3$, even for faint sources ($m_H 24$) for which traditional methods often struggle. Through analysis of the output probability distributions, we demonstrate that our pipeline enables robust quality screening of the results, achieving accuracy rates of up to 99<!PCT!> in redshift determination (defined as predictions within $| z| < 0.01$ relative to the true redshift) with $8 h$ exposure spectra, while automatically identifying potentially problematic cases. Our pipeline thus emerges as a powerful solution for the upcoming challenges in observational astronomy, combining precision, interpretability, and efficiency, all aspects that are crucial for analysing the massive datasets expected from next-generation instruments.

Generating implicit object fragment datasets for machine learning

One of the primary challenges inherent in utilizing deep learning models is the scarcity and accessibility hurdles associated with acquiring datasets of sufficient size to facilitate effective training of these networks. This is particularly significant in object detection, shape completion, and fracture assembly. Instead of scanning a large number of real-world fragments, it is possible to generate massive datasets with synthetic pieces. However, realistic fragmentation is computationally intensive in the preparation (e.g., pre-factured models) and generation. Otherwise, simpler algorithms such as Voronoi diagrams provide faster processing speeds at the expense of compromising realism. In this context, it is required to balance computational efficiency and realism. This paper introduces a GPU-based framework for the massive generation of voxelized fragments derived from high-resolution 3D models, specifically prepared for their utilization as training sets for machine learning models. This rapid pipeline enables controlling how many pieces are produced, their dispersion and the appearance of subtle effects such as erosion. We have tested our pipeline with an archaeological dataset, producing more than 1M fragmented pieces from 1,052 Iberian vessels (). Although this work primarily intends to provide pieces as implicit data represented by voxels, triangle meshes and point clouds can also be inferred from the initial implicit representation. To underscore the unparalleled benefits of CPU and GPU acceleration in generating vast datasets, we compared against a realistic fragment generator that highlights the potential of our approach, both in terms of applicability and processing time. We also demonstrate the synergies between our pipeline and realistic simulators, which frequently cannot select the number and size of resulting pieces. To this end, a deep learning model was trained over realistic fragments and our dataset, showcasing similar results.

Outcome dependent subsampling divide and conquer in generalized linear models for massive data

In order to break the constraints and barriers caused by limited computing power in processing massive datasets, we propose an outcome dependent subsampling divide and conquer strategy in this paper. The proposed strategy can process data on multiple blocks in parallel and concentrate the computing resources of each block on regions with the most information. We develop a distributed statistical inference method and propose a computation-efficient algorithm in the generalized linear models for massive data. The proposed method only need to preserve some summary statistics from each data block and then use them to directly construct the proposed estimator. The asymptotic properties of the proposed method are established. Simulation studies and real data analysis are conducted to illustrate the merits of the proposed method.

IoT-Generated Big Data and Security Issues

Nowadays BD (Big Data) and IoT (Internet of Things) are popular and mostly used terms. The Internet of Things is generating an enormous amount of data. No conventional tool that refers to a massive set of data, such as BD, is available. Speed, Variety and volume are important features of BD, and due to this big data. In this paper, we explore Big data in IoT-driven technologies and the issue of the seven V’s in Big Data. With the rapid growth of IoT, a number of devices are connected. This paper also discusses several cloud-based IoT platforms such as Amazon Web Services and Google Cloud Internet of Things. We studied the importance of Big Data imagining, gives insights on various visualization tools and methods. Lastly, this paper also deals with various significant challenges of Big Data in IoT, security issues and future research directions.

TimeGPT in load forecasting: A large time series model perspective

Machine learning models have made significant progress in load forecasting, but their forecast accuracy is limited in cases where historical load data is scarce. Inspired by the outstanding performance of large language models (LLMs) in computer vision and natural language processing, this paper aims to discuss the potential of large time series models in load forecasting with scarce historical data. Specifically, the large time series model is constructed as a time series generative pre-trained transformer (TimeGPT), which is trained on massive and diverse time series datasets consisting of 100 billion data points (e.g., finance, transportation, banking, web traffic, weather, energy, healthcare, etc.). Then, the scarce historical load data is used to fine-tune the TimeGPT, which helps it to adapt to the data distribution and characteristics associated with load forecasting. Simulation results show that TimeGPT outperforms the popular benchmarks for load forecasting on several real datasets with scarce training samples, particularly for short look-ahead times. However, it cannot be guaranteed that TimeGPT is always superior to benchmarks for load forecasting with scarce data, since the performance of TimeGPT may be affected by the distribution differences between the load data and the training data. In practical applications, operators can divide the historical data into a training set and a validation set, and then use the validation set loss to decide whether TimeGPT is the best choice for a specific dataset.

AI-Driven Fraud Detection in Banking: Enhancing Transaction Security

An important step forward in risk management and fraud detection has been achieved with the integration of Artificial Intelligence (AI) in the banking sector. In this paper, we take a look at how AI has revolutionized various fields, shedding light on the benefits and drawbacks of this technology. The effects of AI on risk management are complex. More complex credit risk assessment models are made possible by algorithms that can see patterns in massive datasets that people might miss. When it comes to market and liquidity issues, real-time transaction monitoring is absolutely essential for quick risk mitigation. Automating compliance with regulatory norms is another critical function of AI, which helps to decrease human mistake and assures quick adaptability to changes in regulations. The automation of mundane processes and the reinforcement of cybersecurity measures further reduce operational risks. By examining client behaviour and transaction data, enhanced algorithms may adeptly spot anomalies that could indicate fraud. Artificial intelligence's capacity to foresee future events enables it to foil possible fraud attempts. The systems are designed to respond to changing fraudster strategies with its adaptive learning feature.

A Subsampling Strategy for AIC-based Model Averaging with Generalized Linear Models

Subsampling is an effective approach to address computational challenges associated with massive datasets. However, existing subsampling methods do not consider model uncertainty. In this article, we investigate the subsampling technique for the Akaike information criterion (AIC) and extend the subsampling method to the smoothed AIC model-averaging framework in the context of generalized linear models. By correcting the asymptotic bias of the maximized subsample objective function used to approximate the Kullback–Leibler divergence, we derive the form of the AIC based on the subsample. We then provide a subsampling strategy for the smoothed AIC model-averaging estimator and study the corresponding asymptotic properties of the loss and the resulting estimator. A practically implementable algorithm is developed, and its performance is evaluated through numerical experiments on both real and simulated datasets.

Closha 2.0: a bio-workflow design system for massive genome data analysis on high performance cluster infrastructure

BackgroundThe explosive growth of next-generation sequencing data has resulted in ultra-large-scale datasets and significant computational challenges. As the cost of next-generation sequencing (NGS) has decreased, the amount of genomic data has surged globally. However, the cost and complexity of the computational resources required continue to be substantial barriers to leveraging big data. A promising solution to these computational challenges is cloud computing, which provides researchers with the necessary CPUs, memory, storage, and software tools.ResultsHere, we present Closha 2.0, a cloud computing service that offers a user-friendly platform for analyzing massive genomic datasets. Closha 2.0 is designed to provide a cloud-based environment that enables all genomic researchers, including those with limited or no programming experience, to easily analyze their genomic data. The new 2.0 version of Closha has more user-friendly features than the previous 1.0 version. Firstly, the workbench features a script editor that supports Python, R, and shell script programming, enabling users to write scripts and integrate them into their pipelines. This functionality is particularly useful for downstream analysis. Second, Closha 2.0 runs on containers, which execute each tool in an independent environment. This provides a stable environment and prevents dependency issues and version conflicts among tools. Additionally, users can execute each step of a pipeline individually, allowing them to test applications at each stage and adjust parameters to achieve the desired results. We also updated a high-speed data transmission tool called GBox that facilitates the rapid transfer of large datasets.ConclusionsThe analysis pipelines on Closha 2.0 are reproducible, with all analysis parameters and inputs being permanently recorded. Closha 2.0 simplifies multi-step analysis with drag-and-drop functionality and provides a user-friendly interface for genomic scientists to obtain accurate results from NGS data. Closha 2.0 is freely available at https://www.kobic.re.kr/closha2.

Additive Hazards Regression Analysis of Massive Interval-Censored Data via Data Splitting

With the rapid development of data acquisition and storage space, massive datasets exhibited with large sample size emerge increasingly and make more advanced statistical tools urgently need. To accommodate such big volume in the analysis, a variety of methods have been proposed in the circumstances of complete or right censored survival data. However, existing development of big data methodology has not attended to interval-censored outcomes, which are ubiquitous in cross-sectional or periodical follow-up studies. In this work, we propose an easily implemented divide-and-combine approach for analyzing massive interval-censored survival data under the additive hazards model. We establish the asymptotic properties of the proposed estimator, including the consistency and asymptotic normality. In addition, the divide-and-combine estimator is shown to be asymptotically equivalent to the full-data-based estimator obtained from analyzing all data together. Simulation studies suggest that, relative to the full-data-based approach, the proposed divide-and-combine approach has desirable advantage in terms of computation time, making it more applicable to large-scale data analysis. An application to a set of interval-censored data also demonstrates the practical utility of the proposed method.

Machine Learning: A survey of requirements prioritization: A Review Study

In any software systems, the requirements prioritization considered as pivotal task. This paper aims to explain and discuss the works done on requirements prioritization based Machine learning along with the dependency-aware requirements prioritization. Machine learning has become of attention to scientists, researchers, and users because of the existence of vast data and deep learning algorithms that can analyze massive sets of data. The basic algorithms are used for dependency learning calculation, resolve the stakeholder’s conflicts, classify requirements, and scalability improvement. This paper will present a brief background and comprehensive presentation of a number of machine learning techniques for requirements prioritization and those concerning the requirement dependency in its simple, complex, and Hybrid form. There are a number of papers, articles, and research papers that deal with requirements prioritization, few of them handling the dependency. This paper will present a brief background for several requirements prioritization based on machine learning. then make a comparison and discussion of a number of selected techniques in terms of algorithm type, issues addressed, and evaluated data level. method that handling the dependency regarding of strengths and weakness.

Automated Water Quality Assessment Using Big Data Analytics

Water is one of the world's most precious resources, essential to life. Industrial waste, agricultural runoff, and urban discharge degrade water, rendering it unfit for consumption. Water quality monitoring and evaluation are more important than ever. Big Data analytics is used to examine water quality utilizing enormous datasets of pH, hardness, solids concentration, chloramine, sulfate, conductivity, organic carbon, trihalomethanes, and turbidity. This work classifies water potability, which is vital for human consumption, using strong machine learning on massive datasets. Classifiers were Random Forest, Gradient Boosting, and Support Vector Machine on 3,276 water bodies. The Random Forest classifier obtained the highest accuracy at 66.77% after significant data preparation and training, followed by Gradient Boosting at 66.01% and SVM at 62.80%. This shows that Big Data analytics and machine learning algorithms can interpret complex water quality data for public health and natural resource management. The Random Forest classifier and SVM in this study accurately calculate water potability. Prediction algorithms consider water cleanliness data and may aid public safety and water resource monitoring.

A unified LOD model for river network and DEM based on an improved 3D_DP algorithm

The level of detail (LOD) modelling of vector and terrain data is individual, resulting in geometric and topological inconsistencies in simplification processes. The three dimension Douglas Peucker (3D_DP)algorithm can realize gradual discrete point selection through threshold, which is mainly used in DEM synthesis, and its simplified process is very suitable for the dynamic establishment of massive data sets. A new LOD modeling method based on 3D_DP algorithm is proposed to simplify the consistency of river network vector elements and DEM in this paper. The specific steps are as follows: Firstly, the “Bending Adjustment Index (BAI)” is introduced to improve the 3D_DP algorithm, called the improved 3D_DP algorithm; Secondly, the DEM data is extracted into a 3D discrete point dataset, and the river line vector data is also converted into a discrete point dataset, assigned with elevation attributes, and merges with the DEM’s 3D discrete points. The merged point datasets are equenced based on the importance of each point, which are computed by the improved 3D_DP algorithm. The order of deleting points is determined by the sequence and the corresponding model spatial errors are calculated with the deletion of points. Then, the DEM’s 3D discrete points are constructed a Delaunay network with the river line as a mandatory constraint condition. The required triangulation is called in real time with the change of sight distance depending on the simple correspondence between screen projection error and model space error, and the unified LOD model for river line vector and DEM is established. The results show that the river’s overall shape and the terrain’s main features can be reserved under the same simplified factor based on the improved 3D_DP algorithm. The unified LOD model for the river network and DEM is feasible under the importance sequence of merged point datasets by the improved 3D_DP algorithm. Under the proper operation of data blocking, the rendering frame rate can meet practical application requirements.

Distributed Least Product Relative Error estimation for semi-parametric multiplicative regression with massive data

Distributed systems have been widely used for massive data analysis, but few studies focus on multiplicative regression models. We consider a communication-efficient surrogate likelihood method using the Least Product Relative Error criterion for semi-parametric multiplicative models on massive datasets. The non-parametric component is efficiently handled via B-spline approximation. We derive the asymptotic properties for both parametric and non-parametric components, while the SCAD and adaptive Lasso penalty functions are developed and their oracle properties for variable selection are validated. Simulation studies and an application to an energy prediction dataset are used to demonstrate the effectiveness and practical utility of the proposed method.

Advancing Fraud Detection in Banking: Integration of Data Pipelines, Machine Learning, and Cloud Computing

This article examines the transformation of fraud detection in the banking sector through the integration of advanced data pipelines, machine learning (ML), artificial intelligence (AI), and cloud computing technologies. We analyze how modern data pipelines enable real-time processing of vast transactional datasets, significantly improving the timeliness and accuracy of fraud detection. The article explores the application of ML models in identifying suspicious patterns and the role of AI-driven systems in continuously adapting to evolving fraud schemes while reducing false positives. We evaluate the impact of cloud platforms such as AWS, Azure, and Google Cloud in providing scalable, cost-efficient infrastructures for processing massive datasets and supporting seamless integration with ML models. The article presents case studies of successful implementations by major banks, demonstrating substantial reductions in processing times and improved detection efficiency. Additionally, we address key challenges including data quality maintenance, model interpretability, and false positive mitigation. The article concludes by discussing future innovations such as Federated Learning and Explainable AI (XAI), which promise to enhance cross-institutional collaboration and decision-making transparency in fraud detection. This comprehensive analysis provides valuable insights for financial institutions seeking to enhance their fraud detection capabilities in an increasingly complex digital landscape.

Parallelization of the Bison Algorithm Applied to Data Classification

In data science and machine learning, efficient and scalable algorithms are paramount for handling large datasets and complex tasks. Classification algorithms, in particular, play a crucial role in a wide range of applications, from image recognition and natural language processing to fraud detection and medical diagnosis. Traditional classification methods, while effective, often struggle with scalability and efficiency when applied to massive datasets. This challenge has driven the development of innovative approaches that leverage modern computational frameworks and parallel processing capabilities. This paper presents the Bison Algorithm, applied to classification problems. The algorithm, inspired by the social behavior of bison, aims to enhance the accuracy of classification tasks. The Bison Algorithm is implemented using PySpark, leveraging the distributed computing power to handle large datasets efficiently. This study compares the performance of the Bison Algorithm on several dataset sizes using speedup and scaleup as the performance measure.

Sine Cosine Algorithm Based on Optimal Convolutional Autoencoder for Intrusion Detection and Classification Models

Network security comprises a multifaceted method that aims to protect computer networks from malicious activities, unauthorized access, and data breaches. The security mechanism is Intrusion Detection which is an important constituent that is employed to monitor and analyse the network traffic for recognizing and responding to intrusive or suspicious behavior. Innovative methods such as deep learning (DL) are employed to enhance the effectiveness of Intrusion Detection Systems (IDSs). DL is extremely implemented for IDS owing to its proficiency for automatically learning and extracting complex patterns and features from massive and multifaceted network datasets. Neural network (NN) models, permit the system to distinguish between anomalous patterns and normal network behaviors, increasing the accuracy of intrusion detection. The flexibility of DL methods to emerging cyberattacks with their adeptness to handle large - scale and various data, positions them as a strong and efficient tool for proactive and intelligent intrusion detection in existing cybersecurity settings. This article presents a Sine Cosine Algorithm with Optimal Convolutional Autoencoder for Intrusion Detection and Classification (SCAOCAE - IDC) method. The developed SCAOCAE - IDC system presents a wide - ranging strategy to improve the precision and effectiveness of IDSs. The method combines diverse advanced mechanisms like Min - Max scalar normalization for data preprocessing, Sine Cosine Algorithm (SCA) for feature selection (FS), Convolutional Autoencoder (CAE) for better feature extraction and classification, and Heap - Based Optimization (HBO) for hyperparameter tuning. The Min - Max scalar makes sure of robust data normalization, SCA increasingly chooses main features, CAE capably captures complex patterns in the data, and HBO fine - tunes hyperparameters for improved system performance. By employing the synergistic combination of such modules, the presented SCAOCAE - IDC algorithm indicates considerable outcomes for increasing the reliability and accuracy of IDSs and classification systems.

Robust malicious software detection and classification using global whale optimization algorithm with deep learning approach

Software malware detection and classification leverage sophisticated procedures and methods from the cybersecurity domain for identifying and categorizing malicious software, generally called malware. This procedure analyses code behaviour, file structures, and other features to distinguish between benign and malicious programs. Machine learning (ML) and artificial intelligence (AI) are vital in this domain, allowing the progress of dynamic and adaptive systems that identify novel and developing malware attacks. By training on massive datasets of benign and malicious instances, these systems learn patterns and signatures indicative of malware. This lets them correctly categorize and respond to potential attacks in real-time. This study presents a Global Whale Optimization Algorithm with Neutrosophic Logic for Software Malware Detection and Classification (GWOANL-SMDC) technique. The GWOANL-SMDC technique secures the software via the Android malware recognition process. Primarily, the GWOANL-SMDC technique employs the Neutrosophic Cognitive Maps (NCM) model for the feature selection process. The GWOANL-SMDC technique uses a convolutional long short-term memory (ConvLSTM) model for software malware detection. At last, the GWOA-based parameter tuning is performed to improve the performance of the ConvLSTM model. The simulation values of the GWOANL-SMDC technique are examined on the malware dataset. The obtained results ensured that the GWOANL-SMDC technique improved capability in detecting software malware.

Pitfalls in Machine Learning for Computer Security

With the growing processing power of computing systems and the increasing availability of massive datasets, machine learning algorithms have led to major breakthroughs in many different areas. This development has influenced computer security, spawning a series of work on learning-based security systems, such as for malware detection, vulnerability discovery, and binary code analysis. Despite great potential, machine learning in security is prone to subtle pitfalls that undermine its performance and render learning-based systems potentially unsuitable for security tasks and practical deployment. In this paper, we look at this problem with critical eyes. First, we identify common pitfalls in the design, implementation, and evaluation of learning-based security systems. We conduct a study of 30 papers from top-tier security conferences within the past 10 years, confirming that these pitfalls are widespread in the current security literature. In an empirical analysis, we further demonstrate how individual pitfalls can lead to unrealistic performance and interpretations, obstructing the understanding of the security problem at hand. As a remedy, we propose actionable recommendations to support researchers in avoiding or mitigating the pitfalls where possible. Furthermore, we identify open problems when applying machine learning in security and provide directions for further research.