Huge Datasets Research Articles

Assessment of Zero-Day Vulnerability using Machine Learning Approach

Organisations and people are seriously threatened by zero-day vulnerabilities because they may be utilised by attackers to infiltrate systems and steal private data. Currently, Machine Learning (ML) techniques are crucial for finding zero-day vulnerabilities since they can analyse huge datasets and find patterns that can point to a vulnerability. This research’s goal is to provide a reliable technique for detecting intruders and zero-day vulnerabilities in software systems. The suggested method employs a Deep Learning (DL) model and an auto-encoder model to find unusual data patterns. Additionally, a model for outlier detection that contrasts the autoencoder model with the single class-based Support Vector Machine (SVM) technique will be developed. The dataset of known vulnerabilities and intrusion attempts will be used to train and assess the models.

Large-Scale Data Processing and Machine Learning Analysis Model Based on Distributed Algorithm

The model of large-scale data processing and ML(machine learning) analysis based on DA(distributed algorithm) is a powerful computing method, which aims at processing huge data sets and performing efficient ML analysis. In this paper, a cluster topology driver module based on gradient switching and aggregate communication is designed, and its core goal is to adapt the distributed system to various underlying network topologies. By designing decentralized gradient exchange algorithm and aggregate communication framework, the parallel transmission ability of multi-interface network can be fully exerted, thus improving the model synchronization efficiency of ML task. The experimental results show that the cluster topology driver module can provide better performance than the existing methods in terms of training convergence, cluster scalability and communication overhead. Large-scale data processing and ML analysis model based on DA is widely used in processing massive data and realizing complex analysis tasks.

Noise learning of instruments for high-contrast, high-resolution and fast hyperspectral microscopy and nanoscopy

The low scattering efficiency of Raman scattering makes it challenging to simultaneously achieve good signal-to-noise ratio (SNR), high imaging speed, and adequate spatial and spectral resolutions. Here, we report a noise learning (NL) approach that estimates the intrinsic noise distribution of each instrument by statistically learning the noise in the pixel-spatial frequency domain. The estimated noise is then removed from the noisy spectra. This enhances the SNR by ca. 10 folds, and suppresses the mean-square error by almost 150 folds. NL allows us to improve the positioning accuracy and spatial resolution and largely eliminates the impact of thermal drift on tip-enhanced Raman spectroscopic nanoimaging. NL is also applicable to enhance SNR in fluorescence and photoluminescence imaging. Our method manages the ground truth spectra and the instrumental noise simultaneously within the training dataset, which bypasses the tedious labelling of huge dataset required in conventional deep learning, potentially shifting deep learning from sample-dependent to instrument-dependent.

Enhanced Support Vector Machine Methods Using Stochastic Gradient Descent and Its Application to Heart Disease Dataset

Support Vector Machines (SVMs) are supervised learning models used to examine data sets in order to classify or predict dependent variables. SVM is typically used for classification by determining the best hyperplane between two classes. However, working with huge datasets can lead to a number of problems, including time-consuming and inefficient solutions. This research updates the SVM by employing a stochastic gradient descent method. The new approach, the extended stochastic gradient descent SVM (ESGD-SVM), was tested on two simulation datasets. The proposed method was compared with other classification approaches such as logistic regression, naive model, K Nearest Neighbors and Random Forest. The results show that the ESGD-SVM has a very high accuracy and is quite robust. ESGD-SVM is used to analyze the heart disease dataset downloaded from Harvard Dataverse. The entire analysis was performed using the program R version 4.3.

Construction of Handwritten Indus Signs Dataset Employing Social Approach

This paper constructs a dataset of handwritten Indus signs employing a social approach. A writing system called the Indus script was created in the Indus civilization. It has been decoded numerous times throughout the years, but it has not yet been fully deciphered. Due to a lack of information and the scarcity of evidence, the mystery of the Indus signs has not yet been fully solved. Recently, there has been an increase in demand for huge datasets in order to use cutting-edge machine learning techniques. Considering the restricted availability of images of authentic Indus signs, this paper proposes creating an Indus signs dataset by asking participants to draw the Indus signs while referring to the image of the original Indus signs. A web application was developed and used to collect the 44 participants’ handwritten images of ten Indus signs. To show the availability of the constructed dataset, it is used to train convolutional neural networks. The experimental result demonstrates that the model can classify the images of original Indus script with 70% accuracy.

Sustainable resource management and green economic growth: A global prospective

The search for green economic development has taken center stage on the international agenda as nations strive to reconcile economic advancement with environmental sustainability. The present research aims to clarify the complex link between long-term financial viability and environmental responsibility by using a huge dataset from the Chinese energy sector spanning the years 2012–2020. This paper examines the important topic of sustainable resource management and how it supports the growth of green economies globally. This study examines the connection between sustainable resource management techniques and green economic growth using an abundance of published literature, case studies, and empirical data. The increasing urgency of environmental concerns such as depletion of resources, biodiversity loss, and climate change highlights the need for effective resource management. Our research sheds light on the methods and strategies that have been used successfully around the world to support the development of green economies via sustainable resource management.

Pre-trained transformer model as a surrogate in multiscale computational homogenization framework for elastoplastic composite materials subjected to generic loading paths

A composite material typically exhibits complex behavior at the engineer scale, arising from the interactions between its underlying constituent phases, as well as the competitions between micro-processes. It is generally a daunting task to develop an engineering model to adequately capture the essential micro mechanisms that propagate onto the macro scale. To this end, the multiscale computational homogenization (FE2) method enables a consistent coupling across length scales, to give results that compare well with direct numerical simulations having the full micro-structural details, without the need for any constitutive assumptions nor calibrations at the macro scale. Despite its predictive capabilities, the typical computational homogenization method is still computationally too expensive for most practical problems, as the coupling between micro and macro scales are solved simultaneously during its numerical implementation. In this presentation, focusing on the elastoplastic behavior of fiber-reinforced composite, we address this bottleneck with an offline development of a microscopic surrogate model for a given micro-structure, to be incorporated into a standard nonlinear FE framework, for rapid online implementations at the macro scale. For the offline training phase, we adopt the transformer-based architecture within a pre-training and fine-tuning framework. The proposed pre-trained transformer model is capable of parallelizing computations to effectively capture global dependencies within the strain-stress data sequences. To reduce the data generation cost, a constructed source representative volume element (RVE) having a single central heterogeneity with an identical volume fraction with the target RVE is utilized, to rapidly generate a huge source dataset for a pre-training process. The performance of the surrogate model is first demonstrated by comparing its predictions under random loading paths against the reference homogenized RVE responses. Next, the surrogate model is incorporated into a macro FE framework, and its predictive capabilities illustrated via the generic loading of two specimens with different microstructures, each having a different loading–unloading path. Finally, a chunking method is discussed as a potential remedy for managing very long load sequences.

Modifyng modern power systems quality by integrating grid computing technology

The need for reliable, powerful, and clean power generation in power systems is becoming more importance. This need requires geographically-distributed power systems to be integrated as a single entity where among the main features of this integration are large data base and computing intensive. Hence the current power systems are not able to handle this huge datasets required for that integration, Grid Computing is a gateway to virtual storage media and processing power. This paper describes the research work on why grid computing is needed for power systems, what are the major challenges and problem to implement grid computing into power systems, and how grid computing can be utilized to fulfil the requirement for efficient power generation.

Impact of Efficiency on Sustainability of Banks: An Empirical Study of Pakistani Banks

Purpose: This research aims to empirically investigate the impact of efficiency on the sustainable performance of banks in Pakistan through technical efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE). Design/Methodology/Approach: The research methodology that is adopted in this research is Census is based on the ideology of Positivism by testing an old theory with new data to draw new conclusions and new results. It uses mainly quantitative methods. This study has a novelty because a huge set of data was collected. Data Envelopment Analysis (DEA) has been used to calculate relative efficiency by using the DEA method while taking fixed assets, deposits & short-term funding, and equity as inputs, whereas net income securities and loans are outputs. Secondary data from Five years was collected for Pakistani Banks on a time series basis and analyzed using E-Views v10 to test the study’s hypothesis. Findings: The results suggest that efficiency significantly impacts the sustainable performance of banks. Pure technical efficiency has a significant impact on the sustainable performance of banks, in contrast, technical efficiency, and scale efficiency have no impact on the sustainable banks of Pakistan. Practical Implications: This paper will assist banking sector management to focus on efficiency to achieve banks' sustainability. Originality / Value: During the research process, the strands of reviewing literature show that renowned authors recommended the conduct of studies that measures the impact of efficiency on the sustainable performance of banks. This paper has a novelty in that impact of technical efficiency (TE), pure technical efficiency (PTE), and scale efficiency (SE) has also been measured on the sustainable performance of banks in one holistic unison study.

Effective Risk Detection for Natural Gas Pipelines Using Low-Resolution Satellite Images

Natural gas pipelines represent a critical infrastructure for most countries and thus their safety is of paramount importance. To report potential risks along pipelines, several steps are taken such as manual inspection and helicopter flights; however, these solutions are expensive and the flights are environmentally unfriendly. Deep learning has demonstrated considerable potential in handling a number of tasks in recent years as models rely on huge datasets to learn a specific task. With the increasing number of satellites orbiting the Earth, remote sensing data have become widely available, thus paving the way for automated pipeline monitoring via deep learning. This can result in effective risk detection, thereby reducing monitoring costs while being more precise and accurate. A major hindrance here is the low resolution of images obtained from the satellites, which makes it difficult to detect smaller changes. To this end, we propose to use transformers trained with low-resolution images in a change detection setting to detect pipeline risks. We collect PlanetScope satellite imagery (3 m resolution) that captures certain risks associated with the pipelines and present how we collected the data. Furthermore, we compare various state-of-the-art models, among which ChangeFormer, a transformer architecture for change detection, achieves the best performance with a 70% F1 score. As part of our evaluation, we discuss the specific performance requirements in pipeline monitoring and show how the model’s predictions can be shifted accordingly during training.

Flood susceptibility mapping using a novel integration of multi-temporal sentinel-1 data and eXtreme deep learning model

Flash floods (FFs) are amongst the most devastating hazards in arid regions in response to climate change and can cause the loss of agricultural land, human lives and infrastructure. One of the major challenges is the high-intensity rainfall events affecting low-lying areas that are vulnerable to FF. Several works in this field have been conducted using ensemble machine learning models and geohydrological models. However, the current advancement of eXtreme deep learning, which is named eXtreme deep factorisation machine (xDeepFM), for FF susceptibility mapping (FSM) is lacking in the literature. The current study introduces a new model and employs a previously unapplied approach to enhance FSM for capturing the severity of floods. The proposed approach has three main objectives: (i) During- and after-flood effects are assessed through flood detection techniques using Sentinel-1 data. (ii) Flood inventory is updated using remote sensing-based methods. The derived flood effects are implemented in the next step. (iii) An FSM map is generated using an xDeepFM model. Therefore, this study aims to apply xDeepFM to estimate susceptible areas using 13 factors in the emirates of Fujairah, UAE. The performance metrics show a recall of 0.9488), an F1-score of 0.9107), precision of (0.8756) and an overall accuracy of 90.41%. The accuracy of the applied xDeepFM model is compared with that of traditional machine learning models, specifically the deep neural network (78%), support vector machine (85.4%) and random forest (88.75%). Random forest achieves high accuracy, which is due to its strong performance that depends on factors contribution, dataset size and quality, and available computational resources. Comparatively, the xDeepFM model works efficiently for complicated prediction problems having high non-collinearity and huge datasets. The obtained map denotes that the narrow basins, lowland coastal areas and riverbank areas up to 5 km (Fujairah) are highly prone to FF, whilst the alluvial plains in Al Dhaid and hilly regions in Fujairah show low probability. The coastal city areas are bounded by high-rise steep hills and the Gulf of Oman, which can elevate the water levels during heavy rainfall. Four major synchronised influencing factors, namely, rainfall, elevation, drainage density, distance from drainage and geomorphology, account for nearly 50% of the total factors contributing to a very high flood susceptibility. This study offers a platform for planners and decision makers to take timely actions on potential areas in mitigating the effects of FF.

A Survey of Machine Learning Approaches for Mobile Robot Control

Machine learning (ML) is a branch of artificial intelligence that has been developing at a dynamic pace in recent years. ML is also linked with Big Data, which are huge datasets that need special tools and approaches to process them. ML algorithms make use of data to learn how to perform specific tasks or make appropriate decisions. This paper presents a comprehensive survey of recent ML approaches that have been applied to the task of mobile robot control, and they are divided into the following: supervised learning, unsupervised learning, and reinforcement learning. The distinction of ML methods applied to wheeled mobile robots and to walking robots is also presented in the paper. The strengths and weaknesses of the compared methods are formulated, and future prospects are proposed. The results of the carried out literature review enable one to state the ML methods that have been applied to different tasks, such as the following: position estimation, environment mapping, SLAM, terrain classification, obstacle avoidance, path following, learning to walk, and multirobot coordination. The survey allowed us to associate the most commonly used ML algorithms with mobile robotic tasks. There still exist many open questions and challenges such as the following: complex ML algorithms and limited computational resources on board a mobile robot; decision making and motion control in real time; the adaptability of the algorithms to changing environments; the acquisition of large volumes of valuable data; and the assurance of safety and reliability of a robot’s operation. The development of ML algorithms for nature-inspired walking robots also seems to be a challenging research issue as there exists a very limited amount of such solutions in the recent literature.

Optimizing ant colony system algorithm with rule-based data classification for smart aquaculture

<span>Aquaculture is one of many industries where the use of artificial intelligence (AI) techniques has increased dramatically in recent years. Internet of things (IoT), AI, and big data are just a few of the technologies being used in smart aquaculture to increase productivity, efficiency, and system sustainability of aquaculture systems. Data classification, which involves finding patterns and relationships in huge datasets, is one of the most important tasks in smart aquaculture. The ant colony system (ACS) has been used to solve a number of optimization issues, including data classification. To provide a more practical and successful solution, this study proposes an improved ACS algorithm for rule-based data classification in smart aquaculture. The proposed algorithm combines the advantages of ACS and rule-based classification to optimize the number of rules and accuracy. The experimental results showed that the proposed algorithm outperformed the traditional AntMiner algorithm in terms of the number of rules and accuracy. The improved pheromone update technique could potentially increase data classification accuracy and convergence in smart aquaculture systems.</span>

Renewable Energy Forecasting using Deep Learning Techniques

A detailed research on deep learning in renewable energy forecasting shows how sophisticated algorithms may improve prediction accuracy. The research explores deep learning models and finds intriguing aspects that improve predictions. Long Short-Term Memory (LSTM) networks can capture temporal relationships in energy data, making them successful in predicting short-term variations with a prediction accuracy boost of 18.18% over ARIMA. Convolutional Neural Networks (CNNs) capture spatial correlations in huge datasets with up to 13% accuracy. With its capacity to analyze sequential data, Recurrent Neural Networks (RNNs) can capture long-term patterns and improve forecasting accuracy by 29.41% over Support Vector Machines. In addition, LSTM’s better handling of non-linear connections in wind energy data has improved prediction accuracy by 14.29% over feedforward networks. These results demonstrate how deep learning approaches improve renewable energy forecasting with unparalleled precision and dependability. As shown in diverse applications, LSTM, CNN, and RNN models improve renewable energy forecasting efficiency and efficacy, boosting sustainable energy solution innovation.

Everted U-Net for 3D scene reconstruction and segmentation

Abstract. The field of data science related to the processing of three-dimensional objects is becoming more and more relevant. After the successes in image processing, the apotheosis of which was the development of generative neural net-works, the intensification of efforts in the direction of three-dimensional data processing looks logical. Although there are now numerous systems for the reconstruction of three-dimensional objects and other processing, almost all of the existing solutions are aimed at working with a single object. The advances in image processing with neural networks have largely been made possible by huge datasets. There are also large datasets available for model training in this area. Freely avail-able datasets such as ShapeNet and ModelNet contain many thousands of different models, allowing for a high diversity of data. However, most of them provide single individual objects, which allows them to be used in tasks involving the processing of a single three-dimensional object, but when working with scenes containing many objects, there is often a problem of finding an appropriate dataset. This work is aimed at solving the problem of reconstruction and segmentation of three-dimensional scenes, as well as generation of datasets for the task of processing scenes from the real world.

Optimal entropic properties of SARS-CoV-2 RNA sequences.

The reaction of the scientific community against the COVID-19 pandemic has generated a huge (approx. 106 entries) dataset of genome sequences collected worldwide and spanning a relatively short time window. These unprecedented conditions together with the certain identification of the reference viral genome sequence allow for an original statistical study of mutations in the virus genome. In this paper, we compute the Shannon entropy of every sequence in the dataset as well as the relative entropy and the mutual information between the reference sequence and the mutated ones. These functions, originally developed in information theory, measure the information content of a sequence and allows us to study the random character of mutation mechanism in terms of its entropy and information gain or loss. We show that this approach allows us to set in new format known features of the SARS-CoV-2 mutation mechanism like the CT bias, but also to discover new optimal entropic properties of the mutation process in the sense that the virus mutation mechanism track closely theoretically computable lower bounds for the entropy decrease and the information transfer.

Sentiment Analysis for Financial Markets

Abstract: Predicting movements in the stock market is a novel use of sentiment analysis's growing body of knowledge. The purpose of this study is to investigate the potential of NLP for predicting stock market movements by analyzing textual data sources such as news articles, social media posts, as well as earnings reports. The research examines current approaches, applications, and difficulties in sentiment analysis by drawing on extensive surveys and reviews [1], [2]. It also investigates the use of pre-trained models in NLP and the potential biases of such models [6]. Important research findings [3], [17] suggest that NLP-based sentiment analysis models, especially those employing deep learning architectures, show promising results in financial forecasting. There are, however, several difficulties associated with these models. These include the requirement of huge datasets and the elimination of biases. This study has far-reaching ramifications. One benefit is a more nuanced comprehension of the potential and pitfalls of natural language processing for sentiment analysis in the financial markets, which is useful for both analysts and investors. Second, it provides opportunities for more study to enhance the precision and dependability of such models, which ultimately aids in the development of more steady and well-informed monetary judgements.

The role of Big Data and Data Science in the context of information security and cybersecurity

ObjectivesIn the area of the fastest growing fields of ICT technology which determine the successive stages of progress in the field of online electronic banking, it is necessary to disseminate standards for conducting financial operations carried out in the so-called cloud as well as using the large data sets located in the so-called Big Data platforms.Material and methodsFor the purposes of this article, research has been carried out on the broad issue of Big Data Analytics applications in the context of social, normative, economic, environmental, climate aspects. A questionnaire survey was conducted. The method of analysis of literature sources concerning the subject of the article was applied.ResultsCurrently emerging trends in the development of technology based on Big Data dataset platforms usually allow to perform multidimensional calculations and reported results of analyzes in real time. The analyzes conducted on huge data sets allow for comprehensive, multi-aspect risk assessments at the level of the whole entity. During the SARS-CoV-2 (Covid-19) coronavirus pandemic, there was an increase in the importance of analytics based on Big Data Analytics. During the pandemic, there was also an increase in the digitisation and internetisation of business processes. This also resulted in an increase in the scale of cybercrime development.ConclusionsThe technological revolution currently taking place, defined as Industry 4.0, is motivated by the development of the following factors: Big Data database technologies, cloud computing, machine learning, Internet of Things, artificial intelligence, Business Intelligence and other technologies of advanced data processing Data Mining. On the basis of the development new technological solutions, in recent years there has been a dynamic development of Big Data Analytics.

Harnessing Machine Learning for Effective Cyber security Classifiers

Machine learning has emerged as a transformative force, innovating diverse industries through its capacity to infuse meaningful insights from large datasets. It plays a pivotal role in powering data analysis, discover pattern matching, identifying hidden or evolving risks in securing systems. The ability of categorizing and behavior analysis is central to its efficacy in cybersecurity. This paper highlights the importance of machine learning in landscape of cyber threats. In this paper, we have identified few machine learning algorithms to categorize huge dataset. The complexities of identifying hidden risks increases by many folds, when the input data is voluminous. Evaluating and contemplating the underlying meaning of data is time-consuming and can be missed easily. We compared different types of machine learning algorithms. Each machine learning algorithm has its strength and weakness. It is found that, the TressJ48 algorithm is proficient in classifying the large dataset, better than Naive Bayes and Decision Stump algorithms. The efficient classifier helps to generate insight, which can be further used to make decisions in terms of cybersecurity.

Bioremediation of Recalcitrant Xenobiotic Compounds: Advancements and Challenges

Widespread recalcitrant xenobiotic compounds have posed serious environmental threats. These compounds are known to contaminate groundwater, crops, water bodies, and lands, affecting the lives of people living nearby. Sites polluted with organic contaminants like polycyclic aromatic hydrocarbons (PAHs), chlorinated hydrocarbons (CHC), phenols, and aromatic hydrocarbons (benzene-toluene-ethylbenzene-xylenes, BTEX) are located worldwide, in both developed and developing countries. Many pollutants have been classified as persistent organic pollutants (POPs) whose production and usage have been either limited or banned by the Stockholm Convention. In May 2001, the Stockholm Convention gave a list of 12 POPs and called them “dirty dozens,” which has now increased to 36 POPs. Much advancement has been made in the last decade; however, on-field bioremediation techniques are still not well developed. Detection of ongoing remediation in trials by using metagenomics, metatranscriptomics, metabolomics, and metaproteomics produces huge data sets. These in-silico analyses required efficient algorithms and skilled professionals to contemplate data analyses. Carbon-stable isotope analysis, microarrays, and high-throughput cultivation techniques are also promising methods that have helped researchers track bioremediation. However, challenges faced by the scientific community are numerous, which has affected the pace of advancements in achieving a defined strategy for bioremediation of xenobiotic compounds. This review summarises the need for remediation of recalcitrant xenobiotic compounds that pose a threat to the environment and human health. Also, advancements made in the field of bioremediation and the problems posed in the implementation of designed strategies have been summed up.