Big Domain Research Articles

Combating trade in illegal wood and forest products with machine learning.

Trade in wood and forest products spans the global supply chain. Illegal logging and associated trade in forest products present a persistent threat to vulnerable ecosystems and communities. Illegal timber trade has been linked to violations of tax and conservation laws, as well as broader transnational crimes. The United States is the largest importer globally of wood and forest products, such as pulp, paper, flooring, and furniture-importing $78 billion in 2021. Transaction-level data such as shipping container manifests and bills of lading provide a comprehensive data source that can be used to detect and disrupt trade that may be suspected of containing illegally harvested or traded forest products. Owing to the volume, velocity, and complexity of shipment data, an automated decision support system is required for the purposes of detecting suspicious forest product shipments. We present a proof of concept framework using machine learning and big data approaches-combining domain expertise with automation-to achieve this objective. We formulated the underlying machine learning problem as an anomaly detection problem and collected and collated forest sector-specific domain knowledge to filter and target shipments of interest. In this work, we provide the overview of our framework, with the details of domain knowledge extraction and machine learning models, and discuss initial results and analysis of flagged anomalous and potentially suspicious records to demonstrate the efficacy of this approach. The proof of concept work presented here provides the groundwork for an actionable and feasible approach to assisting enforcement agencies with the detection of suspicious shipments that may contain illegally harvested or traded wood.

Smart Data Driven Decision Trees Ensemble Methodology for Imbalanced Big Data

Differences in data size per class, also known as imbalanced data distribution, have become a common problem affecting data quality. Big Data scenarios pose a new challenge to traditional imbalanced classification algorithms, since they are not prepared to work with such amount of data. Split data strategies and lack of data in the minority class due to the use of MapReduce paradigm have posed new challenges for tackling the imbalance between classes in Big Data scenarios. Ensembles have been shown to be able to successfully address imbalanced data problems. Smart Data refers to data of enough quality to achieve high-performance models. The combination of ensembles and Smart Data, achieved through Big Data preprocessing, should be a great synergy. In this paper, we propose a novel Smart Data driven Decision Trees Ensemble methodology for addressing the imbalanced classification problem in Big Data domains, namely SD_DeTE methodology. This methodology is based on the learning of different decision trees using distributed quality data for the ensemble process. This quality data is achieved by fusing random discretization, principal components analysis, and clustering-based random oversampling for obtaining different Smart Data versions of the original data. Experiments carried out in 21 binary adapted datasets have shown that our methodology outperforms random forest.

Knowledge of non-communicable diseases and access to healthcare services among adults before and during COVID-19 pandemic in rural Tanzania.

The COVID-19 pandemic significantly affected access to healthcare services, particularly among individuals living with Non-Communicable Diseases (NCDs) who require regular healthcare visits. Studies suggest that knowledge about a specific disease is closely linked to the ability to access services for that condition. In preparation for the future, we conducted the study to assess knowledge of NCDs and access to healthcare services among adults residing in rural areas before and during the COVID-19 pandemic. We conducted a community-based cross-sectional study in rural Tanzania in October 2022, a few months after the end of the third wave of the COVID-19 pandemic. A total of 689 community residents participated in the study. The level of knowledge of NCDs was assessed using an 11-item Likert questionnaire, which was later dichotomized into adequate and inadequate levels of knowledge. In addition, access to healthcare was assessed before and during the pandemic. We summarized the results using descriptive statistics and logistic regression was applied to determine factors associated with adequate levels of knowledge of NCDs. All statistical tests were two-sided; a p-value <0.05 was considered statistically significant. All data analyses were performed using SPSS. Among 689 participants, more than half 369 (55%) had adequate knowledge of whether a disease is NCD or not; specifically, 495 (73.8%), 465 (69.3%), and 349 (52%) knew that hypertension, diabetes mellitus, and stroke are NCDs while 424 (63.2%) know that UTI is not NCD. Of the interviewed participants, 75 (11.2%) had at least one NCD. During the COVID-19 pandemic the majority 57 (72.2%) accessed healthcare services from nearby health facilities followed by traditional healers 10 (12.7%) and community drug outlets 8 (10.1%). Residence and education level were found to be significantly associated with knowledge of NCDs among participants. The study revealed that the community has a moderate level of knowledge of NCDs, and during the COVID-19 pandemic outbreaks, people living with NCD (s) relied on nearby health facilities to obtain healthcare services. Health system preparedness and response to pandemics should take into account empowering the community members to understand that NCD care is continuously needed even during pandemic times. We further advocate for a qualitative study to explore contextual factors influencing the knowledge of NCDs and access to healthcare services beyond the big domains of education and residence.

A high-performance electromagnetic code to simulate high-temperature superconductors

Superconductivity is a physical phenomenon of some materials that allow them to conduct electrical current without resistance. This property has a wide range of applications. In particular, type-II high-temperature superconductors appear promising for building strong electromagnets carrying large amounts of current in the extreme conditions foreseen for future fusion reactors. However, their fabrication and related experiments are highly expensive and complex. Therefore, there is an increasing need for numerical models to guide the design optimization of superconducting cables and to predict their performance. In this work, we present a new code to simulate high-temperature superconductors (HTS) based on the edge finite element discretization of Maxwell’s equations in the time domain using the widely adopted H-formulation in the superconductor analysis community. This code is integrated in the High-Performance Computing (HPC) Alya suite and obtain an excellent performance up to 1024 processors in MareNostrum 4 supercomputer. This capacity allows to us to solve a wide variate of problems in big domains in relatively reduce amount of computer time, being a promising tool to aboard HTS coils design problems. Furthermore, validations against experimental data are presented and code results for superconducting tapes with different magnetic properties are analyzed.

Rapid spatio-temporal flood modelling via hydraulics-based graph neural networks

Abstract. Numerical modelling is a reliable tool for flood simulations, but accurate solutions are computationally expensive. In recent years, researchers have explored data-driven methodologies based on neural networks to overcome this limitation. However, most models are only used for a specific case study and disregard the dynamic evolution of the flood wave. This limits their generalizability to topographies that the model was not trained on and in time-dependent applications. In this paper, we introduce shallow water equation–graph neural network (SWE–GNN), a hydraulics-inspired surrogate model based on GNNs that can be used for rapid spatio-temporal flood modelling. The model exploits the analogy between finite-volume methods used to solve SWEs and GNNs. For a computational mesh, we create a graph by considering finite-volume cells as nodes and adjacent cells as being connected by edges. The inputs are determined by the topographical properties of the domain and the initial hydraulic conditions. The GNN then determines how fluxes are exchanged between cells via a learned local function. We overcome the time-step constraints by stacking multiple GNN layers, which expand the considered space instead of increasing the time resolution. We also propose a multi-step-ahead loss function along with a curriculum learning strategy to improve the stability and performance. We validate this approach using a dataset of two-dimensional dike breach flood simulations in randomly generated digital elevation models generated with a high-fidelity numerical solver. The SWE–GNN model predicts the spatio-temporal evolution of the flood for unseen topographies with mean average errors in time of 0.04 m for water depths and 0.004 m2 s−1 for unit discharges. Moreover, it generalizes well to unseen breach locations, bigger domains, and longer periods of time compared to those of the training set, outperforming other deep-learning models. On top of this, SWE–GNN has a computational speed-up of up to 2 orders of magnitude faster than the numerical solver. Our framework opens the doors to a new approach to replace numerical solvers in time-sensitive applications with spatially dependent uncertainties.

Optimizing Big Data Analytics for Reliability and Resilience: A Survey of Techniques and Applications

The advent of big data has revolutionized various industries, enabling organizations to make data- driven decisions and gain valuable insights. However, the sheer volume, velocity, and variety of big data pose significant challenges in ensuring the reliability and resilience of big data analytics pipelines. In this context, optimization techniques play a crucial role in enhancing the reliability and resilience of big data analytics. This paper provides a comprehensive survey of optimization techniques for reliable and resilient big data analytics. The paper first discusses the motivation for optimizing big data analytics for reliability and resilience. Then, it presents a detailed overview of various optimization techniques, including resource optimization, data partitioning, data compression, load balancing, and fault detection and tolerance. Finally, the paper discusses the applications of optimization techniques in various big data analytics domains, such as real-time analytics, fraud detection, recommendation systems, predictive analytics, and risk management.

Efficient Estimation and Validation of Shrinkage Estimators in Big Data Analytics

Shrinkage estimators are often used to mitigate the consequences of multicollinearity in linear regression models. Despite the ease with which these techniques can be applied to small- or moderate-size datasets, they encounter significant challenges in the big data domain. Some of these challenges are that the volume of data often exceeds the storage capacity of a single computer and that the time required to obtain results becomes infeasible due to the computational burden of a high volume of data. We propose an algorithm for the efficient model estimation and validation of various well-known shrinkage estimators to be used in scenarios where the volume of the data is large. Our proposed algorithm utilises sufficient statistics that can be computed and updated at the row level, thus minimizing access to the entire dataset. A simulation study, as well as an application on a real-world dataset, illustrates the efficiency of the proposed approach.

Generative AI advances for data-driven insights in IoT, cloud technologies, and big data challenges

Generative AI has emerged as a transformative force in addressing complex challenges within IoT, cloud technologies, and big data ecosystems. This paper explores the multifaceted role of Generative AI in enabling intelligent data analysis, optimizing resource allocation, and overcoming the inherent complexities of big data processing. It identifies key barriers, including scalability limitations, ethical concerns, and regulatory constraints, and underscores the need for responsible adoption. The paper highlights innovative applications such as predictive analytics, anomaly detection, and advanced simulations, demonstrating how Generative AI fosters system optimization and autonomous decision-making. Finally, actionable recommendations are proposed for researchers, industry practitioners, and policymakers to collaboratively harness the potential of Generative AI while ensuring ethical and sustainable practices. By integrating Generative AI across IoT, cloud, and big data domains, stakeholders can unlock new avenues for technological advancement and societal benefit.

The cryo-EM structure of the BoNT/Wo-NTNH complex reveals two immunoglobulin-like domains.

The botulinum neurotoxin-like toxin from Weissella oryzae (BoNT/Wo) is one of the BoNT-like toxins recently identified outside of the Clostridium genus. We show that, like the canonical BoNTs, BoNT/Wo forms a complex with its non-toxic non-hemagglutinin (NTNH) partner, which in traditional BoNT serotypes protects the toxin from proteases and the acidic environment of the hosts' guts. We here report the cryo-EM structure of the 300 kDa BoNT/Wo-NTNH/Wo complex together with pH stability studies of the complex. The structure reveals molecular details of the toxin's interactions with its protective partner. The overall structural arrangement is similar to other reported BoNT-NTNH complexes, but NTNH/Wo uniquely contains two extra bacterial immunoglobulin-like (Big) domains on the C-terminus. Although the function of these Big domains is unknown, they are structurally most similar to bacterial proteins involved in adhesion to host cells. In addition, the BoNT/Wo protease domain contains an internal disulfide bond not seen in other BoNTs. Mass photometry analysis revealed that the BoNT/Wo-NTNH/Wo complex is stable under acidic conditions and may dissociate at neutral to basic pH. These findings established that BoNT/Wo-NTNH/Wo shares the general fold of canonical BoNT-NTNH complexes. The presence of unique structural features suggests that it may have an alternative mode of activation, translocation and recognition of host cells, raising interesting questions about the activity and the mechanism of action of BoNT/Wo as well as about its target environment, receptors and substrates.

BIGQA: Declarative Big Data Quality Assessment

In the big data domain, data quality assessment operations are often complex and must be implementable in a distributed and timely manner. This article tries to generalize the quality assessment operations by providing a new ISO-based declarative data quality assessment framework (BIGQA). BIGQA is a flexible solution that supports data quality assessment in different domains and contexts. It facilitates the planning and execution of big data quality assessment operations for data domain experts and data management specialists at any phase in the data life cycle. This work implements BIGQA to demonstrate its ability to produce customized data quality reports while running efficiently on parallel or distributed computing frameworks. BIGQA generates data quality assessment plans using straightforward operators designed to handle big data and guarantee a high degree of parallelism when executed. Moreover, it allows incremental data quality assessment to avoid reading the whole dataset each time the quality assessment operation is required. The result was validated using radiation wireless sensor data and Stack Overflow users’ data to show that it can be implemented within different contexts. The experiments show a 71% performance improvement over a 1 GB flat file on a single processing machine compared with a non-parallel application and a 75% performance improvement over a 25 GB flat file within a distributed environment compared to a non-distributed application.

Cross-Modal Distillation for Speaker Recognition

Speaker recognition achieved great progress recently, however, it is not easy or efficient to further improve its performance via traditional solutions: collecting more data and designing new neural networks. Aiming at the fundamental challenge of speech data, i.e. low information density, multimodal learning can mitigate this challenge by introducing richer and more discriminative information as input for identity recognition. Specifically, since the face image is more discriminative than the speech for identity recognition, we conduct multimodal learning by introducing a face recognition model (teacher) to transfer discriminative knowledge to a speaker recognition model (student) during training. However, this knowledge transfer via distillation is not trivial because the big domain gap between face and speech can easily lead to overfitting. In this work, we introduce a multimodal learning framework, VGSR (Vision-Guided Speaker Recognition). Specifically, we propose a MKD (Margin-based Knowledge Distillation) strategy for cross-modality distillation by introducing a loose constrain to align the teacher and student, greatly reducing overfitting. Our MKD strategy can easily adapt to various existing knowledge distillation methods. In addition, we propose a QAW (Quality-based Adaptive Weights) module to weight input samples via quantified data quality, leading to a robust model training. Experimental results on the VoxCeleb1 and CN-Celeb datasets show our proposed strategies can effectively improve the accuracy of speaker recognition by a margin of 10% ∼ 15%, and our methods are very robust to different noises.

Crystal structure of a variable region segment of Leptospira host-interacting outer surface protein, LigA, reveals the orientation of Ig-like domains

Leptospiral immunoglobulin-like (Lig) protein family is a surface-exposed protein from the pathogenic Leptospira. The Lig protein family has been identified as an essential virulence factor of L. interrogan. One of the family members, LigA, contains 13 homologous tandem repeats of bacterial Ig-like (Big) domains in its extracellular portion. It is crucial in binding with the host's Extracellular matrices (ECM) and complement factors. However, its vital role in the invasion and evasion of pathogenic Leptospira, structural details, and domain organization of the extracellular portion of this protein are not explored thoroughly. Here, we described the first high-resolution crystal structure of a variable region segment (LigA8–9) of LigA at 1.87 Å resolution. The structure showed some remarkably distinctive aspects compared with other closely related Immunoglobulin domains. The structure illustrated the relative orientation of two domains and highlighted the role of the linker region in the domain orientation. We also observed an apparent electron density of Ca2+ ions coordinated with a proper interacting geometry within the protein. Molecular dynamic simulations demonstrated the involvement of a linker salt bridge in providing rigidity between the two domains. Our study proposes an overall arrangement of Ig-like domains in the LigA protein. The structural understanding of the extracellular portion of LigA and its interaction with the ECM provides insight into developing new therapeutics directed toward leptospirosis.

Recent advances in agricultural disease image recognition technologies: A review

SummaryIn the world of intelligent agriculture, agricultural disease picture detection plays a critical role. Plant disease identification must be efficient if agricultural production is to be increased sustainably. Anomalies in plants affected by viruses, insects, nutritional deficiencies, or poor weather conditions have traditionally been diagnosed by human experts. Deep learning (DL) and transfer learning, two sophisticated machine learning techniques synonymous with the advancement of artificial intelligence technologies in recent years, have started to be used for the identification of agricultural diseases. However, a variety of significant obstacles exist in the way of widespread use of these approaches. This paper looks at DL and transfer learning in particular, and discusses recent developments in using these advanced technologies to recognize agricultural disease images. Many DL architectures have recently been adopted, along with visualization tools, which are critical for identifying signs and classifying plant diseases. The analysis and evaluation of these two approaches show that transfer learning is the better choice given existing agricultural disease data tools. The paper then looks at the key problems that need to be addressed for research in this area to advance, such as image dataset creation, the collection of big data auxiliary domains, and the optimization of the transfer learning process. The construction of technically feasible agricultural disease image recognition systems requires the creation of image datasets collected under real cultivation conditions. This survey aims to learn more about DL capabilities in plant disease detection to increase device efficiency and accuracy in future research.

Situation-Aware IoT Data Generation towards Performance Evaluation of IoT Middleware Platforms.

With the increasing growth of IoT applications in various sectors (e.g., manufacturing, healthcare, etc.), we are witnessing a rising demand of IoT middleware platform that host such IoT applications. Hence, there arises a need for new methods to assess the performance of IoT middleware platforms hosting IoT applications. While there are well established methods for performance analysis and testing of databases, and some for the Big data domain, such methods are still lacking support for IoT due to the complexity, heterogeneity of IoT application and their data. To overcome these limitations, in this paper, we present a novel situation-aware IoT data generation framework, namely, SA-IoTDG. Given a majority of IoT applications are event or situation driven, we leverage a situation-based approach in SA-IoTDG for generating situation-specific data relevant to the requirements of the IoT applications. SA-IoTDG includes a situation description system, a SySML model to capture IoT application requirements and a novel Markov chain-based approach that supports transition of IoT data generation based on the corresponding situations. The proposed framework will be beneficial for both researchers and IoT application developers to generate IoT data for their application and enable them to perform initial testing before the actual deployment. We demonstrate the proposed framework using a real-world example from IoT traffic monitoring. We conduct experimental evaluations to validate the ability of SA-IoTDG to generate IoT data similar to real-world data as well as enable conducting performance evaluations of IoT applications deployed on different IoT middleware platforms using the generated data. Experimental results present some promising outcomes that validate the efficacy of SA-IoTDG. Learning and lessons learnt from the results of experiments conclude the paper.

Text Data Analysis Using Generalized Linear Mixed Model and Bayesian Visualization

Many parts of big data, such as web documents, online posts, papers, patents, and articles, are in text form. So, the analysis of text data in the big data domain is an important task. Many methods based on statistics or machine learning algorithms have been studied for text data analysis. Most of them were analytical methods based on the generalized linear model (GLM). For the GLM, text data analysis is performed based on the assumption of the error included in the given data and follows the Gaussian distribution. However, the GLM has shown limitations in the analysis of text data, including data sparseness. This is because the preprocessed text data has a zero-inflated problem. To solve this problem, we proposed a text data analysis using the generalized linear mixed model (GLMM) and Bayesian visualization. Therefore, the objective of our study is to propose the use of GLMM to overcome the limitations of the conventional GLM in the analysis of text data with a zero-inflated problem. The GLMM uses various probability distributions as well as Gaussian for error terms and considers the difference between observations by clustering. We also use Bayesian visualization to find meaningful associations between keywords. Lastly, we carried out the analysis of text data searched from real domains and provided the analytical results to show the performance and validity of our proposed method.

Median Filters on FPGAs for Infinite Data and Large, Rectangular Windows

Efficient architectures and implementations of median filters have been well investigated in the past. In this article, we focus on median filters for very big scientific applications with very large windows and an infinite stream of data, inspired by big data needs in the Square Kilometre Array (SKA) pulsar search engine, but transferable to other big data domains. We propose a novel approach for very large rectangular windows on an FPGA accelerator device able to support the processing of infinite streams of data. OpenCL is used for rapid parameter sweeping and design space exploration based on a pipelined model of the system. Evaluation on a host/accelerator system with an Arria 10 device surpassed 64 million values processed per second considered for the SKA real time requirement, achieving 83.4M value/s while reading from/writing to disk. These results are compared with a state-of-the-art software implementation only achieving 41M value/s for over twice the total system energy cost.

Mesogen-jacketed liquid crystal like block copolymers: synthesis and self-assembly to prepare 1D PC thin films

One dimensional photonic crystals (1D PCs) show great potential application in light control, sensors and functional displays. Self-assembly of block copolymer is a facile method to prepare 1D PCs in large area and easy processing. However, due to the low refractive index of polymer and the chain entanglement at very high molecular weight, it is difficult to prepare 1D PCs by self-assembly of linear block copolymers. In this work, mesogen-jacketed liquid crystals like block copolymers with two different lumbar connected benzene terephthalate derivatives and polynorbornene backbone are designed and synthesized. The big steric hindrance of the mesogenic benzene terephthalate derivatives extends the polymer backbone to be rigid, and the resultant block copolymers are rapidly self-assembled into lamellar structures with big domain spacing. The refractive indexes of the two blocks is differentiated by selecting the phenyl and decyl substitute group of benzene terephthalate derivatives. Hence, 1D PCs are prepared by self-assembly of mesogen-jacketed liquid crystals like block copolymers. The maximum reflectance is ranged from ultraviolet to near infrared by increasing the polymerization degree of the block copolymers from 400 to 1400 gradually. The lamellar structures are characterized by cross section scanning electron microscope and small angle X ray scattering. Mesogen-jacketed liquid crystal like block copolymer is favored in 1D PC preparation for its facile synthesis and flexible structure design.

Transparent boundary condition for simulating rogue wave solutions in the nonlinear Schrödinger equation.

This paper addresses the construction of numerical boundary conditions for simulating rogue wave solutions in the nonlinear Schrödinger equation. While three kinds of commonly used boundary conditions require a big enough computational domain to reproduce solutions faithfully in the central domain, we propose transparent boundary conditions for the Peregrine soliton and Kuznetsov-Ma breather solutions, respectively. For both solutions, these boundary conditions require a smaller computational domain than other boundary conditions to attain the best accuracy of the Crank-Nicolson scheme and selected mesh size, which will be referred to as the "acceptable accuracy" below. In particular, the computational domain with these boundary conditions is only 1/16 as small as others in the simulations of the Peregrine soliton solution. As a result, they reduce both the memory requirement and the computing time for the Peregrine soliton solution.

Security and Privacy Challenges of Big Data Adoption: A Qualitative Study in Telecommunication Industry

The telecommunication industry is the leading industry in big data trends as this industry has the most capable infrastructure for big data. However, the adoption of big data in telecommunication services also raises important security and privacy challenges due to the high volume, velocity, and variety of big data characteristics. To address the issue, this study shed light on the security and privacy challenges of big data adoption in the telecommunication industry. This study focuses on investigating the security and privacy challenges for data users in telecommunication services from the lens of the TOE framework and examines the mitigation strategies to address the privacy and security challenges. This study is conducted using qualitative methodology. From the perspectives of data users (telecommunication providers), it could be concluded that data management, data privacy, data compliance, and regulatory orchestration challenges are the most pressing concerns in big data adoption. This study offers contributions in presenting a thematic classification of security and privacy challenges and their mitigation strategies for big data adoption in the telecommunication industry. The thematic classification highlights potential gaps for future research in the big data security domain. This study is significant in that it provides empirical evidence for the perspectives of telecommunication data users in addressing privacy and security issues that are related to big data adoption.

Construction Project Management with Digital Twin Information System

The current stage in the development of the construction industry is its digitalization, which requires the transformation of processes and models based on the use of digital platforms and twins. Datasets in a construction project accumulates quickly and becomes difficult to store and process due to the large file sizes of CAD,GIS and BIM technologies. Managing such datasets is a complex problem, since the usefulness of such data lies in ensuring that it is available and used as needed by all participants in a construction project. The article proposes to develop an information system for creating digital twins of construction project management as an integrator of digital tools based on Big Data Analytics and BIM technology, which is already the standard for the digitalization of the construction industry. The study describes four types of digital twins and ten properties of Big Data required for construction project. Conceptual model of Big Data domains for construction project and BIM-model of Big Data exchange and information transfer for construction project digital twin are considered. The authors propose a framework for creating digital twins of construction project management that uses three main components: BIM-models, Big Data Analytics and Knowledge Base. The results of the study are structure of information system of creating digital twin of territory urban planning project for Design Company and example using a BIMoriented software product. Keywords—BIM-model, Big Data, digital twin, construction projectmanagement, digital construction