Data Analysis Framework Research Articles

Multi-Agent Systems Requirements Analysis for Patient-centered Healthcare Consultancy Service

Agents are currently being discussed in nearly every domain of science and engineering. Through the utilization of IoT and agent-based systems, remote consultations and virtual doctors can provide essential healthcare services by analyzing patient data and medical history. The purpose of this paper is provide an analysis the requirements for multi-agent systems in patient-centered health consultancy services in Kenya. It focuses on various agent types, including hardware, hospital, human, software, database/storage, and technological/framework agents. For hardware, computers, laptops, and mobile devices need specific processing power, memory, network connectivity, storage, and security. Medical equipment must prioritize accuracy, reliability, integration with records, and regulatory compliance. Networking requires high-speed internet, security, and redundancy. Hospital agents offer comprehensive inpatient and outpatient care, categorized into general, mental health, and specialized hospitals. Human agents, like medical specialists, IT professionals, and administrators, play essential roles in healthcare consultancy. Software agents encompass Electronic Health Records (EHRs), Clinical Decision Support Systems (CDSS), Telemedicine platforms, and more. Database/storage agents include EHRs, Health Information Exchange (HIE) systems, and data encryption. Technological/framework agents involve health information systems, telemedicine platforms, decision support systems, and health data analytics frameworks. These agents enable efficient data management, remote care, evidence-based decisions, and secure data exchange.

New features of the RootProf program for model-free analysis of unidimensional profiles

The RootProf computer program applies multivariate model-free analysis to crystallographic data and to any x, y experimental data in general. It has been enhanced with several new features, including a graphical user interface, multithreading implementation and additional pre-processing options. The program also includes novel qualitative analysis methods, such as semiquantitative estimates derived from principal component analysis (PCA) and restrained PCA to extract the diffraction signal from active atoms. Additional quantitative analysis methods have been included, involving the combination of different datasets or the application of the standard addition method as well as tools for crystallinity analysis, kinetic analysis and extraction of free crystal cell parameters from a pair distribution function profile. The ROOT data analysis framework supports the program and can be installed on the current major platforms such as Windows, Linux and Mac OSX with detailed user documentation included. Applications of the new developments are presented and discussed in the paper, and related command files are provided as supporting information.

Geographically distributed data management to support large-scale data analysis

Nowadays, several companies prefer storing their data on multiple data centers with replication for many reasons. The data that spans various data centers ensures the fastest possible response time for customers and workforces who are geographically separated. It also provides protecting the information from the loss in case a single data center experiences a disaster. However, the amount of data is increasing at a rapid pace, which leads to challenges in storage, analysis, and various processing tasks. In this paper, we propose and design a geographically distributed data management framework to manage the massive data stored and distributed among geo-distributed data centers. The goal of the proposed framework is to enable efficient use of the distributed data blocks for various data analysis tasks. The architecture of the proposed framework is composed of a grid of geo-distributed data centers connected to a data controller (DCtrl). The DCtrl is responsible for organizing and managing the block replicas across the geo-distributed data centers. We use the BDMS system as the installed system on the distributed data centers. BDMS stores the big data file as a set of random sample data blocks, each being a random sample of the whole data file. Then, DCtrl distributes these data blocks into multiple data centers with replication. In analyzing a big data file distributed based on the proposed framework, we randomly select a sample of data blocks replicated from other data centers on any data center. We use simulation results to demonstrate the performance of the proposed framework in big data analysis across geo-distributed data centers.

An implementation of Hurdle models for spatial count data. Study case: civil war as a risk factor for the development of childhood leukemia in Colombia

We propose a novel, efficient, and powerful methodology to deal with overdispersion, excess zeros, heterogeneity, and spatial correlation. It is based on the combination of Hurdle models and Spatial filtering Moran eigenvectors. Hurdle models are the best option to manage the presence of overdispersion and excess of zeros, separating the model into two parts: the first part models the probability of the zero value, and the second part models the probability of the non-zero values. Finally, gathering the spatial information in new covariates through a spatial filtering Moran vector method involves spatial correlation and spatial heterogeneity to improve the model fitting and explain spatial effects of variables that were not possible to measure. Thus, our proposal adapts usual regression models for count data so that it is possible to deal with phenomena where the usual theoretical assumptions, such as constant variance, independence, and unique distribution are not fulfilled. In addition, this research shows how a prolonged armed conflict can impact the health of children. The data includes children exposed to armed conflict in Colombia, a country enduring a non-international armed conflict lasting over 60 years. The findings indicate that children exposed to high levels of violence, as measured by the armed conflict index, demonstrate a significant association with the incidence and mortality rate of LAP in children. This fact is illustrated here using one of the most catastrophic conditions in childhood, as is Pediatric Acute Leukemia (LAP). The association between armed conflict and LAP has its conceptual basis in the epidemiology literature, given that, the incidence and mortality rates of neoplastic diseases increase with exposure to toxic and chronic stress during gestation and childhood. Our methodology provides a valuable framework for complex data analysis and contributes to understanding the health implications in conflict-affected regions.

Pynapple, a toolbox for data analysis in neuroscience.

Datasets collected in neuroscientific studies are of ever-growing complexity, often combining high-dimensional time series data from multiple data acquisition modalities. Handling and manipulating these various data streams in an adequate programming environment is crucial to ensure reliable analysis, and to facilitate sharing of reproducible analysis pipelines. Here, we present Pynapple, the PYthon Neural Analysis Package, a lightweight python package designed to process a broad range of time-resolved data in systems neuroscience. The core feature of this package is a small number of versatile objects that support the manipulation of any data streams and task parameters. The package includes a set of methods to read common data formats and allows users to easily write their own. The resulting code is easy to read and write, avoids low-level data processing and other error-prone steps, and is open source. Libraries for higher-level analyses are developed within the Pynapple framework but are contained within a collaborative repository of specialized and continuously updated analysis routines. This provides flexibility while ensuring long-term stability of the core package. In conclusion, Pynapple provides a common framework for data analysis in neuroscience.

A Microservices-based Framework for Scalable Data Analysis in Agriculture with IoT Integration

We propose a microservices-based framework for scalable data analysis in agriculture with IoT integration, leveraging the flexibility and modularity of microservices architecture to build a highly adaptable, maintainable, and efficient data analysis system. This framework allows for faster data processing and carry a diversity of agricultural data analysis tasks while maintaining scalability and fault tolerance. Despite the potential benefits, several challenges and obstacles need to be addressed, such as data integration and standardization, the development of agricultural-specific analytical microservices, and ensuring data security and privacy. Practical application and real-world validation are required to assess the impact of the proposed framework on the agricultural sector and inform future research directions.

Tensor generalized canonical correlation analysis

Regularized Generalized Canonical Correlation Analysis (RGCCA) is a general statistical framework for multiblock data analysis. RGCCA enables deciphering relationships between several sets of variables and subsumes many well-known multivariate analysis methods as special cases. However, RGCCA only deals with vector-valued blocks, disregarding their possible higher-order structures. This paper presents Tensor GCCA (TGCCA), a new method for analyzing higher-order tensors with canonical vectors admitting an orthogonal rank-R CP decomposition. Moreover, two algorithms for TGCCA, based on whether a separable covariance structure is imposed or not, are presented along with convergence guarantees. The efficiency and usefulness of TGCCA are evaluated on simulated and real data and compared favorably to state-of-the-art approaches.

Explore Testing Performance and Learning Behaviors

ABSTRACT This study explores various approaches to investigate participants’ testing performance and learning behaviors in a computer-based spatial rotation learning program. Using multivariate learning and assessment data, including responses, response times, learning times and selected covariates, a comprehensive data analytic framework is developed that not only utilizes the test level information but also the item level information. This top-down and multivariate data analytic framework can shed light on conducting exploratory analysis with high-dimensional and mixed-type multivariate data, especially on how to aggregate information from the test-level and item-level. The findings about participants’ testing performance and learning behaviors are valuable in guiding the design of an adaptive learning platform in the future and can also provide some support in developing confirmatory statistical methods to model testing and learning behaviors.

Deep learning based processing framework for spatio-temporal analysis and classification of laser biospeckle data

Laser biospeckle is an advanced optical technique with the ability to non-destructively visualize various transient phenomenon via spatial and temporal statistics. However, accuracy of the existing image processing techniques used to process biospeckle data is hampered by various experimental and processing dependent factors. Therefore, in this work, a novel 3D convolution neural network (3D CNN) based deep learning (DL) architecture is developed for spatio-temporal analysis of biospeckle data in both qualitative and quantitative domains that effectively reduces errors introduced due to the influence of varying experimental parameters. Firstly, 3D CNN based image processing model is proposed for spatio-temporal analysis and classification of biospeckle data. Furthermore, a novel DL based numerical indexing strategy is developed for identification of level of activity in a sample. Finally, impact of varying experimental parameters on accuracy of the proposed technique is analyzed. In this direction, multiple experiments were performed to examine the effect of variation in input data parameters such as frame dimension, frame rate, number of frames, and background noise on accuracy of the trained model. Performance of the proposed model was analyzed and compared with respect to synthetic data generated by using rotating diffuser based simulation model. Robustness of the proposed strategy was also validated experimentally on practical data associated with identification of disease in seeds. Obtained results demonstrated that the proposed technique is accurate and can perform spatio-temporal classification and numerical indexing of the biospeckle data under varying experimental parameters.

Framework for liner shipping connectivity data analytics and research

PurposeThis research proposes a framework to conceptualise the potential realm of data regarding shipping connectivity for application of data analytics which can be used to generate deeper insights with respect to the state of such linkages and potential areas for practical application.Design/methodology/approachThe study method involved comprehensive presentation of different perspectives of assessing shipping connectivity and levels of data contained within container shipping services and proposed potential application to analyse profitability, performance, competitiveness, risk and environmental impact.FindingsAdvances in capabilities to handle large volumes of data offer scope for an integrated approach which utilises all available data from various stakeholders in analyses of liner shipping connectivity. Research shows how different types of data contained in container shipping services are related and can be organised for application of data analytics.Research limitations/implicationsResearch implications are offered to shipping lines, port managers and operators and policymakers.Practical implicationsThis research presented a conceptual framework that captures the range of data involved in container shipping services and how data analytics can be practically applied in an integrated manner.Originality/valueThis paper is the first in literature to discuss in detail the different levels of data that reside within shipping services that constitute liner shipping connectivity for application of data analytics.

Turkish Adaptation of Principal-Teacher Relationship Scale: Validity and Reliability Study

In the study, the Principal-Teacher Relationship Scale, created originally by Zee, Roorda, and Hanna (2023), was subjected to an adaptation study for Turkish culture. In line with this aim, data for the study were gathered from 389 teachers working in primary and secondary schools located in the districts of Konya, Turkey. The Principal-Teacher Relationship Scale (PTRS) was used as the data collection tool in the study. The PTRS is a two-dimensional scale consisting of a total of 10 items. As part of the adaptation of the PTRS into Turkish, permission for scale usage was obtained initially. Subsequently, the process of establishing the Turkish language equivalence of the PTRS was initiated. After completing this process, data for the study were collected and analyses were conducted. Within the framework of data analysis, the validity and reliability of the PTRS were tested. Confirmatory factor analysis was used to evaluate the validity of the PTRS's two-dimensional structure. The reliability of the PTRS was evaluated using Cronbach's alpha coefficient. Based on the validity and reliability analyses, it was concluded that PTRS is suitable for use in Turkish culture and is a valid and reliable data collection tool.

A comprehensive data analytics framework for risk management in photovoltaic system design projects

Risk management is an essential step of any project, which leads to a reduction in the cost of sudden losses. In the meantime, given the limited time available to make risk management decisions and the large volume of risks that cannot be considered, they should be prioritized with a comprehensive approach. In this paper, in addition to identifying the risks of the project life cycle, the aim is to use a hybrid multi-criteria decision-making (MCDM) method under fuzzy uncertainty for risk management in any photovoltaic system design (PSD) project. First, by reviewing the background, the risk factors of PSD projects are identified according to the stages of the project life cycle and introduced in the form of 2 factors in 4 categories. Next, these factors have been prioritized with a hybrid MCDM approach. These risk factors are prioritized with the cloud model (CM) and fuzzy TOPSIS method (CM-FTOPISIS). The results obtained from fuzzy TOPSIS integrate CM evaluation, which can reduce the uncertainty of the results. The research results show that incorrect project design, human resources risks, negotiation and contract signing risks, previous data risks, consumption audit risks, and equipment risks are the most critical risks in any PSD project.

The international clinical trials registry platform (ICTRP): data integrity and the trends in clinical trials, diseases, and drugs.

Introduction: Clinical trials are the gold standard for testing new therapies. Databases like ClinicalTrials.gov provide access to trial information, mainly covering the US and Europe. In 2006, WHO introduced the global ICTRP, aggregating data from ClinicalTrials.gov and 17 other national registers, making it the largest clinical trial platform by June 2019. This study conducts a comprehensive global analysis of the ICTRP database and provides framework for large-scale data analysis, data preparation, curation, and filtering. Materials and methods: The trends in 689,793 records from the ICTRP database (covering trials registered from 1990 to 2020) were analyzed. Records were adjusted for duplicates and mapping of agents to drug classes was performed. Several databases, including DrugBank, MESH, and the NIH Drug Information Portal were used to investigate trends in agent classes. Results: Our novel approach unveiled that 0.5% of the trials we identified were hidden duplicates, primarily originating from the EUCTR database, which accounted for 82.9% of these duplicates. However, the overall number of hidden duplicates within the ICTRP seems to be decreasing. In total, 689 793 trials (478 345 interventional) were registered in the ICTRP between 1990 and 2020, surpassing the count of trials in ClinicalTrials.gov (362 500 trials by the end of 2020). We identified 4 865 unique agents in trials with DrugBank, whereas 2 633 agents were identified with NIH Drug Information Portal data. After the ClinicalTrials.gov, EUCTR had the most trials in the ICTRP, followed by CTRI, IRCT, CHiCTR, and ISRCTN. CHiCTR displayed a significant surge in trial registration around 2015, while CTRI experienced rapid growth starting in 2016. Conclusion: This study highlights both the strengths and weaknesses of using the ICTRP as a data source for analyzing trends in clinical trials, and emphasizes the value of utilizing multiple registries for a comprehensive analysis.

Stitching Together My Anishinaabe Research Paradigm: An Approach to Storytelling With Algonquin Ikwewag (Women) and Gender-Diverse People in Mattawa and North Bay, Ontario

This article reviews an Anishinaabe research paradigm that structures a storytelling project with Algonquin Anishinaabekwewag and gender-diverse people from the Mattawa and North Bay area in Ontario, Canada. This Anishinaabe research paradigm contains an ontology, epistemology, methodology, and axiology that are informed by Anishinaabe worldviews and values, participants’ stories, as well as the knowledges and experiences the first author brings to the project as a mixed-ancestry Algonquin Anishinaabekwe. With a ribbon skirt framework of data analysis, analytical approaches from multiple knowledge systems are stitched together to form complex and cohesive stories. The methodology is guided by principles of ownership, control, access, and possession, the Tri-Council Policy Statement: Ethical Conduct of Research Involving Humans (Chapter 9), as well as Anishinaabe relational accountability which honours relationships with all of Creation. Axiology is further informed by Mino-Bimaadiziwin and Anishinaabe Original Instructions, and is expressed through Anishinaabe jiimaan (canoe) teachings and values of respect, relevance, reciprocity, responsibility, and reverence. Making space for Anishinaabe research paradigms that prioritize Anishinaabe ways of knowing and living is a powerful way to decolonize the research process and affirm the continuity of Anishinaabe lifeways.

FAIR and Scalable Education

The Galaxy Project [1][2] is a widely-used open-source platform for data-driven research that offers an extensive suite of tools and services for analyzing and visualizing large-scale data. Besides Galaxy as data analysis framework, the Galaxy Training Network (GTN; [3][4]) provides a huge collection of training material for researchers, developers and trainers. Among the various features offered by the Galaxy platform, the GTN and the Training Infrastructure as a Service (TiaaS; [5]) stands out as an innovative solution for delivering scalable and flexible training to researchers and educators. TIaaS is built on top of the Galaxy platforms which enable users to request dedicated compute resources for training purposes. This infrastructure, in combination with GTN, provides an excellent solution for delivering high-quality training to researchers and educators around the world, regardless of their geographical location or hardware limitations. One of the key benefits of the GTN is its ability to enable users to create and share training materials for a variety of data analysis workflows, including e.g. genomics, proteomics, and metabolomics but also for imaging and climate data. These training materials can be delivered in various formats, including interactive tutorials, videos, and webinars, and can be accessed from any location with an internet connection. Moreover, the platform offers a wide range of community-contributed training materials that cover a vast array of topics in biomedical research and beyond, making it an invaluable resource for researchers and educators. To use TIaaS, instructors simply need to register their training event in a simple form to get access to the dedicated compute resources.

Leveraging cell-cell similarity for high-performance spatial and temporal cellular mappings from gene expression data

Single-cell trajectory mapping and spatial reconstruction are two important developments in life science and provide a unique means to decode heterogeneous tissue formation, cellular dynamics, and tissue developmental processes. The success of these techniques depends critically on the performance of analytical tools used for high-dimensional (HD) gene expression data processing. Existing methods discern the patterns of the data without explicitly considering the underlying biological characteristics of the system, often leading to suboptimal solutions. Here, we present a cell-cell similarity-driven framework of genomic data analysis for high-fidelity spatial and temporal cellular mappings. The approach exploits the similarity features of the cells to discover discriminative patterns of the data. We show that for a wide variety of datasets, the proposed approach drastically improves the accuracies of spatial and temporal mapping analyses compared with state-of-the-art techniques.

BIG DATA PROCESSING WITH PYTHON IN SOCIAL NETWORKS

Social networks generate a huge amount of data about the behavior, interests, preferences and connections of users, which can be used for various purposes such as marketing, advertising, analytics, research, etc. However, these data have a large volume, variety and update rate, and require special methods and tools for their analysis and processing. Python is a high-level programming language that has many advantages for working with big data, such as simplicity and flexibility of syntax, a rich set of libraries and frameworks for data processing and analysis, support for multithreading and distributed computing, open source code, and a large community of developers and users. The article shows how to use Python to parse and collect data from various sources, to clean and pre-process data, such as removing duplicates, gaps, outliers and noise, normalizing and standardizing data, encoding categorical features, etc., for analysis and data visualization, such as building statistical characteristics, graphs, charts and maps to study the distribution, correlation and dependence of data, for machine learning and data mining. Keywords: Python, big data, social networks, data parsing, data analysis, machine learning.

The Effectiveness of Participatory Governance in Budgetary Monitoring and Evaluation in an Information Age: A Case of Fetakgomo-Tubatse Local Municipality

Deepening participatory governance in budgetary monitoring and evaluation is critical to ensuring that local residents' rights to meaningful engagement in municipal governance are realised. It symbolises a paradigm shift in the relationship between municipalities and local residents, particularly in terms of promoting transparency and accountability in the use of municipal financial resources to achieve positive service delivery outcomes. However, proffering opportunities for residents to meaningfully engage in budgetary monitoring and evaluation processes is often met with mixed reactions within municipalities’ governance structures. This paper evaluates the effectiveness of participatory governance in budgetary monitoring and evaluation in the information age in the case of Fetakgomo-Tubatse local municipality. The objectives were to assess participants' understanding of the importance of participatory governance implementation in budgetary monitoring and evaluation, the scope and purpose of the implementation process, associated benefits and barriers, the efficacy of information dissemination and accessibility relating to the outcomes and impact of the process, and the degree of ICTs use towards enhancing timely dissemination and accessibility of the information. The study used a mixed method case study research design. Face-to-face semi-structured interviews and a self-administered semi-structured questionnaire were used to collect primary research data from a purposive and snowball sample of municipal officials (N=8) and ward committee members (N=10). The data was analysed using a seven-stage mixed method data analysis framework developed by Onwuegbuzie and Teddlie (2003). The paper provides the main findings that led to the conclusion that participatory governance in budgetary monitoring and evaluation is implemented ineffectively in the Fetakgomo-Tubatse local municipality; recommendations on how to strengthen its effectiveness and avenues for future research are provided.

An Analytical Framework for Data Collection and Analysis in IP Network

Abstract The primary focus of the study is to investigate the importance of data collection and analysis in IP networks for detecting, identifying, and responding to potential cyber attacks. It examines the use of the Suricata system’s integration with the process of sending detected anomalies to the non-relational Elasticsearch database. The research also looks into the use of Data Lakes, which are centralized storage systems capable of securely storing and analyzing massive amounts of IP traffic data in their native format. An experimental environment is presented, featuring the Elasticsearch database, REDIS cache, and Suricata IDS tool, to conduct experiments. The findings show that combining Suricata with Elasticsearch and Redis cache results a suitable combination, leading to enhanced performance and increased analysis accuracy. In conclusion, by leveraging the strengths of these technologies, it is possible to establish a robust and efficient infrastructure that effectively assists network administrators to safeguard networks against various cyber threats in the network environment.

The framework of parametric and nonparametric operational data analytics

This paper introduces the general philosophy of the Operational Data Analytics (ODA) framework for data‐based decision modeling. The fundamental development of this framework lies in establishing the direct mapping from data to decision by identifying the appropriate class of operational statistics. The efficient decision making relies on a careful balance between data integration and decision validation. Through a canonical decision making problem under uncertainty, we show that the existing approaches (including statistical estimation and then optimization, retrospective optimization, sample average approximation, regularization, robust optimization, and robust satisficing) can all be unified through the lens of the ODA formulation. To make the key concepts accessible, we demonstrate, using a simple running example, how some of the existing approaches may become equivalent under the ODA framework, and how the ODA solution can improve the decision efficiency, especially in the small sample regime.