Python Data Research Articles

APPLICATION OF DATA ANALYSIS TOOLS IN PYTHON TO ASSESS THE STANDARD OF LIVING OF THE POPULATION: THE CASE OF KAZAKHSTAN

The article discusses the application of Python data analysis tools to assess the standard of living of the Kazakhstani population based on statistical data for the period 1991–2021. The main focus is using modern methods and technologies, such as linear regression, correlation analysis, and time series processing. The study uses Python libraries (Pandas, NumPy, Matplotlib, Seaborn, Scikit-learn) and the PostgreSQL relational database to store and process large amounts of information. The analysis revealed key trends, including a steady increase in per capita income and a decrease in unemployment. The constructed regression models showed that population incomes are closely related to the subsistence minimum and minimum wage. Forecasting for the coming years indicates a continued increase in living standards while maintaining current economic trends. The results confirm that information technology can improve calculation accuracy, optimize data analysis, and present the results in an easy-to-interpret form. This work can be helpful for government agencies, analysts, and researchers involved in assessing socio-economic indicators. This article highlights the importance of information technologies in data analysis and forecasting and demonstrates their potential to solve urgent social policy problems. Keywords: standard of living, data analysis, Python, regression analysis, forecasting, correlation analysis, time series.

Enhancing Jakarta Faces Web App with AI Data-Driven Python Data Analysis and Visualization

Python is widely used in artificial intelligence (AI) and machine learning (ML) because of its flexibility, adaptability, rich libraries, active community, and broad environment, which makes it a popular choice for AI development. Python compatibility has already been examined with Java using TCP socket programming on both non-graphical and graphical user interfaces, which is highly essential to implement in the Jakarta Faces web application to grab potential competitive advantages. Python data analysis library modules such as numpy, pandas, and scipy, as well as visualization library modules such as Matplotlib and Seaborn, and machine-learning module Scikit-learn, are intended to be integrated into the Jakarta Faces web application. The research method uses similar TCP socket programming for the enhancement process, which allows instruction and data exchange between Python and Jakarta Faces web applications. The outcome of the findings emphasizes the significance of modernizing data science and machine learning (ML) workflows for Jakarta Faces web developers to take advantage of Python modules without using any third-party libraries. Moreover, this research provides a well-defined research design for an execution model, incorporating practical implementation procedures and highlighting the results of the innovative fusion of AI from Python into Jakarta Faces.

Exploration and Construction of Professional Curriculum System in the Field of Intelligent Manufacturing Based on Data Analysis

Intelligent manufacturing is an important means of developing strategic emerging industries and accelerating the formation of a modern industrial system. Currently, many universities are upgrading their related majors or offering new majors such as Intelligent Manufacturing Engineering to provide talent for the field of intelligent manufacturing. Due to the lack of integrated resources between industry and education in the field of intelligent manufacturing, it is difficult for universities to strengthen the correlation between the professional curriculum system and the industry when opening new majors or upgrading majors. With the integration of new generation information technology and manufacturing technology, the country plans to transform traditional enterprises into digital, information-based, and intelligent enterprises, and cultivate innovative, technical, and skilled talents that adapt to the development of intelligent manufacturing. It has become a new development strategy given to vocational colleges by the times. This article first obtains information on professional courses related to intelligent manufacturing and artificial intelligence offered in the region, and then uses Python data analysis technology to conduct statistical analysis on the collected professional courses. The analysis results provide reference and inspiration for universities to offer majors related to intelligent manufacturing.

Processing large amounts of data in Python for the purposes of applied socio-economic research: advantages and current issues

The article is devoted to the study of the features of processing large amounts of data using the Python programming language. Unlike tabular processors or finished software products, programming languages offer the user a flexible toolkit for the implementation of tasks. At the same time, this creates certain risks associated with the effectiveness of using appropriate tools and optimising the operation of the programme. The purpose of the article is to study the features of processing large amounts of data in Python on the examples of immediate research tasks. The relevance of the topic and purpose of the article is due to the existing scientific gap related to a comprehensive consideration of the technical aspects of the use of programming languages and associated tools for socio-economic research. Thus, many authors who use programming languages in their works rarely provide information regarding the advantages of certain algorithms or approaches. Within the framework of the article, the author examines the procedures and algorithm of processing a large array of data on the example of specific research tasks. The conclusions are drawn about the features and advantages of Python when working with large amounts of data as well as about the prospects for the development of the relevant scientific topics.

Application of big data technology in the medical field

This project aims to construct a knowledge graph system applied to the field of traditional Chinese medicine (TCM) by extracting entities (such as drugs, diseases, etc.) and their relationships from TCM medical case data and storing them in a Neo4j database. The project process includes data reading, entity recognition and extraction, data formatting, and data import into the database. The project not only improved the individual's proficiency in Python data processing techniques (including regular expressions and JSON parsing) but also enhanced their skills in knowledge graph construction and database operations. In the future, there is a desire to further improve technical capabilities, explore more cutting-edge technologies in the TCM field, and promote project progress through collaboration, contributing to the modernization of TCM and intelligent healthcare services.

Using a Combination of Novel Research Tools to Understand Social Interaction in the Drosophila melanogaster Model for Fragile X Syndrome.

Fragile X syndrome (FXS), the most common monogenic cause of inherited intellectual disability and autism spectrum disorder, is caused by a full mutation (>200 CGG repeats) in the Fragile X Messenger Ribonucleoprotein 1 (FMR1) gene. Individuals with FXS experience various challenges related to social interaction (SI). Animal models, such as the Drosophila melanogaster model for FXS where the only ortholog of human FMR1 (dFMR1) is mutated, have played a crucial role in the understanding of FXS. The aim of this study was to investigate SI in the dFMR1B55 mutants (the groups of flies of both sexes simultaneously) using the novel Drosophila Shallow Chamber and a Python data processing pipeline based on social network analysis (SNA). In comparison with wild-type flies (w1118), SNA analysis in dFMR1B55 mutants revealed hypoactivity, fewer connections in their networks, longer interaction duration, a lower ability to transmit information efficiently, fewer alternative pathways for information transmission, a higher variability in the number of interactions they achieved, and flies tended to stay near the boundaries of the testing chamber. These observed alterations indicate the presence of characteristic strain-dependent social networks in dFMR1B55 flies, commonly referred to as the group phenotype. Finally, combining novel research tools is a valuable method for SI research in fruit flies.

Persist: Persistent and Reusable Interactions in Computational Notebooks

AbstractComputational notebooks, such as Jupyter, support rich data visualization. However, even when visualizations in notebooks are interactive, they are a dead end: Interactive data manipulations, such as selections, applying labels, filters, categorizations, or fixes to column or cell values, could be efficiently applied in interactive visual components, but interactive components typically cannot manipulate Python data structures. Furthermore, actions performed in interactive plots are lost as soon as the cell is re‐run, prohibiting reusability and reproducibility. To remedy this problem, we introduce Persist, a family of techniques to (a) capture interaction provenance, enabling the persistence of interactions, and (b) map interactions to data manipulations that can be applied to dataframes. We implement our approach as a JupyterLab extension that supports tracking interactions in Vega‐Altair plots and in a data table view. Persist can re‐execute interaction provenance when a notebook or a cell is re‐executed, enabling reproducibility and re‐use. We evaluate Persist in a user study targeting data manipulations with 11 participants skilled in Python and Pandas, comparing it to traditional code‐based approaches. Participants were consistently faster and were able to correctly complete more tasks with Persist.

A Comparative Study on the Market Competition Strategies of Tencent and Alibaba Based on Python Data Analysis

A Comparative Study on the Market Competition Strategies of Tencent and Alibaba Based on Python Data Analysis

Python Data Analysis and Visualization in Java GUI Applications Through TCP Socket Programming

Python is popular in artificial intelligence (AI) and machine learning (ML) due to its versatility, adaptability, rich libraries, and active community. The existing Python interoperability in Java was investigated using socket programming on a non-graphical user interface (GUI). Python's data analysis library modules such as numpy, pandas, and scipy, together with visualization library modules such as Matplotlib and Seaborn, and Scikit-learn for machine-learning, aim to integrate into Java graphical user interface (GUI) applications such as Java applets, Java Swing, and Java FX. The substantial method used in the integration process is TCP socket programming, which makes instruction and data transfers to provide interoperability between Python and Java GUIs. This empirical research integrates Python data analysis and visualization graphs into Java applications and does not require any additional libraries or third-party libraries. The experimentation confirmed the advantages and challenges of this integration with a concrete solution. The intended audience for this research extends to software developers, data analysts, and scientists, recognizing Python's broad applicability to artificial intelligence (AI) and machine learning (ML). The integration of data analysis and visualization and machine-learning functionalities within the Java GUI. It emphasizes the self-sufficiency of the integration process and suggests future research directions, including comparative analysis with Java's native capabilities, interactive data visualization using libraries like Altair, Bokeh, Plotly, and Pygal, performance and security considerations, and no-code and low-code implementations.

An easy‐to‐follow handbook for electroencephalogram data analysis with Python

AbstractThis easy‐to‐follow handbook offers a straightforward guide to electroencephalogram (EEG) analysis using Python, aimed at all EEG researchers in cognitive neuroscience and related fields. It spans from single‐subject data preprocessing to advanced multisubject analyses. This handbook contains four chapters: Preprocessing Single‐Subject Data, Basic Python Data Operations, Multiple‐Subject Analysis, and Advanced EEG Analysis. The Preprocessing Single‐Subject Data chapter provides a standardized procedure for single‐subject EEG data preprocessing, primarily using the MNE‐Python package. The Basic Python Data Operations chapter introduces essential Python operations for EEG data handling, including data reading, storage, and statistical analysis. The Multiple‐Subject Analysis chapter guides readers on performing event‐related potential and time‐frequency analyses and visualizing outcomes through examples from a face perception task dataset. The Advanced EEG Analysis chapter explores three advanced analysis methodologies, Classification‐based decoding, Representational Similarity Analysis, and Inverted Encoding Model, through practical examples from a visual working memory task dataset using NeuroRA and other powerful packages. We designed our handbook for easy comprehension to be an essential tool for anyone delving into EEG data analysis with Python (GitHub website: https://github.com/ZitongLu1996/Python‐EEG‐Handbook; For Chinese version: https://github.com/ZitongLu1996/Python‐EEG‐Handbook‐CN).

Pairtools: From sequencing data to chromosome contacts.

The field of 3D genome organization produces large amounts of sequencing data from Hi-C and a rapidly-expanding set of other chromosome conformation protocols (3C+). Massive and heterogeneous 3C+ data require high-performance and flexible processing of sequenced reads into contact pairs. To meet these challenges, we present pairtools-a flexible suite of tools for contact extraction from sequencing data. Pairtools provides modular command-line interface (CLI) tools that can be flexibly chained into data processing pipelines. The core operations provided by pairtools are parsing of.sam alignments into Hi-C pairs, sorting and removal of PCR duplicates. In addition, pairtools provides auxiliary tools for building feature-rich 3C+ pipelines, including contact pair manipulation, filtration, and quality control. Benchmarking pairtools against popular 3C+ data pipelines shows advantages of pairtools for high-performance and flexible 3C+ analysis. Finally, pairtools provides protocol-specific tools for restriction-based protocols, haplotype-resolved contacts, and single-cell Hi-C. The combination of CLI tools and tight integration with Python data analysis libraries makes pairtools a versatile foundation for a broad range of 3C+ pipelines.

Reform and Exploration of Teaching Modes in Data Processing Courses for Humanities and Social Sciences Majors

This article explores innovative teaching methods such as personalized instruction, fostering students' self-learning abilities and creative thinking, integrating multimedia technology for blended learning both online and offline, incorporating research into teaching, diversifying evaluation methods, and enhancing teachers' self-efficacy for improving teaching effectiveness. Taking the course "Python Data Processing and Analysis" as a case study, the article applies these approaches in practical teaching. The research findings indicate a significant improvement in teaching effectiveness, with noticeable enhancements in student engagement and performance.

Assessing Inequality in Urban Green Spaces with Consideration for Physical Activity Promotion: Utilizing Spatial Analysis Techniques Supported by Multisource Data

Urban green spaces (UGSs) play a significant role in promoting public health by facilitating outdoor activities, but issues of spatial and socioeconomic inequality within UGSs have drawn increasing attention. However, current methods for assessing UGS inequality still face challenges such as data acquisition difficulties and low identification accuracy. Taking Harbin as a case study, this research employs various advanced technologies, including Python data scraping, drone imagery collection, and Amap API, to gather a diverse range of data on UGSs, including photos, high-resolution images, and AOI boundaries. Firstly, elements related to physical activity within UGSs are integrated into a supply adjustment index (SAI), based on which UGSs are classified into three categories. Then, a supply–demand improved two-step floating catchment area (SD2SFCA) method is employed to more accurately measure the accessibility of these three types of UGSs. Finally, using multiple linear regression analysis and Mann–Whitney U tests, socioeconomic inequalities in UGS accessibility are explored. The results indicate that (1) significant differentiation exists in the types of UGS services available in various urban areas, with a severe lack of small-scale, low-supply UGSs; (2) accessibility of all types of UGSs is significantly positively associated with housing prices, with higher-priced areas demonstrating notably higher accessibility compared to lower-priced ones; (3) children may be at a disadvantage in accessing UGSs with medium-supply levels. Future planning efforts need to enhance attention to vulnerable groups. This study underscores the importance of considering different types of UGSs in inequality assessments and proposes a method that could serve as a valuable tool for accurately assessing UGS inequality.

DEIMoS GUI: An Open-Source User Interface for a High-Dimensional Mass Spectrometry Data Processing Tool.

We report here the creation of a graphical user interface (GUI) for the Data Extraction for Integrated Multidimensional Spectrometry (DEIMoS) tool. DEIMoS is a Python package that processes data from high-dimensional mass spectrometry measurements. It is divided into several modules, each representing a data processing step such as peak detection, alignment, and tandem mass spectra extraction and deconvolution. The inputs for and outputs from DEIMoS can include millions of N-dimensional data points, which can be challenging to visualize in a way that is interactive, informative, and responsive. Here, we used the HoloViz Python data visualization stack, including DataShader and Param, to create an interactive visualization of the mass spectrometry data. We believe the GUI will increase the accessibility of DEIMoS and that the visualization methods could be useful for other open-source mass spectrometry tools.

Pycallingcards: an integrated environment for visualizing, analyzing, and interpreting Calling Cards data.

Unraveling the transcriptional programs that control how cells divide, differentiate, and respond to their environments requires a precise understanding of transcription factors' (TFs) DNA-binding activities. Calling cards (CC) technology uses transposons to capture transient TF binding events at one instant in time and then read them out at a later time. This methodology can also be used to simultaneously measure TF binding and mRNA expression from single-cell CC and to record and integrate TF binding events across time in any cell type of interest without the need for purification. Despite these advantages, there has been a lack of dedicated bioinformatics tools for the detailed analysis of CC data. We introduce Pycallingcards, a comprehensive Python module specifically designed for the analysis of single-cell and bulk CC data across multiple species. Pycallingcards introduces two innovative peak callers, CCcaller and MACCs, enhancing the accuracy and speed of pinpointing TF binding sites from CC data. Pycallingcards offers a fully integrated environment for data visualization, motif finding, and comparative analysis with RNA-seq and ChIP-seq datasets. To illustrate its practical application, we have reanalyzed previously published mouse cortex and glioblastoma datasets. This analysis revealed novel cell-type-specific binding sites and potential sex-linked TF regulators, furthering our understanding of TF binding and gene expression relationships. Thus, Pycallingcards, with its user-friendly design and seamless interface with the Python data science ecosystem, stands as a critical tool for advancing the analysis of TF functions via CC data. Pycallingcards can be accessed on the GitHub repository: https://github.com/The-Mitra-Lab/pycallingcards.

Bioframe: operations on genomic intervals in Pandas dataframes.

Genomic intervals are one of the most prevalent data structures in computational genome biology, and used to represent features ranging from genes, to DNA binding sites, to disease variants. Operations on genomic intervals provide a language for asking questions about relationships between features. While there are excellent interval arithmetic tools for the command line, they are not smoothly integrated into Python, one of the most popular general-purpose computational and visualization environments. Bioframe is a library to enable flexible and performant operations on genomic interval dataframes in Python. Bioframe extends the Python data science stack to use cases for computational genome biology by building directly on top of two of the most commonly-used Python libraries, NumPy and Pandas. The bioframe API enables flexible name and column orders, and decouples operations from data formats to avoid unnecessary conversions, a common scourge for bioinformaticians. Bioframe achieves these goals while maintaining high performance and a rich set of features. Bioframe is open-source under MIT license, cross-platform, and can be installed from the Python Package Index. The source code is maintained by Open2C on GitHub at https://github.com/open2c/bioframe.

Efficient Feature Selection for Classification of Immunotherapy and Medical Treatments Utilising Random Forest and Decision Trees

Immunotherapy is an important topic in healthcare as it affects patients’ treatments for breast cancer, diabetes, and immunotherapy. However, immunotherapy for warts is less representative because of the lack of data. Machine learning is frequently utilised for treatment diagnosis by converting raw immunotherapy data into useful insights. Efficient classification of immunotherapy treatments is crucial for a productive diagnosis. This study considers immunotherapy with a data-driven and ’less is more perspective’. Despite using a portion of the available imbalance and complex data, the process of diagnosis of immunotherapy treatment is made reasonably precise by considering the parameters of accuracy, sensitivity, and specificity. The contribution of this study is focused on ”more is less” feature selection, which states that approximately 80% of the effects or results of a system are caused by 20% of the inputs. The features that contribute most to the classification of immunotherapy treatments are prioritised. This study proposes the implementation of Random Forest and Decision Trees for the classification of immunotherapy treatments. The relevant experimental medical data are explored as a case study. The experiments are conducted using Weka and Python data analysis tools, performing data preprocessing, class balancing, and feature selection. Random Forest performed better than the Decision Trees. By Applying Random Forest and utilising only one feature (time) as an input variable, a classification accuracy of 88.88%, sensitivity of 95.45%, and specificity of 60% are attained. By using 12.5% of the dataset, when implementing Random Forest together with ordinary feature selection, the diagnosis of immunotherapy treatments is become more efficient, despite using a portion of data features reasonable results are obtained.

An Algorithm for Exploratory Analysis and Normalization of Big Data with Pandas

The digitization of business processes and the extraction of answers to user requests for big data are modern problems that are of great interest to scientists and researchers. The data generated so far, located in various corpora, is much more than can be analyzed. Therefore, they are collected, identified, cleaned and normalized to be used most adequately. Segmentation, assumptions and hypotheses contribute to the degree of satisfaction with the returned result. The research proposed a general method for collecting, cleaning and normalizing data from various sources, structurally modelling it into appropriate models, then testing hypotheses and analyzing the obtained results to conclude large academic data that will benefit the business in making management decisions. This is possible with the means of computational linguistics and with the help of Python data manipulation libraries.

Forecasting stock prices of fintech companies of India using random forest with high-frequency data

The fintech segment is currently one of the most rapidly growing industries, attracting numerous investors who anticipate substantial returns in the future. Notably, not only individual retail investors but also mutual fund agencies are actively engaged in predicting stock prices within this sector to maximize their trading gains. The purpose of the study is to formulate stock forecasting models for top three Fintech Companies of India i.e., Policy Bazar, One 97 Communications Paytm Ltd., and Niyogin Ltd. Using Random Forest model with high-frequency data in Python. The literature review section also proves that this study is a novel piece of work as none of the existing research study focused on predicting stock prices of Fintech Companies of India using Random Forest model. The data is extracted from www.moneycontrol.com and www.kotaksecurities.com, for the period from 1st October, 2022–30 th September, 2023. The study deals about 293,280 data points i.e., 3 companies @ 97,760 each. It has been found that the forecasting model of random forest provides very successful results for prediction as the co-efficient of determination of all the selected companies is more than 95%.

Predicting Student Performance: Analyzing Socio-economic and Personal Factors

The purpose of this study is to predict student performance in Mathematics, Reading, and Writing exams by analyzing a comprehensive dataset containing student profiles and exam scores. Python's data processing and machine learning libraries are utilized for data analysis and modeling. The dataset is explored, followed by data preprocessing that includes feature encoding and standardization. Support Vector Regression (SVR) is employed as a multi-target regression model to train and evaluate the model for predicting student exam scores. Feature importance analysis and visualization of prediction errors are performed to gain deeper insights into student performance. The results demonstrate significant correlations between factors such as gender, lunch type, ethnic group, and parent education level with exam scores. The prediction model performs well on the test dataset and effectively predicts student exam scores in different subjects. The findings of this study provide valuable insights for educational decision-making and promote personalized and effective education.