Real Datasets Research Articles

Cervical Cancer Prediction Based on Imbalanced Data Using Machine Learning Algorithms with a Variety of Sampling Methods

Cervical cancer affects a large portion of the female population, making the prediction of this disease using Machine Learning (ML) of utmost importance. ML algorithms can be integrated into complex, intelligent, agent-based systems that can offer decision support to resident medical doctors or even experienced medical doctors. For instance, an experienced medical doctor may diagnose a case but need expert support that related to another medical specialty. Data imbalance is frequent in healthcare data and has a negative influence on predictions made using ML algorithms. Cancer data, in general, and cervical cancer data, in particular, are frequently imbalanced. For this study, we chose a messy, real-life cervical cancer dataset available in the Kaggle repository that includes large amounts of missing and noisy values. To identify the best imbalanced technique for this medical dataset, the performances of eleven important resampling methods are compared, combined with the following state-of-the-art ML models that are frequently applied in predictive healtchare research: K-Nearest Neighbors (KNN) (with k values of 2 and 3), binary Logistic Regression (bLR), and Random Forest (RF). The studied resampling methods include seven undersampling methods and four oversampling methods. For this dataset, the imbalance ratio was 12.73, with a 95% confidence interval ranging from 9.23% to 16.22%. The obtained results show that resampling methods help improve the classification ability of prediction models applied to cervical cancer data. The applied oversampling techniques for handling imbalanced data generally outperformed the undersampling methods. The average balanced accuracy for oversampling was 77.44%, compared to 62.28% for undersampling. When detecting the minority class, oversampling achieved an average score of 60.80%, while undersampling scored 41.36%. The logistic regression classifier had the greatest impact on balanced techniques, while random forest achieved promising performance, even before applying balancing techniques. Initially, KNN2 outperformed KNN3 across all metrics, including balanced accuracy, for which KNN2 achieved 53.57%, compared to 52.71% for KNN3. However, after applying oversampling techniques, KNN3 significantly improved its balanced accuracy to 73.78%, while that of KNN2 increased to 63.89%. Additionally, KNN3 outperformed KNN2 in minority class performance, scoring 55.72% compared to KNN2’s 33.93%.

Characteristics of the GLLE Family of Lifetime Distributions, and Applications

ABSTRACT. A Generalized Log-Logistic (GLL) family of lifetime distributions is one in which any pair of distributions are related through a GLL transformation, for some (non-negative) value of the transformation parameter (the odds function of the second distribution is the power of the odds function of the first distribution). We consider GLL families generated from an exponential distribution. We derive some useful characteristics of these distributions; moreover, we discuss the hazard rates and Kullback-Leibler divergence, then illustrate the usefulness of this distribution family by fitting real data sets.

Two sample test for covariance matrices in ultra-high dimension

In this paper, we propose a new test for testing the equality of two population covariance matrices in the ultra-high dimensional setting that the dimension is much larger than the sizes of both of the two samples. Our proposed methodology relies on a data splitting procedure and a comparison of a set of well selected eigenvalues of the sample covariance matrices on the split data sets. Compared to the existing methods, our methodology is adaptive in the sense that (i). it does not require specific assumption (e.g., comparable or balancing, etc.) on the sizes of two samples; (ii). it does not need quantitative or structural assumptions of the population covariance matrices; (iii). it does not need the parametric distributions or the detailed knowledge of the moments of the two populations. Theoretically, we establish the asymptotic distributions of the statistics used in our method and conduct the power analysis. We justify that our method is powerful under weak alternatives. We conduct extensive numerical simulations and show that our method significantly outperforms the existing ones both in terms of size and power. Analysis of two real data sets is also carried out to demonstrate the usefulness and superior performance of our proposed methodology. An R package UHDtst is developed for easy implementation of our proposed methodology.

Simulating the Cellular Context in Synthetic Datasets for Cryo-Electron Tomography.

Cryo-electron tomography (cryo-ET) allows to visualize the cellular context at macromolecular level. To date, the impossibility of obtaining a reliable ground truth is limiting the application of deep learning-based image processing algorithms in this field. As a consequence, there is a growing demand of realistic synthetic datasets for training deep learning algorithms. In addition, besides assisting the acquisition and interpretation of experimental data, synthetic tomograms are used as reference models for cellular organization analysis from cellular tomograms. Current simulators in cryo-ET focus on reproducing distortions from image acquisition and tomogram reconstruction, however, they can not generate many of the low order features present in cellular tomograms. Here we propose several geometric and organization models to simulate low order cellular structures imaged by cryo-ET. Specifically, clusters of any known cytosolic or membrane-bound macromolecules, membranes with different geometries as well as different filamentous structures such as microtubules or actin-like networks. Moreover, we use parametrizable stochastic models to generate a high diversity of geometries and organizations to simulate representative and generalized datasets, including very crowded environments like those observed in native cells. These models have been implemented in a multiplatform open-source Python package, including scripts to generate cryo-tomograms with adjustable sizes and resolutions. In addition, these scripts provide also distortion-free density maps besides the ground truth in different file formats for efficient access and advanced visualization. We show that such a realistic synthetic dataset can be readily used to train generalizable deep learning algorithms.

A New Mixture of Two Components of Exponentiated Family with Applications to Real Life Data Sets

A New Mixture of Two Components of Exponentiated Family with Applications to Real Life Data Sets

A New Method of Generating Truncated Bivariate Families of Distributions

The modeling of complex data has attracted several researchers for the quest of generating new probability distributions. The joint modeling of two variables asks for some additional complexities as a bivariate distribution is needed. The field of research in developing bivariate families of distributions is somewhat new. In certain situations, the domain of data is restricted and some truncated distribution is required. Several univariate truncated families of distributions are available for modeling of a single variable but the bivariate truncated families of distributions has not been studied and in this paper, we have proposed a new bivariate truncated families of distributions. A specific sub-family has been proposed by using the bivariate Burr as a base-line distribution, resulting in a bivariate truncated Burr family of distributions. Some important statistical properties of the proposed family has been studied, which include the marginal and conditional distributions, bivariate reliability, and bivariate hazard rate functions. The maximum likelihood estimation for the parameters of the family is also carried out. The proposed bivariate truncated Burr family of distributions is studied for the Burr baseline distributions, giving rise to the bivariate truncated Burr-Burr distribution. The new bivariate truncated Burr-Burr distribution is explored in detail and several statistical properties of the new distribution are studied, which include the marginal and conditional distributions, product, ratio, and conditional moments. The maximum likelihood estimation for the parameters of the proposed distribution is done. The proposed bivariate truncated Burr-Burr distribution is used to model some real data sets. It is found that the proposed distribution performs better than the other distributions considered in this study.

Seismic Random Noise Attenuation Using DARE U-Net

Seismic data processing plays a pivotal role in extracting valuable subsurface information for various geophysical applications. However, seismic records often suffer from inherent random noise, which obscures meaningful geological features and reduces the reliability of interpretations. In recent years, deep learning methodologies have shown promising results in performing noise attenuation tasks on seismic data. In this research, we propose modifications to the standard U-Net structure by integrating dense and residual connections, which serve as the foundation of our approach named the dense and residual (DARE U-Net) network. Dense connections enhance the receptive field and ensure that information from different scales is considered during the denoising process. Our model implements local residual connections between layers within the encoder, which allows earlier layers to directly connect with deep layers. This promotes the flow of information, allowing the network to utilize filtered and unfiltered input. The combined network mechanisms preserve the spatial information loss during the contraction process so that the decoder can locate the features more accurately by retaining the high-resolution features, enabling precise location in seismic image denoising. We evaluate this adapted architecture by applying synthetic and real data sets and calculating the peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). The effectiveness of this method is well noted.

CityLearn v2: energy-flexible, resilient, occupant-centric, and carbon-aware management of grid-interactive communities

As more distributed energy resources become part of the demand-side infrastructure, quantifying their energy flexibility on a community scale is crucial. CityLearn v1 provided an environment for benchmarking control algorithms. However, there is no standardized environment utilizing realistic building-stock datasets for distributed energy resource control benchmarking without co-simulation or third-party frameworks. CityLearn v2 extends CityLearn v1 by providing a stand-alone simulation environment that leverages the End-Use Load Profiles for the U.S. Building Stock dataset to create grid-interactive communities for resilient, multi-agent, and objective control of distributed energy resources with dynamic occupant feedback. While the v1 environment used pre-simulated building thermal loads, the v2 environment uses data-driven thermal dynamics and eliminates the need for co-simulation with building energy performance software. This work details the v2 environment and provides application examples that use reinforcement learning control to manage battery energy storage system, vehicle-to-grid control, and thermal comfort during heat pump power modulation.

Question Answering for Electronic Health Records: Scoping Review of Datasets and Models.

Question answering (QA) systems for patient-related data can assist both clinicians and patients. They can, for example, assist clinicians in decision-making and enable patients to have a better understanding of their medical history. Substantial amounts of patient data are stored in electronic health records (EHRs), making EHR QA an important research area. Because of the differences in data format and modality, this differs greatly from other medical QA tasks that use medical websites or scientific papers to retrieve answers, making it critical to research EHR QA. This study aims to provide a methodological review of existing works on QA for EHRs. The objectives of this study were to identify the existing EHR QA datasets and analyze them, study the state-of-the-art methodologies used in this task, compare the different evaluation metrics used by these state-of-the-art models, and finally elicit the various challenges and the ongoing issues in EHR QA. We searched for articles from January 1, 2005, to September 30, 2023, in 4 digital sources, including Google Scholar, ACL Anthology, ACM Digital Library, and PubMed, to collect relevant publications on EHR QA. Our systematic screening process followed PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. A total of 4111 papers were identified for our study, and after screening based on our inclusion criteria, we obtained 47 papers for further study. The selected studies were then classified into 2 non-mutually exclusive categories depending on their scope: "EHR QA datasets" and "EHR QA models." A systematic screening process obtained 47 papers on EHR QA for final review. Out of the 47 papers, 53% (n=25) were about EHR QA datasets, and 79% (n=37) papers were about EHR QA models. It was observed that QA on EHRs is relatively new and unexplored. Most of the works are fairly recent. In addition, it was observed that emrQA is by far the most popular EHR QA dataset, both in terms of citations and usage in other papers. We have classified the EHR QA datasets based on their modality, and we have inferred that Medical Information Mart for Intensive Care (MIMIC-III) and the National Natural Language Processing Clinical Challenges datasets (ie, n2c2 datasets) are the most popular EHR databases and corpuses used in EHR QA. Furthermore, we identified the different models used in EHR QA along with the evaluation metrics used for these models. EHR QA research faces multiple challenges, such as the limited availability of clinical annotations, concept normalization in EHR QA, and challenges faced in generating realistic EHR QA datasets. There are still many gaps in research that motivate further work. This study will assist future researchers in focusing on areas of EHR QA that have possible future research directions.

Determining the prior mean in Bayesian logistic regression with sparse data: a nonarbitrary approach

Abstract Logistic regression models lead to a bias of the odds ratio (OR) as estimated using the maximum-likelihood (ML) method, when there are few study participants at the binary outcome and factor levels. Although Bayesian methods are frequently applied to reduce this sparse data bias, the specification of the prior distribution for regression coefficients is the most controversial feature. We propose a nonarbitrary and empirical method to determine the prior mean for regression coefficients in Bayesian logistic regression analysis for sparse data. The proposed prior mean is calculated as the difference between the observed log OR and the quasi-expectation of log OR, and is interpreted as a shrinkage statistic of the ML estimate. Further, for easy and fast inference, the proposed method applies to Bayesian logistic regression with the normal prior and the log-F prior via data augmentation. Simulation results indicate that the OR bias based on the proposed method is consistently smaller than that based on the ML method. The OR bias estimated using the proposed method is generally smaller than that based on the mean prior of zero for the regression coefficient. We apply the proposed methods to 2 real data sets.

THE ODD LOMAX TOPP LEONE DISTRIBUTION: PROPERTIES AND APPLICATION

Lifetime distributions are parametric models used in statistical analyses of time-to-event data. Several probability distributions have been very used in literature to model lifetime data sets which have been useful in the analysis of lifetime data, but in most cases, they are not flexible enough to analyze some complex lifetime data in practice. Due to the importance of these lifetime distributions in modeling real lifetime data, there has several modifications and generalization of lifetime distributions, particularly the Lomax distribution to develop more flexible distributions to address the drawback presented by some classical lifetime distributions including the Lomax distribution. Several attempts have been made by researchers to generalize classical lifetime distributions which offer more flexibility in modeling lifetime data. Our interest in this study is to introduce a new extension of Lomax distribution called the “The odd lomax Topp-Leone distribution” which is bounded on a unit interval data such that its flexibility can accommodate increasing, decreasing, right skewed, left skewed, symmetric and u-shaped data sets. An application to a real lifetime data set clearly shows that the proposed extension of the Lomax distribution is a better alternative to some existing distributions bounded on a unit interval.

Optimal class of memory type imputation methods for time-based surveys using EWMA statistics

Time-based surveys often experience missing data due to several reasons, like non-response or data collection limitations. Imputation methods play an essential role in incorporating these missing values to secure the accuracy and reliability of the survey outcomes. This manuscript proposes some optimal class of memory type imputation methods for imputing missing data in time-based surveys by utilizing exponentially weighted moving average (EWMA) statistics. The insights into the optimal conditions for incorporating our proposed methods are provided. A comprehensive examination of the proposed method utilizing simulated and real-life datasets is conducted. Comparative analyses against the existing imputation methods exhibit the superior performance of our methods, particularly in the scenarios characterized by developing trends and dynamic response patterns. The outcomes highlight the effectiveness of utilizing EWMA statistics into memory type imputation methods, displaying their flexibility to changing survey dynamics.

Per Segment Plane Sweep Line Segment Intersection on the GPU

Polygon overlay operations are used for various purposes such as GIS searches and queries, VLSI and basic geometric operations of intersection, union and difference. There have been recent research articles presenting algorithms using the GPU to perform line segment intersection for geometric operations. We present two parallel algorithms implemented on the GPU that focus on the active list portion of the traditional serial plane sweep algorithm. The first algorithm uses a single block of threads to simulate the active list data structure in hardware; this algorithm is slow due to GPU thread block size limitations and synchronization points, but demonstrates favorable time complexity. The second algorithm uses dynamic parallelism to remove synchronization and scales to utilize available GPU hardware (single GPU). We perform experiments on both synthetic and real world data sets. The presented results show improvement in execution time with respect to recent algorithms, and low memory usage compared to recent algorithms. We achieve speedups of up to 38.8 over the serial sweep line algorithm on real world data.

SNP Genotype Imputation in Forensics—A Performance Study

Background/Objectives: Emerging forensic genetic applications, such as forensic investigative genetic genealogy (FIGG), advanced DNA phenotyping, and distant kinship inference, increasingly require dense SNP genotype datasets. However, forensic-grade DNA often contains missing genotypes due to its quality and quantity limitations, potentially hindering these applications. Genotype imputation, a method that predicts missing genotypes, is widely used in population and medical genetics, but its utility in forensic genetics has not been thoroughly explored. This study aims to assess the performance of genotype imputation in forensic contexts and determine the conditions under which it can be effectively applied. Methods: We employed a simulation-based approach to generate realistic forensic SNP genotype datasets with varying numbers, densities, and qualities of observed genotypes. Genotype imputation was performed using Beagle software, and the performance was evaluated based on the call rate and imputation accuracy across different datasets and imputation settings. Results: The results demonstrate that genotype imputation can significantly increase the number of SNP genotypes. However, imputation accuracy was dependent on factors such as the quality of the original genotype data and the characteristics of the reference population. Higher SNP density and fewer genotype errors generally resulted in improved imputation accuracy. Conclusions: This study highlights the potential of genotype imputation to enhance forensic SNP datasets but underscores the importance of optimizing imputation parameters and understanding the limitations of the original data. These findings will inform the future application of imputation in forensic genetics, supporting its integration into forensic workflows.

A Brief Introduction on Latent Variable Based Ordinal Regression Models With an Application to Survey Data.

The analysis of survey data is a frequently arising issue in clinical trials, particularly when capturing quantities which are difficult to measure. Typical examples are questionnaires about patient's well-being, pain, or consent to an intervention. In these, data is captured on a discrete scale containing only a limited number of possible answers, from which the respondent has to pick the answer which fits best his/her personal opinion. This data is generally located on an ordinal scale as answers can usually be arranged in an ascending order, for example, "bad", "neutral", "good" for well-being. Since responses are usually stored numerically for data processing purposes, analysis of survey data using ordinary linear regression models are commonly applied. However, assumptions of these models are often not met as linear regression requires a constant variability of the response variable and can yield predictions out of the range of response categories. By using linear models, one only gains insights about the mean response which may affect representativeness. In contrast, ordinal regression models can provide probability estimates for all response categories and yield information about the full response scale beyond the mean. In this work, we provide a concise overview of the fundamentals of latent variable based ordinal models, applications to a real data set, and outline the use of state-of-the-art-software for this purpose. Moreover, we discuss strengths, limitations and typical pitfalls. This is a companion work to a current vignette-based structured interview study in pediatric anesthesia.

Improving Teacher Training Through Emotion Recognition and Data Fusion

ABSTRACTThe quality of education hinges on the proficiency and training of educators. Due to the importance of teacher training, the innovative platform Teacher Moments creates simulated classroom scenarios. In this scenario‐based learning, confusion is an important indicator to detect users who struggle with the simulations. Through Teacher Moments, we gathered 7975 audio recording responses from participants who self‐labelled their recordings according to whether they sounded confused. Our dataset stands out for its size, for not including actor‐generated audio, and for measuring confusion, a neglected emotion in artificial intelligence (AI). Our experiments tested unimodal approaches and feature‐level, model‐level and decision‐level fusion. Feature‐level fusion demonstrated superior performance to unimodal methods, achieving a balanced accuracy of 0.6607 on the test set. This outcome highlights the necessity for further investigation in the overlooked area of confusion detection, particularly employing realistic datasets like the one used in this study and exploring new methods. Beyond teacher training, the insights of this research also extend to other directions, such as other professionals making critical decisions, user interface design or adaptive learning systems.

A non-linear integer-valued autoregressive model with zero-inflated data series

A new non-linear stationary process for time series of counts is introduced. The process is composed of the survival and innovation component. The survival component is based on the generalized zero-modified geometric thinning operator, where the innovation process figures in the survival component as well. A few probability distributions for the innovation process have been discussed, in order to adjust the model for observed series with the excess number of zeros. The conditional maximum likelihood and the conditional least squares methods are investigated for the estimation of the model parameters. The practical aspect of the model is presented on some real-life data sets, where we observe data with inflation as well as deflation of zeroes so we can notice how the model can be adjusted with the proper parameter selection.

ECM+: An improved evidential c-means with adaptive distance

Evidential c-means (ECM) is a prototype-based clustering algorithm that generates a credal partition. Such a partition encompasses the notions that can be encountered with a hard, fuzzy or possibilistic partition, allowing the representation of various situations concerning the class membership of an object. The ECM method provides a prototype for each subset of the possible classes, calculated by averaging the prototypes of the classes included in the subsets. Although this definition perfectly suits ECM when employing a Euclidean distance, it becomes inappropriate when using a Mahalanobis distance. In this context, a new definition of prototypes for the subsets is proposed. The ECM objective function is then optimized using the new definition of prototypes. The subsequent algorithm, named ECM+, is finally tested on various synthetic and real data sets to demonstrate its interest compared to ECM.

A Gaussian process model for stellar activity in 2D line profile time-series

ABSTRACT Stellar active regions like spots and faculae can distort the shapes of spectral lines, inducing variations in the radial velocities that are often orders of magnitude larger than the signals from Earth-like planets. Efforts to mitigate these activity signals have hitherto focused on either the time or the velocity (wavelength) domains. We present a physics-driven Gaussian process (GP) framework to model activity signals directly in time series of line profiles or cross-correlation functions (CCFs). Unlike existing methods that correct activity signals in line profile time series, our approach exploits the time correlation between velocity (wavelength) bins in the line profile variations, and is based on a simplified but physically motivated model for the origin of these variations. When tested on both synthetic and real data sets with signal-to-noise ratios down to ∼100, our method was able to separate the planetary signal from the activity signal, even when their periods were identical. We also conducted injection/recovery tests using two years of realistically sampled HARPS-N solar data, demonstrating the ability of the method to accurately recover a signal induced by a 1.5-Earth mass planet with a semi-amplitude of 0.3 m s−1 and a period of 33 d during high solar activity.

Trace Ratio Based Manifold Learning With Tensor Data

ABSTRACTIn this article, we propose an extension of trace ratio based Manifold learning methods to deal with multidimensional data sets. Based on recent progress on the tensor‐tensor product, we present a generalization of the trace ratio criterion by using the properties of the t‐product. This will conduct us to introduce some new concepts such as Laplacian tensor and we will study formally the trace ratio problem by discussing the conditions for the existence of solutions and optimality. Next, we will present a tensor Newton QR decomposition algorithm for solving the trace ratio problem. Manifold learning methods such as Laplacian eigenmaps, linear discriminant analysis and locally linear embedding will be formulated in a tensor representation and optimized by the proposed algorithm. Finally, we will evaluate the performance of the different studied dimension reduction methods on several synthetic and real world data sets.