Split Data Research Articles

Two sample test for covariance matrices in ultra-high dimension

In this paper, we propose a new test for testing the equality of two population covariance matrices in the ultra-high dimensional setting that the dimension is much larger than the sizes of both of the two samples. Our proposed methodology relies on a data splitting procedure and a comparison of a set of well selected eigenvalues of the sample covariance matrices on the split data sets. Compared to the existing methods, our methodology is adaptive in the sense that (i). it does not require specific assumption (e.g., comparable or balancing, etc.) on the sizes of two samples; (ii). it does not need quantitative or structural assumptions of the population covariance matrices; (iii). it does not need the parametric distributions or the detailed knowledge of the moments of the two populations. Theoretically, we establish the asymptotic distributions of the statistics used in our method and conduct the power analysis. We justify that our method is powerful under weak alternatives. We conduct extensive numerical simulations and show that our method significantly outperforms the existing ones both in terms of size and power. Analysis of two real data sets is also carried out to demonstrate the usefulness and superior performance of our proposed methodology. An R package UHDtst is developed for easy implementation of our proposed methodology.

Stock Price Prediction Research Based on CNN-LSTM

In recent years, neural network technology has been widely used in the field of stock forecasting, especially in the aspect of LSTM, which has become a research hotspot because of its unique advantages in dealing with time series problems in stock forecasting. By analyzing and summarizing the recent research literature based on LSTM and its extended model, this paper aims to deeply explore the effect and potential limitations of LSTM in capturing the dynamic and nonlinear characteristics of the stock market. By constructing different combination models combining LSTM and CNN, the paper uses normalization to preprocess the split data and realize the generalization prediction of stock price. Compared with the experimental results, a suitable combination model of LSTM and CNN is found. This hybrid model can significantly improve the accuracy and stability of prediction, and also shows the advantages in multi-input data processing. The results of this study not only provide a new technical means for the direction of stock prediction, but also explore a new direction for the further application of deep learning technology in the financial field.

Data-adaptive binary classifiers in high dimensions using random partitioning

Classification in high dimensions has been highlighted for the past two decades since Fisher's linear discriminant analysis (LDA) is not optimal in a smaller sample size n comparing the number of covariates p, i.e. p>n, which is mostly due to the singularity of the sample covariance matrix. Rather than modifying how to estimate the sample covariance and sample mean vector in constructing a classifier, we build two types of high-dimensional classifiers using data splitting, i.e. single data splitting (SDS) and multiple data splitting (MDS). Moreover, we introduce a weighted version of MDS classifier that improves classification performance as illustrated in numerical studies. Each of the split data sets has a smaller size of covariates compared to the sample size so that LDA is applicable, and classification results can be combined with respect to minimizing the misclassification rate. We present theoretical justification backing up our proposed methods by comparing misclassification rates with LDA in high dimension. We also conduct a wide range of simulations and analyse four microarray data sets, which demonstrates that our proposed methods outperform some existing methods or at least yield comparable performances.

A stochastic optimization procedure to design the fair aggregation of energy users in a Renewable Energy Community

Optimizing unit sizes and operation within a Renewable Energy Community (REC) can match intermittent renewable energy generation with variable user energy demands. These uncertain variables are often represented by pre-defined stochastic scenarios, without searching for the “best” scenarios and testing the optimization models with these scenarios. Moreover, little work both optimized RECs under uncertainty and distributed optimal life-cycle costs (investment and operation) among members. Thus, the objectives are: i) identifying the “best” set of stochastic scenarios of solar irradiance and user electricity demands and ii) assessing the accuracy of the “stochastic forecasts” of the total system costs and unit sizes, obtained by solving a stochastic programming model based on the “best” scenarios. The proposed novel procedure shifts the “present moment” back in time to split historical data into “past” and “future” periods used to identify the “best” scenarios and compare the “stochastic forecasts” with the utopic “perfect forecasts” based on the perfect knowledge of real data, respectively. The small errors between these forecasts in the optimal life-cycle costs (less than 2 %) and sizes (3–13 %) indicate good effectiveness of the suggested procedure. Also, the optimal life-cycle costs of “stochastic forecasts” are fairly distributed among users by applying the Shapley value mechanism.

Forecasting Occupancy Rate using Neural Network and Decision Tree at Hotel X

Occupancy rate is a critical factor in a hotel because it is used to measure the operational success of a hotel. The higher the hotel occupancy rate, the more successful the hotel business is in generating revenue. In the hospitality business, occupancy rates are very important to monitor and analyse as marketing strategies and pricing policies. This study compares the forecasting of occupancy rates at Hotel X using neural network method and decision tree method. The dataset used in this study is room available, room sold, and available occupancy percentage data at Hotel X from April 2018 to June 2023. The simulation was carried out by dividing the data into training data and testing data with a percentage ratio of 70:30, 75:25, 80:20, 85:15, and 90:10. The forecasting results that have been carried out using a neural network with one hidden layer have an optimal RSME result of 0.010 for split data of 70%:30% and 80%:20% while using a neural network with two hidden layers the optimal result of RSME is 0.013 for split data of 75%:25%. Forecasting results using decision tree RSME optimal results of 0.022 for split data 85%:15% and 90%:10%. From this forecasting, the most optimal results use a neural network with one hidden layer for data splits of 70%:30% and 80%:20% with RSME results of 0.010. The results of the research can be used by Hotel X as a policy determination in the next hotel management.

Rapid point-of-care breath test predicts breast cancer and abnormal mammograms in symptomatic women

Previous studies have reported volatile organic compounds (VOCs) in the breath as biomarkers of breast cancer. These biomarkers may be derived from cancer-associated fibroblasts, in which oxidative stress degrades polyunsaturated fatty acids to volatile alkanes and methylated alkane derivatives that are excreted in the breath. We evaluated a rapid point-of-care test for breath VOC biomarkers as predictors of breast cancer and abnormal mammograms.&#xD;Methods: We studied 593 women aged ≥ 18 yr referred to three sites for mammography for a symptomatic breast-related concern (e.g. breast mass, nipple discharge). A rapid point-of-care breath testing system collected and concentrated alveolar breath VOCs on a sorbent trap and analyzed them with gas chromatography and surface acoustic wave detection in < 6 min. Breath VOC chromatograms were randomly assigned to a training set or to a validation set. Monte Carlo analysis identified significant breath VOC biomarkers of breast cancer and abnormal mammograms in the training set, and these biomarkers were incorporated into a multivariate algorithm to predict disease in the validation set. &#xD;Results: Prediction of breast cancer: 50 women had biopsy-proven breast cancer (invasive cancer 41, ductal non-invasive cancer 9)&#xD;Unsplit data set: Breath VOCs identified breast cancer with 83% accuracy (area under curve of receiver operating characteristic), 82% sensitivity and 77.1% specificity.&#xD;Split data sets: Training set breath VOCs identified breast cancer with 80.3% accuracy, 84% sensitivity and 74.3% specificity. Corresponding values in the validation set were 68%% accuracy, 72.4% sensitivity and 61.5% specificity.&#xD;Prediction of BIRADS 4 and 5 mammograms (versus BIRADS 1, 2 and 3): &#xD;Unsplit data set: Breath VOCs identified abnormal mammograms with 76.2% accuracy.&#xD;Split data sets: Breath VOCs identified abnormal mammograms with 74.2% accuracy, 73.3% sensitivity and 60% specificity. Corresponding values in the validation set were 60.5% accuracy, 64.2% sensitivity and 51% specificity.&#xD;Conclusions: A rapid point-of-care test for breath VOC biomarkers predicted risk of breast cancer and abnormal mammograms in women with breast-related symptoms.&#xD;&#xD.

Modified data classification for extreme values in Şen’s innovative trend analysis: A comparative trend study for the Aegean and Eastern Anatolia Regions of Türkiye

The increase in greenhouse gases in the atmosphere has worsened global warming, and marked changes have been observed in meteorological and climatic events, especially since the early 2000s. Trend analysis studies are important for determining changes in meteorological and climatic events over time. This study investigated the trends of maximum precipitation and minimum temperature in the Aegean Region and Eastern Anatolia Region of Türkiye by conducting an innovative trend analysis (ITA), the Mann–Kendall (MK) test, and linear regression analysis (LRA). As a method, ITA has been used together with traditional methods in the last decade, and its advantages have been demonstrated in comparative trend studies. An important contribution of ITA is that it can categorize datasets according to their size (low, medium, and high). The classification technique of the ITA method includes dividing the sorted dataset into three equal parts and separately examining the trends of low, medium, and high data values. This approach is reasonable for datasets with low skewness (or normally distributed series). However, the normal distribution acceptance of ITA data classification is insufficient for trend analysis of data series with extreme values. Therefore, we propose a modified data classification method to rationally examine skewed datasets with the use of quartiles. Our study was performed for the trend analysis of maximum rainfall and minimum temperature data in two regions located in the west and east of Türkiye showing different climatic characteristics. In the first part of the study in which the numerical trend analysis of ITA was evaluated, the MK and LRA methods showed similar results, whereas the ITA detected trends at a greater number of stations owing to its sensitivity feature in detecting trends. In the second part, which included data classification in trend analysis, the equal split data classification used in the ITA and the modified data classification proposed in the study were compared. The comparative results of the trend analysis of the maximum rainfall and minimum temperature data showed the superiority of the proposed data classification in examining the trend of extreme values, especially for maximum rainfall data with relatively high skewness.

Smart Data Driven Decision Trees Ensemble Methodology for Imbalanced Big Data

Differences in data size per class, also known as imbalanced data distribution, have become a common problem affecting data quality. Big Data scenarios pose a new challenge to traditional imbalanced classification algorithms, since they are not prepared to work with such amount of data. Split data strategies and lack of data in the minority class due to the use of MapReduce paradigm have posed new challenges for tackling the imbalance between classes in Big Data scenarios. Ensembles have been shown to be able to successfully address imbalanced data problems. Smart Data refers to data of enough quality to achieve high-performance models. The combination of ensembles and Smart Data, achieved through Big Data preprocessing, should be a great synergy. In this paper, we propose a novel Smart Data driven Decision Trees Ensemble methodology for addressing the imbalanced classification problem in Big Data domains, namely SD_DeTE methodology. This methodology is based on the learning of different decision trees using distributed quality data for the ensemble process. This quality data is achieved by fusing random discretization, principal components analysis, and clustering-based random oversampling for obtaining different Smart Data versions of the original data. Experiments carried out in 21 binary adapted datasets have shown that our methodology outperforms random forest.

SplitDF: Splitting Dataframes for Memory-Efficient Data Analysis

Dataframe is a popular construct in data analysis libraries that offers a tabular view of the data. However, data within a dataframe often has redundancy, which can lead to high memory utilization of data analysis libraries. Inspired by the process of normalization in relational database systems, we propose a technique called splitting that can be applied to tabular data to reduce redundancy. Splitting involves performing lossless join decomposition by explicitly adding joining keys, and unlike normalization, splitting can be applied to tabular data without the need to perform functional dependency discovery. A split dataframe provides the same unified tabular view to the data, while internally operating on split data to improve memory efficiency. We develop SplitDF, an implementation of split dataframes in Ibis for DuckDB backend, which enables data analysis on split data with minimal changes to the Ibis API. Generation of split tabular data is automated using an algorithm SplitGen implemented in Velox. In our analysis involving ten handwritten notebooks running on SplitDF, we observe a reduction in memory usage of 19--61% when operating on split data as compared to operating on original data.

The Thai intrapreneur – A mixed methods study exploring intrapreneurship in Thailand

PurposeThis study explores the concept and antecedents of Intrapreneurship among Thai employees in Bangkok, Thailand. Study design/methodology/approachThe study uses an exploratory sequential mixed methodology. The data were collected from twenty-two in-depth interviews with Thai entrepreneurs and employees, followed by a survey of 648 salaried employees from Bangkok. FindingsThe findings from the interviews suggested that a sense of self-belief and ownership, collaboration in a team, the immediate supervisor, certain characteristics of the job and organization, and a culture of innovation in the organization affected intrapreneurship. The quantitative survey findings confirmed that self-efficacy, entrepreneurial orientation, team cohesion, and organizational support were significantly related to intrapreneurship in Thai companies. Additionally, a split data analysis revealed that the factors affecting intrapreneurship were distinct for the male and female cohorts, whereby self-efficacy was significant in the male data, whereas entrepreneurial orientation and team cohesion were significantly related to intrapreneurship in the female data. Practical implicationsThe study proposes the creation of small teams, introduction of role models and internships, and creation of a platform for exchange of ideas as recommendations to encourage intrapreneurship. The study also notes that these initiatives might affect male and female intrapreneurs distinctly. Originality/valueThis is the first notable mixed methods study on intrapreneurship in Thailand. The findings of this study will, therefore, help managers to encourage employees towards intrapreneurial behavior and pave the way for future research on intrapreneurship in Thailand.

ADVANCING LYME DISEASE PREVENTION THROUGH COMPUTER VISION: A ROBUST APPROACH FOR TICK IDENTIFICATION

Lyme is a disease that is caused by Borrelia burgdorferi, a bacterium that is spread by ticks. The prevalence of Lyme disease has made it a major public health problem. Immediate identification of the bacteria-carrying parasites is important in preventing the epidemic. This research suggests an alternative approach which uses computer vision to identify Lyme diseases related to ticks. A dataset containing images of ticks was used to create and train a Convolutional Neural Network (CNN) model. Preprocessing and augmentation were done on the dataset with split data into training and testing sets prior to boosting model generalization. The architecture of the CNN consists of convolutional, batch normalization and pooling layers followed by fully connected layers for classification. The Adam optimizer trains the model with a piecewise learning rate schedule. Test set evaluation shows promising results with high accuracy in categorizing tick pictures. Furthermore, this study calculates precision, recall and F1 score metrics which indicates strong performance from this model. A confusion matrix as well as visualization is also used to prove that model can distinguish between different tick classes. This computer vision approach provides a powerful tool for automatic tick recognition thus aiding in early detection as well as prevention of Lyme disease KEYWORDS; Image analysis, deep learning, tick identification, epidemiological surveillance, disease management, public health interventions, artificial intelligence, zoonotic diseases, tick-borne pathogens, predictive modeling.

대용량 자료의 이항 분류에서의 충분 차원 축소를 위한 전향적 접근 방법

Sufficient dimension reduction, aimed at finding a lower-dimensional subspace in explanatory variables that contains response variable information, typically relies on inverse-based methodologies. These methods are easy to implement but often require linear or constant variance conditions. To address these limitations, techniques for forwardly estimating the central subspace have been developed. In particular, methods utilizing the Reproducing Kernel Hilbert Space have gained attention, but their use in analyzing large datasets is limited due to the characteristics of the kernel space. In this paper, we study a novel forward approach for sufficient dimension reduction in binomial classification of large-scale data. We propose a method that employs a divide-and-conquer technique to split data into subsets, then independently perform dimension reduction on each subset before synthesizing them into a final model. It was shown that when the number of partitions of data was appropriately selected, the loss in prediction accuracy was not significant compared to the existing method, while being efficient in terms of storage space and calculation cost. In addition, simulations in various models showed superior prediction accuracy than other inverse-based techniques. The utility of the proposed method was confirmed through the real data analysis.

The Effect of Stock Split on Stock Trading Volume and Stock Returns in Companies Listed on the Indonesia Stock Exchange (IDX) in the Years 2020-2023

This study aims to determine the effect of stock splits on stock trading volume and stock returns in companies listed on the Indonesian stock exchange. The population in this study were 35 companies, 31 of which were used as samples selected by purposive sampling method. The type of data used is secondary data, with the source of company financial report data which includes stock split data, stock trading volume, and stock returns published on the official website of the Indonesian stock exchange. With an observation period of 5 days before and 5 days after the stock split. The data analysis used is a paired sample difference test, namely the Wilcoxon signed rank-test test because the data is not normally distributed. The results showed that stock splits had no significant effect on stock trading volume with a significance value of 0.387&gt; 0.05 while stock splits had a significant effect on stock returns with a significance value of 0.037 &lt;0.05.

Advancing scaffold porosity through a machine learning framework in extrusion based 3D bioprinting

Three Dimensional (3D) bioprinting holds great promise for tissue and organ regeneration due to its inherent capability to deposit biocompatible materials containing live cells in precise locations. Extrusion-based 3D bioprinting (EBP) method stands out for its ability to achieve a higher cell release rate, ensuring both external and internal scaffold structures. The systematic adjustment of key process parameters of EBP, including nozzle diameter, printing speed, print distance, extrusion pressure, material fraction, and viscosity allows for precise control over filament dimensions, ultimately shaping the desired scaffold porosity as per user specifications. However, managing these factors with all possible interactions simultaneously to achieve the desired filament width can be intricate and resource intensive. This study presents a novel framework designed to construct a predictive model for the filament width of 3D bioprinted scaffolds for various process parameters. A total of 157 experiments have been conducted under various combinations of process parameters and biomaterial’s weight fraction for this study purpose. A regression-based machine learning approach is employed to develop the predictive model utilizing Adj. R2, Mallow’s Cp, and Bayesian Information Criterion (BIC). Following model development, rigorous experimental validations are conducted to assess the accuracy and reliability of the model. Based on the cross-validation of randomly split test data, Adj. R2 model emerges as the highest performing machine learning model (Mean Squared Error, MSE = 0.0816) compared to Mallow’s Cp and BIC (MSE = 0.0841 and 0.0877, respectively) models. The comparative analysis results between the experimental and model’s data demonstrate that our predictive model achieves an accuracy of approximately 85% in filament width prediction. This framework presents a significant advancement in the precise control and optimization of 3D bioprinted scaffold fabrication, offering valuable insights for the advancement of tissue engineering and regenerative medicine applications.

Klasifikasi Multi Class Pada Metode Kerja Jarak Jauh Menggunakan Algoritma Decision Tree dan Imbalance Data

The COVID-19 pandemic that has hit the world has forced workers to work from home to prevent the spread of the virus. However, until the pandemic is over, based on survey conducted on 100 employee, found that working from home (Work from Home) is still a choice that many workers are interested in because it considered to provide flexibility and save more times. But some of them prefer to work from the office because it is considered easier than to focus and can increase productivity and more interested mixed mode of working. The analysis and comparison determined to find out about which work locations are more popular with workers. One solution to overcome this problem is that a classification method is needed to group the factors that influence the choice of work location. The classification method used for data processing is the Decision Tree method. The method for class imbalance problems uses the Synthetic Minority Over-sampling Technique (SMOTE) method. Tests were execute using Decision Tree and SMOTE split data which obtained an accuracy of up to 83.08% at a ratio of 0.5 (5:5). In this research, it was found that 13% of workers preferred to work from home, 25% chose to work from an office, and 27% chose mixed work models.

Linear Kernel Optimization of Support Vector Machine Algorithm on Online Marketplace Sentiment Analysis

Twitter is a short message platform commonly used as a means of news information, commentary, and social interaction. One of the utilization of twitter is to analyze the sentiment of the online marketplace which can be used to determine the service, quality of goods, and delivery of goods on a product, service or application. This research aims to categorize the reviews or responses of the Indonesian people, especially to the online marketplace using the linear Support Vector Machine (SVM) algorithm. In order to make continuous improvements to the role of the Indonesian online marketplace in the future, sentiment analysis is needed. The analysis research tweets used were 4165 datasets using the python programming language. Sentiment analysis research stages include data collection, preprocessing, labeling, tf-idf weighting, split data, SVM model analysis and result evaluation. The data is then divided into 80% training data and 20% testing data, 50% training data and 50% testing data, 20% training data and 80% testing data. The results of the svm algorithm testing scenario obtained the highest optimization with an accuracy value of 97%, F1-score value on positive labels 88% and negative 98%, also obtained a positive recall value of 80% and negative 100% precision value on positive labels 98% and negative 97%, on 80% training data and 20% testing. It can be concluded that in this case, the linear svm algorithm is able to work to recognize models with a high level of accuracy so that in the future it can be used in similar cases.

Machine Learning on Opinion Mining of Netizen's Hate Speech

&#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; Netizen comments written in an online news portal through social media platforms, one of which is Instagram, can be used as material in the sentiment analysis process, which can be classified into positive, negative, or neutral sentiments. Sentiment analysis is part of the study of text mining, the science of discovering unknown knowledge by automatically extracting information from large volumes of unstructured text into useful information. The resulting information is in the form of sentiment towards a topic, whether it tends to be positive, negative, or neutral. The classification method used in this research is Support Vector Machine (SVM) and TF-IDF data weighting to classify text. Stages to perform data analysis are pre-processing to clean data, word weighting, labeling data into positive, negative, or neutral classes, and classifying and visualizing data with graphs. Accuracy tests using 70:30 split data showed that the accuracy reached 98%. Tests with 80:20 and 90:10 split data also showed high accuracy of 98% and 99%.&#x0D; &#x0D; &#x0D; &#x0D;

Comparison PSO And IWPSO Performance In Optimizing Decision Tree Algorithm On Heart Disease Dataset

Heart disease, one of the most common and potentially fatal chronic diseases, has become a major focus in global health efforts. In this study, researchers used the decision tree algorithm on the heart disease dataset with the stages of the decision algorithm including the EDA, Split Data, and Decision tree modeling stages. Furthermore, hyperparameters use PSO and IWPSO to optimize the algorithm. The purpose of this research is to analyze the performance of Particle Swarm Optimization (PSO) and Inertia Weight Particle Swarm Optimization (IWPSO) in heart disease prediction based on relevant datasets. PSO and IWPSO were applied to the heart disease dataset, with the results showing an accuracy rate of 78% for PSO and 84% for IWPSO. These results indicate that IWPSO provides significant performance improvement compared to PSO in the context of heart disease prediction. The implications of these findings can support the development of more efficient prediction systems for early detection of heart disease, making a positive contribution to prevention efforts and further treatment of this critical health condition. In addition, the purpose of this research is to continue research in the form of C4.5 on heart disease with a result of 80.43%. In this study, IWPSO got the best accuracy of 84.23% greater than previous research. The results of this study are to provide insight that PSO and IWPSO hyperparameters can optimize decision trees in handling heart disease datasets and continue research.

Longitudinal clinical data improve survival prediction after hematopoietic cell transplantation using machine learning

AbstractSerial prognostic evaluation after allogeneic hematopoietic cell transplantation (allo-HCT) might help identify patients at high risk of lethal organ dysfunction. Current prediction algorithms based on models that do not incorporate changes to patients’ clinical condition after allo-HCT have limited predictive ability. We developed and validated a robust risk-prediction algorithm to predict short- and long-term survival after allo-HCT in pediatric patients that includes baseline biological variables and changes in the patients’ clinical status after allo-HCT. The model was developed using clinical data from children and young adults treated at a single academic quaternary-care referral center. The model was created using a randomly split training data set (70% of the cohort), internally validated (remaining 30% of the cohort) and then externally validated on patient data from another tertiary-care referral center. Repeated clinical measurements performed from 30 days before allo-HCT to 30 days afterwards were extracted from the electronic medical record and incorporated into the model to predict survival at 100 days, 1 year, and 2 years after allo-HCT. Naïve-Bayes machine learning models incorporating longitudinal data were significantly better than models constructed from baseline variables alone at predicting whether patients would be alive or deceased at the given time points. This proof-of-concept study demonstrates that unlike traditional prognostic tools that use fixed variables for risk assessment, incorporating dynamic variability using clinical and laboratory data improves the prediction of mortality in patients undergoing allo-HCT.

SIMPD: an algorithm for generating simulated time splits for validating machine learning approaches

Time-split cross-validation is broadly recognized as the gold standard for validating predictive models intended for use in medicinal chemistry projects. Unfortunately this type of data is not broadly available outside of large pharmaceutical research organizations. Here we introduce the SIMPD (simulated medicinal chemistry project data) algorithm to split public data sets into training and test sets that mimic the differences observed in real-world medicinal chemistry project data sets. SIMPD uses a multi-objective genetic algorithm with objectives derived from an extensive analysis of the differences between early and late compounds in more than 130 lead-optimization projects run within the Novartis Institutes for BioMedical Research. Applying SIMPD to the real-world data sets produced training/test splits which more accurately reflect the differences in properties and machine-learning performance observed for temporal splits than other standard approaches like random or neighbor splits. We applied the SIMPD algorithm to bioactivity data extracted from ChEMBL and created 99 public data sets which can be used for validating machine-learning models intended for use in the setting of a medicinal chemistry project. The SIMPD code and simulated data sets are available under open-source/open-data licenses at github.com/rinikerlab/molecular_time_series.