Population Stability Index Research Articles

Detecting drifts in data streams using Kullback-Leibler (KL) divergence measure for data engineering applications

The exponential growth of data coupled with the widespread application of artificial intelligence(AI) presents organizations with challenges in upholding data accuracy, especially within data engineering functions. While the Extraction, Transformation, and Loading process addresses error-free data ingestion, validating the content within data streams remains a challenge. Prompt detection and remediation of data issues are crucial, especially in automated analytical environments driven by AI. To address these issues, this study focuses on detecting drifts in data distributions and divergence within data fields processed from different sample populations. Using a hypothetical banking scenario, we illustrate the impact of data drift on automated decision-making processes. We propose a scalable method leveraging the Kullback-Leibler (KL) divergence measure, specifically the Population Stability Index (PSI), to detect and quantify data drift. Through comprehensive simulations, we demonstrate the effectiveness of PSI in identifying and mitigating data drift issues. This study contributes to enhancing data engineering functions in organizations by offering a scalable solution for early drift detection in data ingestion pipelines. We discuss related research works, identify gaps, and present the methodology and experiment results, underscoring the importance of robust data governance practices in mitigating risks associated with data drift and improving data observability.

Establishing the temporal stability of machine learning models that detect online gambling-related harms

Artificial Intelligence (AI) models can detect at-risk online gamblers by analyzing patterns in their betting behaviour, but their performance over time has not been assessed. Linking online gamblers’ self-reported gambling problems to their transactional gambling data, we investigated the temporal stability of online gambling behaviours at two timepoints (2019–2022). We then assessed the impacts of shifting gambling behaviours on the performance of two previously-validated AI harm detection models.Adult users of a Canadian online gambling website (n2019 = 9,145, n2022 = 11,258) completed the Problem Gambling Severity Index (PGSI), a validated questionnaire examining past-year gambling problems. Population Stability Index assessed the temporal stability of 10 previously-validated indicators of problematic online gambling. Two AI models were then revalidated using overall and threshold-dependent performance metrics.All measured indicators of problematic online gambling behaviour shifted significantly from 2019 to 2022 (all p < 0.001). Existing AI models showed significant changes in Area Under the Precision-Recall Curve (PGSI 5+ ΔAUPRC = +2.87%; t(20401) = 2.83, p = 0.004; PGSI 8+ ΔAUPRC = +7.06%; t(20401) = 7.21, p < 0.001). Adjusting these models’ decision thresholds realigned their classification performance with originally-validated levels. These results support the use of AI for online gambling harm detection.

Effective sampling for drift mitigation in machine learning using scenario selection: A microgrid case study

Predictive modeling is pervasive across a variety of industries (e.g., driving patterns in insurance, fraud detection in finance, and forecasting in energy). However, the deterioration of prediction accuracy for most machine learning (ML) models due to the drift is inevitable despite having acceptable performance in a test setting. Drift is a natural occurrence as cumulative information changes over time but should be regulated to ensure an ML model’s prediction remains relevant in real-time when decisions are being made. This study proposes a drift mitigation framework (DMF) to obtain an effective sample size in a scalable and unbiased manner for sustainable predictive performance from ML models using microgrids (MGs) as a testbed. Scenario selection was used as an alternative sampling technique to obtain an effective sample from historical data contributing to probability theory. These scenarios were evaluated using a quantile-quantile (Q-Q) plot, Welch’s T-test, and relative entropy measures to ensure the obtained sample effectively captures the general population behavior. The results showed two of the three climate factors used in our experiment formed a nearly straight line on their Q-Q plots, suggesting the effective sample retained comparable statistical properties to theoretical distributions fitted to the original population and complete training data. At an alpha equal to 0.05, a hypothesis test was conducted with the null hypothesis claiming the means of reduced and complete data were statistically equivalent for ambient temperature, solar irradiance, and wind speed. The p-values obtained for each stochastic climate factor were 1.01, 0.296, and 4.13 suggesting insufficient evidence to reject the null hypothesis. Similarly, the resulting population stability index (PSI) values for each stochastic climate factor were 0.175, 0.352, and 0.023 respectively. Hence, there was no significant shift between the population data and the effective sample across each factor.

Дослідження скорингової моделі для кредитопозичальників банку

In the paper we investigate scoring models as a tool for credit risk management, their importance, types, features and applications. We consider the history of scoring models and the development of the modern concepts of creditworthiness and scoring, examine scoring types and their features, methods (logistic regression, decision trees, linear programming, decision trees, and others), strengths and weaknesses of each considered method and stages of building scoring models; we stress the importance of scoring cards for building scoring models, indicate the main quantitative and qualitative features, their classification used for and describe the procedure of creating scoring cards. In the paper we consider the factors needed to be considered for more effective scoring model building process; we indicate the main formulas used to assess the creditworthiness of borrowers and improve the accuracy of scoring models, including Population Stability Index (PSI), R-Square Coefficient, Kolmogorov-Smirnov Coefficient, GINI Coefficient, and others; the practical application of scoring models in banks of Ukraine and their steps after applying scoring models are described on the example of a typical borrower.

Características morfogênicas, estruturais e índice de estabilidade populacional de perfilhos de azevém submetidos a métodos de lotação

ABSTRACT This study aimed to evaluate the morphogenic and structural characteristics and the population stability index of ryegrass (Lolium multiflorum Lam.) tillers when the pasture was submitted to two stocking methods during grazing cycles. The experimental design was a complete randomized block, with repeated measurements over time (n=6 cycles), two treatments (stocking methods) and three area replicates. In the continuous stocking method, the highest population density of tillers was observed. The highest tiller weight occurred in the rotational stocking method. The morphogenic variables and the other structural variables were not altered by the stocking methods and showed differences during the grazing cycles. The tiller population stability index was similar in the two pasture management strategies, and both can be used for ryegrass management considering this parameter.

Prediction Model for the Risk of HIV Infection among MSM in China: Validation and Stability

The impact of psychosocial factors on increasing the risk of HIV infection among men who have sex with men (MSM) has attracted increasing attention. We aimed to develop and validate an integrated prediction model, especially incorporating emerging psychosocial variables, for predicting the risk of HIV infection among MSM. We surveyed and collected sociodemographic, psychosocial, and behavioral information from 547 MSM in China. The participants were split into a training set and a testing set in a 3:1 theoretical ratio. The prediction model was constructed by introducing the important variables selected with the least absolute shrinkage and selection operator (LASSO) regression, applying multivariate logistic regression, and visually assessing the risk of HIV infection through the nomogram. Receiver operating characteristic curves (ROC), Kolmogorov–Smirnov test, calibration plots, Hosmer–Lemeshow test and population stability index (PSI) were performed to test validity and stability of the model. Four of the 15 selected variables—unprotected anal intercourse, multiple sexual partners, involuntary subordination and drug use before sex—were included in the prediction model. The results indicated that the comprehensive prediction model we developed had relatively good predictive performance and stability in identifying MSM at high-risk for HIV infection, thus providing targeted interventions for high-risk MSM.

Monitoring the health status of water mains using a scorecard modelling approach

Abstract There has been considerable research into prediction of water mains failure, however, those models are very complex and fail to convey the message of the health status of an asset to the relevant stakeholders. The study developed a ‘pipe health scorecard’ based on historical failure data which could be used for operation, maintenance, refurbishment, or replacement decisions by a water utility. This scorecard model was developed by using 160,413 pipe-condition data sets from the South Australian Water Corporation over ten years. Measures such as the Kolmogorov–Smirnov (KS) statistic, Area Under the ROC Curve (AUC), and Population Stability Index (PSI) showed the model is strong enough to predict the health status of water mains. The study found the factors influencing water mains failure to be in the order of importance: length, material, age, location (road vs verge), diameter, and operating parameters. The development of a simple but reliable model for the assessment of the health status of water mains will have major benefits to the water utility with the ability to identify and potentially replace water pipes prior to failure. Additional benefits of flexible scheduling of maintenance and replacement programs would contribute to cost savings.

Prediction Model for COVID-19 Vaccination Intention among the Mobile Population in China: Validation and Stability.

Since China’s launch of the COVID-19 vaccination, the situation of the public, especially the mobile population, has not been optimistic. We investigated 782 factory workers for whether they would get a COVID-19 vaccine within the next 6 months. The participants were divided into a training set and a testing set for external validation conformed to a ratio of 3:1 with R software. The variables were screened by the Lead Absolute Shrinkage and Selection Operator (LASSO) regression analysis. Then, the prediction model, including important variables, used a multivariate logistic regression analysis and presented as a nomogram. The Receiver Operating Characteristic (ROC) curve, Kolmogorov–Smirnov (K-S) test, Lift test and Population Stability Index (PSI) were performed to test the validity and stability of the model and summarize the validation results. Only 45.54% of the participants had vaccination intentions, while 339 (43.35%) were unsure. Four of the 16 screened variables—self-efficacy, risk perception, perceived support and capability—were included in the prediction model. The results indicated that the model has a high predictive power and is highly stable. The government should be in the leading position, and the whole society should be mobilized and also make full use of peer education during vaccination initiatives.

Height and mowing of pasture at the end of winter modulate the tillering of Marandu palisadegrass in spring

In pastures subjected to stockpiling, the tiller population goes through an intense process of self-thinning, hindering the recruitment of new tillers in the subsequent season. We evaluated different pasture management strategies in late winter in an attempt to modify tiller recruitment during spring. Urochloa brizantha cv. Marandu was maintained at 4 different levels (heights) of stockpiled pasture at the end of winter: short (15.1 cm), medium (23.2 cm), tall (31.4 cm) and tall/mown (31.3 cm, mown to 8 cm). In October (early spring), the short and tall/mown pastures had a tiller appearance rate (TAR) and a population stability index (PSI) superior (P&lt;0.05) to that of the tall pasture. During the remainder of the growing season, these characteristics (TAR and PSI) were similar for all pastures. Tiller survival rate (TSR) was also highest (P&lt;0.05) in short pasture in early spring. TAR values were highest in early spring and these tillers persisted throughout the growing season. When stockpiling Marandu palisadegrass pasture during spring it is important to have it short at the end of winter to ensure early and intense tillering in spring. If pasture is tall at the end of winter mowing at this time before spelling is advantageous.

Dataset shift assessment measures in monitoring predictive models

The article presents the results of the study, which is a fragment of the work carried out under the project entitled “Hybrid system for intelligent diagnostics of prognostic models”, co-financed through the National Centre for Research and Development from the European Regional Development Fund. They concerned the analysis of the phenomenon of dataset shift, also known as the shift of variable distributions. It is important to answer the question: has the distribution of current data for the implemented forecasting model changed significantly compared to the distribution of data used to develop it? If so, it could lead to incorrect operation. In the context of assessing and monitoring the stability of variable distributions of predictive models, the aim of the study was to compare the properties of two indicators, the Population Stability Index (PSI) and Population Accuracy Index (PAI). These measures were calculated for 78 controlled shifts of the distribution of the 3 explanatory variables of the hypothetical prognostic model. The research procedure was carried out in 2 scenarios. The first involved a comparison of PSI and PAI for the distributions of categorical variables. In scenario 2, an answer was sought to the question whether discretization of variables significantly influenced the assessment of the stability of their distributions using PSI compared to PAI, which does not require such a procedure? The results of the research proved that both indicators complement each other well, and when used together to assess the stability of the model’s variable distributions, they compensate each other’s shortcomings. PSI and PAI measure subtly different concepts of stability – PSI measures any change in the distribution of explanatory variables, and PAI only measures how this change affects the prognostic accuracy of the model – therefore they should be treated as complementary measures.

Lessons learned from data stream classification applied to credit scoring

The financial credibility of a person is a factor used to determine whether a loan should be approved or not, and this is quantified by a ‘credit score,’ which is calculated using a variety of factors, including past performance on debt obligations, profiling, amongst others. Machine learning has been widely applied to automate the development of effective credit scoring models over the years. Yet, studies show that the development of robust credit scoring models may take longer than a year, and thus, if the behavior of customers changes over time, the model will be outdated even before its deployment. In this paper, we made 3 anonymized real-world credit scoring datasets available alongside the results obtained. In each of these datasets, we verify whether the credit scoring task should be thought as an ephemeral scenario since many of the variables may drift over time, and thus, data stream mining techniques should be used since they were tailored for incremental learning and to detect and adapt to changes in the data distribution. Therefore, we compare both traditional batch machine learning algorithms with data stream algorithms in different validation schemes using both Kolmogorov–Smirnov and Population Stability Index metrics. Furthermore, we also provide insights on the importance of features according to their Information Value, Mean Decrease Impurity, and Mean Positional Gain metrics, such that the last depicts changes in the importance of features over time. For 2 of the 3 tested datasets, the results obtained by data stream learners are comparable to predictive models currently in use, thus showing the efficiency of data stream classification for the credit scoring task.

On testing the hypothesis of population stability for credit risk scorecards

Scorecards are models used in credit risk modelling. These models segments a population into various so-called \risk based on the risk characteristics of the individual clients. Once a scorecard has been developed, the credit provider typically prefers to keep this model in use for an extended period. As a result, it is important to test whether or not the model still ts the population. To this end, the hypothesis of population stability is tested; this hypothesis speci es that the current proportions of the population in the various risk buckets are the same as was the case at the point in time at which the scorecard was developed. In practice, this assumption is usually tested using a measure known as the population stability index (which corresponds to the asymmetric Kullback-Leibler discrepancy between discrete distributions) together with a well-known rule of thumb. This paper considers the statistical motivation for the use of the population stability index. Numerical examples are provided in order to demonstrate the e ect of the rule of thumb as well as other critical values. Although previous numerical studies relating to this statistic are available, the sample sizes are not realistic for the South African credit market. The paper demonstrates that the population stability index has little statisticalmerit as either a goodness-of- t statistic to test the hypothesis of population stability or as an intuitive discrepancy measure. As a result, a novel methodology for testing the mentioned hypothesis is proposed. This methodology includes a restatement of the hypothesis to specify a range of \acceptable deviations from the speci ed model. An alternative test statistic is also employed as discrepancy measure; this measure has the advantage of having a simple heuristic interpretation in the context of credit risk modelling. Key words: Goodness-of- t testing, hypothesis testing, population stability, risk analysis.

Dinámica poblacional de tallos e índice de estabilidad del pasto llanero

En trópico de México las praderas con pastos constituyen la base de la alimentación de los rumiantes. El objetivo del estudio fue determinar la densidad, tasa de aparición, muerte y la sobrevivencia de la población de tallos del pasto Llanero (Andropogon gayanus Kunt) cosechado a diferentes edades de rebrote (ER: 7,14, 21, 28, 35, 42, 49 y 56 d), en dos periodos de crecimiento, época de sequía con riego y época de lluvias. Las ER (tratamientos) se asignaron a cuatro unidades experimentales de 3 m2, mediante un diseño de bloques completos al azar, con cuatro repeticiones. En ambos periodos de crecimiento, la densidad de tallos se incrementó hasta los 28 días de rebrote posteriormente, disminuyó hasta el final del estudio con mayor intensidad durante el periodo de sequía con riego respecto al de lluvias (325 y 378 tallos m-2, respectivamente). Hubo mayor aparición y muerte de tallos en el periodo de sequía con riego (97 a 100 tallos semana-1). El índice de estabilidad de la población fue mayor o próximo de uno durante la sequía con riego, mientras que, en lluvias a partir de los 42 días de rebrote, este fue menor que uno. En conclusión, se obtuvo que edades de rebrote de 7 a 28 días no se compromete la estabilidad poblacional de tallos y favorece la persistencia de las praderas. Mayores edades de rebrote durante las lluvias no permiten recuperar la población de tallos, lo que propicia inestabilidad de la población.

Tillering and characterisation of tillers on marandu palisadegrass deferred and fertilised with nitrogen

Evaluations of tillering dynamics and characteristics of the individual tillers allow the changes that occur in the growth and structure of the deferred and nitrogen (N) fertilised pasture to be understood. This work was carried out to evaluate the tillering and morphological characteristics of basal tillers in deferred canopies of Brachiaria brizantha syn. Urochloa brizantha cv. Marandu (marandu palisadegrass) submitted to two N doses (50 and 200 kg ha-1). The evaluations were carried out at the beginning, middle and end of the deferment period. The experimental design was completely randomised in split-plot design with four replications. Tiller appearance rate (TAR) was higher at the beginning than in the middle and at the end of the deferment period, contrary to that observed for the number and weight of the tiller. Tiller mortality rate (TMR) was higher at the end than at the beginning and in the middle of the deferment period. The tiller population stability index (SI) was similar at the beginning and middle, but lower at the end of the deferment period. The leaf area of the tiller was higher in the middle compared to at the beginning and end of deferment. N dose did not influence TAR, TMR and SI. The canopy fertilised with 200 kg ha-1 of N presented a higher number and weight of tiller compared to the canopy fertilised with 50 kg ha-1 of N. The N increases the number and weight of tillers, but does not change tillering dynamics during the deferment period for marandu palisadegrass. The tillering decreases and the morphological composition of the tiller worsens with the increase in the deferment period in marandu palisadegrass.

Statistical properties of the population stability index

The population stability index (PSI) is a widely used statistic that measures how much a variable has shifted over time. A high PSI may alert the business to a change in the characteristics of a population. This shift may require investigation and possibly a model update. PSI is commonly used among banks to measure the shift between model development data and current data. Banks may face additional risks if models are used without proper validation. The incorrect use of PSI may bring unexpected risks for these institutions. However, there are not many studies about the statistical properties of PSI. In practice, the following “rule of thumb” is used: PSI < 0.10 means a “little change”, 0.10 < PSI < 0.25 means a “moderate change”, and PSI > 0.25 means a “significant change, action required”. These benchmarks are used without reference to statistical type I or type II error rates. This paper aims to fill the gap by providing statistical properties of the PSI and some recommendations on its use.

The Population Accuracy Index: A New Measure of Population Stability for Model Monitoring

Risk models developed on one dataset are often applied to new data and, in such cases, it is prudent to check that the model is suitable for the new data. An important application is in the banking industry, where statistical models are applied to loans to determine provisions and capital requirements. These models are developed on historical data, and regulations require their monitoring to ensure they remain valid on current portfolios—often years since the models were developed. The Population Stability Index (PSI) is an industry standard to measure whether the distribution of the current data has shifted significantly from the distribution of data used to develop the model. This paper explores several disadvantages of the PSI and proposes the Prediction Accuracy Index (PAI) as an alternative. The superior properties and interpretation of the PAI are discussed and it is concluded that the PAI can more accurately summarise the level of population stability, helping risk analysts and managers determine whether the model remains fit-for-purpose.

CHANGES IN TILLERING DYNAMICS OF INTERCROPPED BLACK OAT AND ANNUAL RYEGRASS ENSURE A STABLE SWARD

SUMMARYMixing species with different tillering peaks may enhance pasture stability, but intercropping may also alter the plants’ tillering patterns. This study aimed to evaluate tillering dynamics in black oat (Avena strigosa Schreb.) and annual ryegrass (Lolium multiflorum Lam.) pastures grown as monocultures or intercropped. Three following treatments are established: black oat and annual ryegrass pastures grown as monocultures and an intercrop composed of these two species. Tillering dynamics were measured for black oat and annual ryegrass populations separately. When intercropped, tiller birth rates of black oat decreased (20.0 vs. 28.9 tillers 100 tillers−1) and those of annual ryegrass increased (30.5 vs. 14.3 tillers 100 tillers−1), compared to their monocultures. Tiller death rates for annual ryegrass did not differ between monoculture and intercropping (23.9 tillers 100 tillers−1), but black oat presented higher mortality in monocultures (48.8 vs. 36.4 tillers 100 tillers−1). The black oat monoculture had the lowest population stability index (0.80), whereas annual ryegrass in monoculture and intercropped pastures exhibited greater values (on average, 0.92). Our results indicated that black oat and annual ryegrass present distinct tillering dynamics whether grown as monoculture or intercropped, and suggest that intercropping species with elevated death rates (black oat) with later species (annual ryegrass) could be an important tool for maintaining pasture stability throughout the growing season.

Tillering dynamics of Mulato grass subjected to strategies of rotational grazing management

SUMMARYPlants’ ability to rebuild their tiller population is affected by weather changes and management strategies. The hypothesis of the present study was that frequency and severity of defoliation alter Mulato grass (Brachiaria ruziziensis × Brachiaria brizantha cv. Marandu) sward development, and the proportion of aerial and basal tillers, interfering with sward stability. The objective was to evaluate aerial, basal and total tiller population density and changes in tillering dynamics. Treatments corresponded to strategies of rotational grazing characterized by combinations between two pre-grazing (95% and maximum canopy light interception during regrowth; LI95% and LIMax) and two post-grazing conditions (15 and 20 cm stubble height). The experimental period comprised four seasons of the year: summer 2008 (February–March 2008); autumn–early spring (April to mid-November 2008); late spring (mid-November–December 2008); and summer 2009 (January–March 2009). Density of aerial tillers decreased from summer 2008 to late spring. Tiller death decreased from summer 2008 to autumn–early spring. Aerial and total tiller death increased from late spring to summer 2009, mainly for the LI95% treatment. No differences were observed in the tiller population stability index of aerial tillers for LI treatments or post-grazing heights. The balance between tiller appearance and death for basal tillers remained relatively stable from summer 2008 to late spring. For aerial tillers the balance decreased to negative values from summer 2008 to autumn–early spring for both LI treatments. For total tiller population, the balance decreased to negative values from summer 2008 to autumn–early spring. Results indicated a strong seasonal effect on Mulato grass tillering and growth. Tillering dynamics were affected primarily by grazing frequency, which changed the relative importance of basal and aerial tillers. The results suggested that basal tillering was the predominant perennation pathway.

Grazing management and tussock distribution in elephant grass

AbstractSoil occupation capacity via lateral expansion of tussocks in elephant grass (Pennisetum purpureum Schum.) may be associated with basal tillering. As grazing management alters the proportion of basal and aerial tillers in a tiller population, the hypothesis of this work was that grazing management affects tussock size and distribution with implications for plant population stability. The objective of this study was to evaluate the tiller population stability index, the proportion of basal and aerial tillers, tussock size, and the frequency of tussocks and bare ground in rotationally managed elephant grass cv. Napier. Treatments resulted from the combination of two post‐grazing heights (35 and 45 cm) and two pre‐grazing conditions (95% and maximum canopy light interception during regrowth – LI0·95 and LIMax) and were allocated to experimental units (850 m2 paddocks) according to a 2 × 2 factorial arrangement in a randomized complete block design, with four replications. Measurements were taken from January 2011 to April 2012. The post‐grazing height treatments affected the tiller population stability index, but did not influence the pattern of tussock distribution. On the other hand, the different grazing frequencies (targets of LI pre‐grazing) altered the pattern of tussock distribution and the proportion of bare ground. In general, the tiller population stability index and frequency of tussocks were higher and the frequency of bare ground lower on swards managed with the LI0·95 target relative to those managed with the LIMax target, regardless of the post‐grazing height used, indicating a larger soil occupation capacity of plants under the more frequent defoliation regime. Such responses were associated with larger population of basal tillers and highlight the importance of tiller category and perennation pathway in defining patterns of plant growth and tussock distribution.

&lt;b&gt;Tiller dynamics of ryegrass managed under two stocking rates

This study investigated the tiller dynamics of Italian ryegrass (Lolium multiflorum Lam) under low and high stocking rates. These rates were determined by heifers exclusively on pasture or grazing and supplemented with oats and corn grain. The experimental design was completely randomized with repeated measurements over time, with two stocking rates, three and six replications of area for low and high stocking rates, respectively. The appearance rate (1.0 tillers tiller -1 m -2 ), survival rate (0.8 tillers tiller -1 m -2 ), population stability index (1.6) and site occupation (0.3) were similar for high and low stocking rates. The tiller density was similar for different stocking rates. The first generation of tillers was 56% of the tiller population at end of the ryegrass cycle. The increase by 22.5% in the stocking rate derived from the use of supplements for grazing heifers did not alter the tiller dynamics of ryegrass.