GitHub Projects Research Articles

Foundation models in robotics: Applications, challenges, and the future

We survey applications of pretrained foundation models in robotics. Traditional deep learning models in robotics are trained on small datasets tailored for specific tasks, which limits their adaptability across diverse applications. In contrast, foundation models pretrained on internet-scale data appear to have superior generalization capabilities, and in some instances display an emergent ability to find zero-shot solutions to problems that are not present in the training data. Foundation models may hold the potential to enhance various components of the robot autonomy stack, from perception to decision-making and control. For example, large language models can generate code or provide common sense reasoning, while vision-language models enable open-vocabulary visual recognition. However, significant open research challenges remain, particularly around the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution. In this survey, we study recent papers that have used or built foundation models to solve robotics problems. We explore how foundation models contribute to improving robot capabilities in the domains of perception, decision-making, and control. We discuss the challenges hindering the adoption of foundation models in robot autonomy and provide opportunities and potential pathways for future advancements. The GitHub project corresponding to this paper can be found here: https://github.com/robotics-survey/Awesome-Robotics-Foundation-Models .

Architecture decisions in quantum software systems: An empirical study on Stack Exchange and GitHub

Context:Quantum computing provides a new dimension in computation, utilizing the principles of quantum mechanics to potentially solve complex problems that are currently intractable for classical computers. However, little research has been conducted about the architecture decisions made in quantum software development, which have a significant influence on the functionality, performance, scalability, and reliability of these systems. Objective:The study aims to empirically investigate and analyze architecture decisions made during the development of quantum software systems, identifying prevalent challenges and limitations by using the posts and issues from Stack Exchange and GitHub. Methods:We used a qualitative approach to analyze the obtained data from Stack Exchange Sites and GitHub projects — two prominent platforms in the software development community. Specifically, we collected data from 385 issues (from 87 GitHub projects) and 70 posts (from 3 Stack Exchange sites) related to architecture decisions in quantum software development. Results:The results show that in quantum software development (1) architecture decisions are articulated in six linguistic patterns, the most common of which are Solution Proposal and Information Giving, (2) the two major categories of architectural decisions are Implementation Decision and Technology Decision, (3) Software Development Tools are the most common application domain among the twenty application domains identified, (4) Maintainability is the most frequently considered quality attribute, and (5) Design Issues and High Error Rates are the major limitations and challenges that practitioners face when making architecture decisions in quantum software development. Conclusions:Our results show that the limitations and challenges encountered in architecture decision-making during the development of quantum software systems are strongly linked to the particular features (e.g., quantum entanglement, superposition, and decoherence) of those systems. These issues mostly pertain to technical aspects and need appropriate measures to address them effectively.

Describing and Sharing Molecular Visualizations Using the MolViewSpec Toolkit.

With the ever-expanding toolkit of molecular viewers, the ability to visualize macromolecular structures has never been more accessible. Yet, the idiosyncratic technical intricacies across tools and the integration complexities associated with handling structure annotation data present significant barriers to seamless interoperability and steep learning curves for many users. The necessity for reproducible data visualizations is at the forefront of the current challenges. Recently, we introduced MolViewSpec (homepage: https://molstar.org/mol-view-spec/, GitHub project: https://github.com/molstar/mol-view-spec), a specification approach that defines molecular visualizations, decoupling them from the varying implementation details of different molecular viewers. Through the protocols presented herein, we demonstrate how to use MolViewSpec and its 3D view-building Python library for creating sophisticated, customized 3D views covering all standard molecular visualizations. MolViewSpec supports representations like cartoon and ball-and-stick with coloring, labeling, and applying complex transformations such as superposition to any macromolecular structure file in mmCIF, BinaryCIF, and PDB formats. These examples showcase progress towards reusability and interoperability of molecular 3D visualization in an era when handling molecular structures at scale is a timely and pressing matter in structural bioinformatics as well as research and education across the life sciences. © 2024 The Authors. Current Protocols published by Wiley Periodicals LLC. Basic Protocol 1: Creating a MolViewSpec view using the MolViewSpec Python package Basic Protocol 2: Creating a MolViewSpec view with reference to MolViewSpec annotation files Basic Protocol 3: Creating a MolViewSpec view with labels and other advanced features Support Protocol 1: Computing rotation and translation vectors Support Protocol 2: Creating a MolViewSpec annotation file.

Deep semi-supervised learning for recovering traceability links between issues and commits

Traceability links between issues and commits record valuable information about the evolutionary history of software projects. Unfortunately, these links are often missing. While deep learning stands as the current state-of-the-art (SOTA) in automated traceability links recovery (TLR), its effectiveness is faced with the practical problem of limited labeled data during training. To overcome this challenge, in this paper, we propose DSSLink, a novel method based on deep semi-supervised learning, enhancing deep-learning-based link recovery tasks. DSSLink first learns knowledge from labeled data through pre-trained model and then leverages deep semi-supervised learning to infer pseudo-labels on unlabeled data. The extended dataset of pseudo-labeled and labeled data re-trains the deep learning model in an iterative process. Our extensive evaluations are conducted on two SOTA traceability methods (T-BERT and BTLink) across four GitHub projects and 11 Apache projects. Specifically, the maximum F1-score improvements for GitHub and Apache projects reached 22.9% and 43.5%, respectively. Evaluation results show that DSSLink is effective in enhancing TLR performance and outperforms TraceFUN, a recent approach that utilizes unlabeled data for TLR. The source code of DSSLink is available at https://github.com/DSSLink.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

DaGAN++: Depth-Aware Generative Adversarial Network for Talking Head Video Generation.

Predominant techniques on talking head generation largely depend on 2D information, including facial appearances and motions from input face images. Nevertheless, dense 3D facial geometry, such as pixel-wise depth, plays a critical role in constructing accurate 3D facial structures and suppressing complex background noises for generation. However, dense 3D annotations for facial videos is prohibitively costly to obtain. In this paper, first, we present a novel self-supervised method for learning dense 3D facial geometry (i.e., depth) from face videos, without requiring camera parameters and 3D geometry annotations in training. We further propose a strategy to learn pixel-level uncertainties to perceive more reliable rigid-motion pixels for geometry learning. Second, we design an effective geometry-guided facial keypoint estimation module, providing accurate keypoints for generating motion fields. Lastly, we develop a 3D-aware cross-modal (i.e., appearance and depth) attention mechanism, which can be applied to each generation layer, to capture facial geometries in a coarse-to-fine manner. Extensive experiments are conducted on three challenging benchmarks (i.e., VoxCeleb1, VoxCeleb2, and HDTF). The results demonstrate that our proposed framework can generate highly realistic-looking reenacted talking videos, with new state-of-the-art performances established on these benchmarks.

How Are Multilingual Systems Constructed: Characterizing Language Use and Selection in Open-Source Multilingual Software

For many years now, modern software is known to be developed in multiple languages (hence termed as multilingual or multi-language software). Yet, to date, we still only have very limited knowledge about how multilingual software systems are constructed. For instance, it is not yet really clear how different languages are used, selected together, and why they have been so in multilingual software development. Given the fact that using multiple languages in a single software project has become a norm, understanding language use and selection (i.e., language profile ) as a basic element of the multilingual construction in contemporary software engineering is an essential first step. In this article, we set out to fill this gap with a large-scale characterization study on language use and selection in open-source multilingual software. We start with presenting an updated overview of language use in 7,113 GitHub projects spanning the 5 past years by characterizing overall statistics of language profiles, followed by a deeper look into the functionality relevance/justification of language selection in these projects through association rule mining. We proceed with an evolutionary characterization of 1,000 GitHub projects for each of the 10 past years to provide a longitudinal view of how language use and selection have changed over the years, as well as how the association between functionality and language selection has been evolving. Among many other findings, our study revealed a growing trend of using three to five languages in one multilingual software project and the noticeable stableness of top language selections. We found a non-trivial association between language selection and certain functionality domains, which was less stable than that with individual languages over time. In a historical context, we also have observed major shifts in these characteristics of multilingual systems both in contrast to earlier peer studies and along the evolutionary timeline. Our findings offer essential knowledge on the multilingual construction in modern software development. Based on our results, we also provide insights and actionable suggestions for both researchers and developers of multilingual systems.

Planktonic, benthic and sympagic copepods collected from the desalination unit of Mario Zucchelli Research Station in Terra Nova Bay (Ross Sea, Antarctica).

Distributional data on planktonic, benthic and sympagic copepods collected in the framework of the XXXIVth Expeditions of the Italian National Antarctic Programme (PNRA) to the Ross Sea sector from 2018-2019 are here provided. These occurrences correspond to specimens collected from the 25 μm filters used in the desalination plant of the Italian research station "Mario Zucchelli" (MZS), located in the Terra Nova Bay area (TNB; Ross Sea, Antarctica). This dataset is a contribution to the Antarctic Biodiversity Portal, the thematic Antarctic node for both the Ocean Biogeographic Information System (AntOBIS) and the Global Biodiversity Information Facility Antarctic Biodiversity Information Facility (ANTABIF). The dataset was uploaded and integrated with the SCAR-AntOBIS database (the geospatial component of SCAR-MarBIN). Please follow the guidelines from the SCAR Data Policy (ISSN 1998-0337) when using the data. If you have any questions regarding this dataset, please contact us via the contact information provided in the metadata or via data-biodiversity-aq@naturalsciences.be. Issues with the dataset can be reported at the biodiversity-aq GitHub project. We describe the diversity of marine copepods Terra Nova Bay sampled by the filters installed in the desalination unit (DU) of the Italian research station "Mario Zucchelli" described in recent work. The opening of the intake pipe of the DU is positioned at a depth of 4 m and allowed a total of 2,116 specimens to be sampled and recognised. In addition, new occurrence records of copepod genera and species are reported in the same zone. We provide an overview of the marine copepod diversity reported for TNB. The total of 2,116 individuals corresponds to 14 genera and 15 species and is represented by 136 occurrence records in this dataset. Around 52% of the total number of species are new records for the TNB area. The publication of this data paper was funded by the Belgian Science Policy Office (BELSPO, contract n°FR/36/AN1/AntaBIS) in the Framework of EU-Lifewatch as a contribution to the SCAR Antarctic biodiversity portal (biodiversity.aq).

SafeNet: Towards mitigating replaceable unsafe Rust code via a recommendation‐based approach

AbstractRust is a system‐level programming language with advantages in memory safety. It ensures that any Rust programs without unsafe code should not incur undefined behaviours. However, unsafe code still plays an essential role in Rust to achieve low‐level control. Therefore, a major design pattern of Rust programs is interior unsafe, which wraps unsafe code as safe APIs and handles all undefined behaviours internally. Rust standard library already provides a rich set of safe APIs to facilitate Rust code development. Nevertheless, due to unfamiliarity with these APIs, developers may misuse unnecessary unsafe code and suffer memory‐safety risks. In this paper, we investigate an approach to mitigate replaceable unsafe code. We first analyse unsafe APIs of the Rust standard library and summarize their common usage patterns. Each pattern corresponds to one or several code samples in our knowledge base. Then, we develop an approach to automatically recognize the usage pattern and recommend corresponding code samples. Our approach leverages dataflow analysis to exclude impossible patterns and employs a BERT‐based machine learning model to find the most similar pattern among the rest. We have conducted evaluation experiments with 472 unsafe code snippets collected from GitHub projects and successfully recognized the pattern of 394 snippets. We hope our approach can assist developers in detecting unnecessary unsafe code and suggesting safe alternatives.

Rapid: Zero-shot Domain Adaptation for Code Search with Pre-trained Models

Code search, which refers to the process of identifying the most relevant code snippets for a given natural language query, plays a crucial role in software maintenance. However, current approaches heavily rely on labeled data for training, which results in performance decreases when confronted with cross-domain scenarios including domain-specific or project-specific situations. This decline can be attributed to their limited ability to effectively capture the semantics associated with such scenarios. To tackle the aforementioned problem, we propose a ze R o-shot dom A in ada P tion with pre-tra I ned mo D els framework for code search named RAPID. The framework first generates synthetic data by pseudo labeling, then trains the CodeBERT with sampled synthetic data. To avoid the influence of noisy synthetic data and enhance the model performance, we propose a mixture sampling strategy to obtain hard negative samples during training. Specifically, the mixture sampling strategy considers both relevancy and diversity to select the data that are hard to be distinguished by the models. To validate the effectiveness of our approach in zero-shot settings, we conduct extensive experiments and find that RAPID outperforms the CoCoSoDa and UniXcoder model by an average of 15.7% and 10%, respectively, as measured by the MRR metric. When trained on full data, our approach results in an average improvement of 7.5% under the MRR metric using CodeBERT. We observe that as the model’s performance in zero-shot tasks improves, the impact of hard negatives diminishes. Our observation also indicates that fine-tuning CodeT5 for generating pseudo labels can enhance the performance of the code search model, and using only 100-shot samples can yield comparable results to the supervised baseline. Furthermore, we evaluate the effectiveness of RAPID in real-world code search tasks in three GitHub projects through both human and automated assessments. Our findings reveal RAPID exhibits superior performance, e.g., an average improvement of 18% under the MRR metric over the top-performing model.

A Comparative Analysis of Centralized and Decentralized Developer Autonomous Organizations Managing Conflicts in Discussing External Crises

Understanding whether and how online developer communities resolve conflicts that occur in discussions is critical. Few studies focused on the conflicts caused by external crises, such as geopolitical events (e.g., the 2022 Russo-Ukrainian Crisis). We comparatively studied how a decentralized autonomous organization (DAO), Aave project community, and a centralized autonomous organization (CAO), GitHub project community, managed external crises caused by conflicts. Our mixed-method analysis showed that a DAO could be better than a CAO for mitigating conflicts. And blockchain technologies (i.e., voting and cryptocurrency) played vital roles. To address the low voter turnout, we proposed adding a monetary incentive to engage more DAO members in forming common goals.

Webometric analysis of alternative access to scholarly publication

Purpose: This study aims to uncover the worldwide usage patterns and growth trends of Library Genesis (LibGen) and Sci-Hub, two popular alternative access platforms to scholarly publications. Design/methodology/approach: This study utilizes a webometric approach to analyze worldwide usage patterns and growth trends of LibGen and Sci-Hub. Data was collected between May and June 2023 on online presence, usage metrics, and pattern from different web-based tools like LibGen and Sci-Hub database, Google Trends, SimilarWeb, and existing Python and GitHub projects. Analyses are conducted by incorporating statistical techniques and previous literature reviews. Findings: The study reveals that LibGen and Sci-Hub have experienced a notable surge in popularity despite facing copyright infringements, legal disputes, and restrictions. Conversion rates and unique visitors have significantly increased, with users from various nations, including developed countries like the USA and China. Statistical data shows a preference for accessing science and technology-related resources, particularly in the field of medicine. Most of the downloads originated from reputable publishers and academic journals. The main motivations behind using these platforms are the high costs and limited access to scholarly publications. This phenomenon has attracted researchers, academics, students, and information seekers globally as they seek to overcome financial constraints and institutional barriers hindering access to valuable knowledge. Research limitations: The study acknowledges limitations related to potential discrepancies and biases from external software and incomplete representation of source databases. The dynamic nature of the platforms also means data is only up until 2022, potentially excluding recent developments. Additionally, the country specific data may not give the exact idea as people use Virtual Private Network (VPN) to access the platforms bypassing restrictions. Practical implications: The findings shed light on the challenges faced by the academic community and offer potential implications for policymakers, publishers, and researchers aiming to address the growing demand for affordable and accessible scholarly publications. Originality/value: The originality of this study lies in its application to examine the alternative shadow libraries for scholarly publications. This research contributes to the existing knowledge domain by offering valuable insights into the extent and trends of utilizing shadow platforms, focusing on the implications of copyright infringements, legal concerns, and restrictions on scholarly knowledge dissemination.

Class correlation correction for unbiased scene graph generation

The long-tail distribution in the scene graph generation (SGG) task has spurred immense interest in unbiased SGG. However, current state-of-the-art debiasing techniques extract statistics from the independent category, while ignoring the correlation between categories. To address this issue, we propose a simple but effective method, class correlation correction, to aggregate dependency knowledge among various classes. Specifically, given biased predictions, two kinds of debiasing transformations are developed employing the class correlation aware label to recover the unbiased estimates. We also propose to retrain SGG models with the biasing transformations adapted to the biased data distribution. The proposed debiasing method is evaluated using several biased datasets that are constructed from CIFAR-10 and Fashion-MNIST, as well as the commonly used SGG dataset and Caltech 101 dataset. Multiple evaluation metrics are used to assess the debiasing performance, and the results of extensive experiments show the effectiveness of the proposed method. The Pytorch® implementations can be downloaded from an open-source GitHub project https://github.com/Dlut-lab-zmn/class-correlation-correction.

Detecting outdated code element references in software repository documentation

Outdated documentation is a pervasive problem in software development, preventing effective use of software, and misleading users and developers alike. We posit that one possible reason why documentation becomes out of sync so easily is that developers are unaware of when their source code modifications render the documentation obsolete. Ensuring that the documentation is always in sync with the source code takes considerable effort, especially for large codebases. To address this situation, we propose an approach that can automatically detect code element references that survive in the documentation after all source code instances have been deleted. In this work, we analysed over 3,000 GitHub projects and found that most projects contain at least one outdated code element reference at some point in their history. We submitted GitHub issues to real-world projects containing outdated references detected by our approach, some of which have already led to documentation fixes. As an initiative toward keeping documentation in software repositories up-to-date, we have made our implementation available for developers to scan their GitHub projects for outdated code element references.

Data pipeline quality: Influencing factors, root causes of data-related issues, and processing problem areas for developers

Data pipelines are an integral part of various modern data-driven systems. However, despite their importance, they are often unreliable and deliver poor-quality data. A critical step toward improving this situation is a solid understanding of the aspects contributing to the quality of data pipelines. Therefore, this article first introduces a taxonomy of 41 factors that influence the ability of data pipelines to provide quality data. The taxonomy is based on a multivocal literature review and validated by eight interviews with experts from the data engineering domain. Data, infrastructure, life cycle management, development & deployment, and processing were found to be the main influencing themes. Second, we investigate the root causes of data-related issues, their location in data pipelines, and the main topics of data pipeline processing issues for developers by mining GitHub projects and Stack Overflow posts. We found data-related issues to be primarily caused by incorrect data types (33%), mainly occurring in the data cleaning stage of pipelines (35%). Data integration and ingestion tasks were found to be the most asked topics of developers, accounting for nearly half (47%) of all questions. Compatibility issues were found to be a separate problem area in addition to issues corresponding to the usual data pipeline processing areas (i.e., data loading, ingestion, integration, cleaning, and transformation). These findings suggest that future research efforts should focus on analyzing compatibility and data type issues in more depth and assisting developers in data integration and ingestion tasks. The proposed taxonomy is valuable to practitioners in the context of quality assurance activities and fosters future research into data pipeline quality.

On the validity of retrospective predictive performance evaluation procedures in just-in-time software defect prediction

Just-In-Time Software Defect Prediction (JIT-SDP) is concerned with predicting whether software changes are defect-inducing or clean. It operates in scenarios where labels of software changes arrive over time with delay, which in part corresponds to the time we wait to label software changes as clean (waiting time). However, clean labels decided based on waiting time may be different from the true labels of software changes, i.e., there may be label noise. This typically overlooked issue has recently been shown to affect the validity of continuous performance evaluation procedures used to monitor the predictive performance of JIT-SDP models during the software development process. It is still unknown whether this issue could potentially also affect evaluation procedures that rely on retrospective collection of software changes such as those adopted in JIT-SDP research studies, affecting the validity of the conclusions of a large body of existing work. We conduct the first investigation of the extent with which the choice of waiting time and its corresponding label noise would affect the validity of retrospective performance evaluation procedures. Based on 13 GitHub projects, we found that the choice of waiting time did not have a significant impact on the validity and that even small waiting times resulted in high validity. Therefore, (1) the estimated predictive performances in JIT-SDP studies are likely reliable in view of different waiting times, and (2) future studies can make use of not only larger (5k+ software changes), but also smaller (1k software changes) projects for evaluating performance of JIT-SDP models.

Study the correlation between the readme file of GitHub projects and their popularity

A readme file plays an important role in a GitHub repository to provide a starting point for developers to reuse and make contributions. A good readme could provide sufficient information for users to learn and start a GitHub repository and might be correlated to the popularity of a repository. Given the importance of the role that a readme file plays, we aim to study to understand the correlation between the readme file of GitHub repositories and their popularity. We analyze readme files of 5,000 GitHub repositories across more than 20 languages. We study the relationship between readme file related factors and the popularity of GitHub repositories. We observe that: (1) Most of the studied readme file related factors (e.g., the number of lists, the number and frequency of updates on the readme file) are statistically significantly different between popular and non-popular repositories with non-negligible effect size. (2) After controlling repository-specific factors (e.g., repository topics and license information), the number of lists and the frequency of updates are the most significantly important factors that discriminate between popular and non-popular repositories. (3) The most of updates were made to update references in popular repositories, while in non-popular repositories most updates are for the content of how to use the repository.Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Protecting by attacking: A personal information protecting method with cross-modal adversarial examples

Recent years’ development of AI technology brings more convenience to our life while at the same time increasing the risk of personal information leakage. In this work, we try to protect personal information contained in the images by generating adversarial examples to fool the image captioning models. The generated adversarial examples are user-oriented which means the users can manipulate or hide sensitive information on the text output as they wish. By doing so, our personal information can be well protected from image captioning models. To fulfill the task, we adopt five kinds of adversarial attack. Experimental results show our method can successfully protect user security. The Pytorch® implementations can be downloaded from an open-source GitHub project (https://github.com/Dlut-lab-zmn/Image-Captioning-Attack/).

A Procedure to Continuously Evaluate Predictive Performance of Just-In-Time Software Defect Prediction Models During Software Development

Just-In-Time Software Defect Prediction (JIT-SDP) uses machine learning to predict whether software changes are defect-inducing or clean. When adopting JIT-SDP, changes in the underlying defect generating process may significantly affect the predictive performance of JIT-SDP models over time. Therefore, being able to continuously track the predictive performance of JIT-SDP models during the software development process is of utmost importance for software companies to decide whether or not to trust the predictions provided by such models over time. However, there has been little discussion on how to continuously evaluate predictive performance in practice, and such evaluation is not straightforward. In particular, labeled software changes that can be used for evaluation arrive over time with a delay, which in part corresponds to the time we have to wait to label software changes as ‘clean’ (waiting time). A clean label assigned based on a given waiting time may not correspond to the true label of the software changes. This can potentially hinder the validity of any continuous predictive performance evaluation procedure for JIT-SDP models. This paper provides the first discussion of how to continuously evaluate predictive performance of JIT-SDP models over time during the software development process, and the first investigation of whether and to what extent waiting time affects the validity of such continuous performance evaluation procedure in JIT-SDP. Based on 13 GitHub projects, we found that waiting time had a significant impact on the validity. Though typically small, the differences in estimated predicted performance were sometimes large, and thus inappropriate choices of waiting time can lead to misleading estimations of predictive performance over time. Such impact did not normally change the ranking between JIT-SDP models, and thus conclusions in terms of which JIT-SDP model performs better are likely reliable independent of the choice of waiting time, especially when considered across projects.

On Wasted Contributions: Understanding the Dynamics of Contributor-Abandoned Pull Requests–A Mixed-Methods Study of 10 Large Open-Source Projects

Pull-based development has enabled numerous volunteers to contribute to open-source projects with fewer barriers. Nevertheless, a considerable amount of pull requests (PRs) with valid contributions are abandoned by their contributors , wasting the effort and time put in by both the contributors and maintainers. To better understand the underlying dynamics of contributor-abandoned PRs, we conduct a mixed-methods study using both quantitative and qualitative methods. We curate a dataset consisting of 265,325 PRs including 4,450 abandoned ones from ten popular and mature GitHub projects and measure 16 features characterizing PRs, contributors, review processes, and projects. Using statistical and machine learning techniques, we find that complex PRs, novice contributors, and lengthy reviews have a higher probability of abandonment and the rate of PR abandonment fluctuates alongside the projects’ maturity or workload. To identify why contributors abandon their PRs, we also manually examine a random sample of 354 abandoned PRs. We observe that the most frequent abandonment reasons are related to the obstacles faced by contributors, followed by the hurdles imposed by maintainers during the review process. Finally, we survey the top core maintainers of the studied projects to understand their perspectives on dealing with PR abandonment and on our findings.

A Proposed Simulation Technique for Population Stability Testing in Credit Risk Scorecards

Credit risk scorecards are logistic regression models, fitted to large and complex data sets, employed by the financial industry to model the probability of default of potential customers. In order to ensure that a scorecard remains a representative model of the population, one tests the hypothesis of population stability; specifying that the distribution of customers’ attributes remains constant over time. Simulating realistic data sets for this purpose is nontrivial, as these data sets are multivariate and contain intricate dependencies. The simulation of these data sets are of practical interest for both practitioners and for researchers; practitioners may wish to consider the effect that a specified change in the properties of the data has on the scorecard and its usefulness from a business perspective, while researchers may wish to test a newly developed technique in credit scoring. We propose a simulation technique based on the specification of bad ratios, this is explained below. Practitioners can generally not be expected to provide realistic parameter values for a scorecard; these models are simply too complex and contain too many parameters to make such a specification viable. However, practitioners can often confidently specify the bad ratio associated with two different levels of a specific attribute. That is, practitioners are often comfortable with making statements such as “on average a new customer is 1.5 times as likely to default as an existing customer with similar attributes”. We propose a method which can be used to obtain parameter values for a scorecard based on specified bad ratios. The proposed technique is demonstrated using a realistic example, and we show that the simulated data sets adhere closely to the specified bad ratios. The paper provides a link to a Github project with the R code used to generate the results.