Full Reproducibility Research Articles

Feather keratin in Pavo cristatus: A tentative structure

Abstract Background F-keratin forms an evolutionary conserved nanocomposite which allows birds to fly. Structural models for F-keratin are all based on the pioneering X-ray diffraction studies on seagull F-keratin, first published in 1932, confirmed for other species and refined over the years. There is, however, no experimental proof because native F-keratin does not form a perfect molecular crystal as required for structure determination. Methods Peacock’s tail feathers were systematically re-investigated by taking diffraction patterns at different rotation angles. Using the recently developed AlphaFold algorithm, a collection of 3D models of arbitrarily truncated and multiplied Pavo cristatus F-keratin sequences was created. The shape, dimensions, density and interfacial exposure of functionally relevant amino acid side chains of the calculated 3D building blocks were used as the initial selection criteria for filamentous F-keratin precursors. Full reproducibility of in silico folding and agreement with previous results from mechanical testing, biochemical analyses and SAXS experiments was mandatory for suggesting the tentative structure for the novel F-keratin repeating unit. Results The filament of the F-keratin polymer is an alternating arrangement of two units called "N-block" and "C-block": Four strands AA 1–52 form a disulfide-stabilized twisted parallelepiped, with 89° internal rotation within eight levels of β-sandwiches. Four strands AA 81–100 form a two-level “β-sandwich” in which aromatic residues provide resilience, like vertebral discs in a spinal column. The pitch of an N+C-block octamer is 10 nm. Solidification may involve "C-blocks" to temporarily mold into "C-wedges" of 18° tilt, which align F-keratin into laterally amorphous fiber-reinforced composites of 9.5 nm axial periodicity. This experimentally significant distance corresponds to the fully stretched AA 53–80 matrix segment. The deformed “spinal column” unwinds under compression when F-keratin filaments perfectly align horizontally into stacked sheets in the solid state. Conclusions At present, the tentative structure presented here is without alternatives.

Recommended Minimum Elements for Transparent Reporting of Multi-Jurisdiction Algorithm Feasibility Studies

BackgroundResearch and surveillance using routinely collected health data rely on algorithms to ascertain disease cases or health measures. Where algorithm validation studies are not possible due to lack of a reference standard, algorithm feasibility studies can be used to create and assess algorithms for use in more than one population or jurisdiction. Publication of the methods used to conduct feasibility studies is critical for transparency and reproducibility. Existing guidelines applicable to feasibility studies, including the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) and REporting of studies Conducted using Observational Routinely collected health Data (RECORD) statements, may benefit from additional elements to capture aspects particular to multi-jurisdiction algorithm feasibility studies and ensure full reproducibility. MethodsA subcommittee of members from Health Data Research Network (HDRN) Canada’s Algorithms and Harmonized Data Working Group (AHD-WG) reviewed items within the STROBE and RECORD guidelines and compared these to published feasibility studies. Items not contained within STROBE or RECORD but recommended to ensure transparent reporting of feasibility studies were identified. The AHD-WG reviewed and approved these additional recommended elements. ResultsEleven additional recommended elements were identified: one element for the title and abstract, one in the introduction, five in the methods, and four in the results sections. Elements primarily addressed reporting jurisdictional data variabilities, data harmonization methods, and algorithm implementation. SignificanceImplementation of these recommended elements, alongside the RECORD guidelines, is intended to encourage consistent publication of methods that support reproducibility, as well as increase comparability and international collaborations.

Scalable Image Clustering to screen for self-produced CSAM

The number of cases involving Child Sexual Abuse Material (CSAM) has increased dramatically in recent years, resulting in significant backlogs. To protect children in the suspect’s sphere of influence, immediate identification of self-produced CSAM among acquired CSAM is paramount. Currently, investigators often rely on an approach based on a simple metadata search. However, this approach faces scalability limitations for large cases and is ineffective against anti-forensic measures. Therefore, to address these problems, we bridge the gap between digital forensics and state-of-the-art data science clustering approaches. Our approach enables clustering of more than 130,000 images, which is eight times larger than previous achievements, using commodity hardware and within an hour with the ability to scale even further. In addition, we evaluate the effectiveness of our approach on seven publicly available forensic image databases, taking into account factors such as anti-forensic measures and social media post-processing. Our results show an excellent median clustering-precision (Homogeinity) of 0.92 on native images and a median clustering-recall (Completeness) of over 0.92 for each test set. Importantly, we provide full reproducibility using only publicly available algorithms, implementations, and image databases.

Using machine learning to count Antarctic shag (Leucocarbo bransfieldensis) nests on images captured by remotely piloted aircraft systems

Using 51 orthomosaics of 11 breeding locations of the Antarctic shag (Leucocarbo bransfieldensis), we propose a method for automating counting of shag nests. This is achieved by training an object detection model based on the “You Only Look Once” (YOLO) architecture and identifying nests on sections of the orthomosaic, which are later combined with predictions for the entire orthomosaic. Our results show that the current use of Remotely Piloted Aircraft Systems (RPAS) to collect images of areas with shag colonies, combined with machine learning algorithms, can provide reliable and fast estimates of shag nest counts (F1 score > 0.95). By using data from only two shag colonies for training, we show that models can be obtained that generalise well to images of both spatially and temporally distinct colonies. The proposed practical application opens the possibility of using aerial imagery to perform large-scale surveys of Antarctic islands in search of undiscovered shag colonies. We discuss the conditions for optimal performance of the model as well as its limitations. The code, data and trained model allowing for full reproducibility of the results are available at https://github.com/Appsilon/Antarctic-nests.

BASE: Brain Age Standardized Evaluation

Brain age, most commonly inferred from T1-weighted magnetic resonance images (T1w MRI), is a robust biomarker of brain health and related diseases. Superior accuracy in brain age prediction, often falling within a 2–3 year range, is achieved predominantly through deep neural networks. However, comparing study results is difficult due to differences in datasets, evaluation methodologies and metrics. Addressing this, we introduce Brain Age Standardized Evaluation (BASE), which includes (i) a standardized T1w MRI dataset including multi-site, new unseen site, test-retest and longitudinal data, and an associated (ii) evaluation protocol, including repeated model training and upon based comprehensive set of performance metrics measuring accuracy, robustness, reproducibility and consistency aspects of brain age predictions, and (iii) statistical evaluation framework based on linear mixed-effects models for rigorous performance assessment and cross-comparison. To showcase BASE, we comprehensively evaluate four deep learning based brain age models, appraising their performance in scenarios that utilize multi-site, test-retest, unseen site, and longitudinal T1w brain MRI datasets. Ensuring full reproducibility and application in future studies, we have made all associated data information and code publicly accessible at https://github.com/AralRalud/BASE.git.

End-to-end deep learning pipeline for on-board extraterrestrial rock segmentation

Bringing autonomy on board edge devices is inevitable to accelerate the process of space exploration. Although there are various tasks that can be executed autonomously by such vehicles, detecting and segmenting rocks in on-board images of extraterrestrial landscapes is a critical step in the processing chain, as it can allow to navigate safely while avoiding collisions. We tackle this issue and introduce an end-to-end pipeline for building and validating resource-frugal machine learning techniques for this task, offering a high level of flexibility. Deploying such models on board edge devices poses numerous practical challenges, spanning across ensuring their memory and computational efficiency, and understanding their robustness against varying quality of acquired images. These aspects are often overlooked while building deep learning-powered on-board systems—we show that they can (and ultimately should) be a part of the deployment chain. Our extensive experimental study performed over several benchmark data sets shed more light on functional and non-functional capabilities of the investigated models, both in full-precision and compressed by quantisation, the latter delivering statistically same segmentation accuracy while being approximately 11× smaller. Additionally, we show that synthesised images can be utilised to quantify the robustness of deep learning models against on-board acquisition conditions which directly affect the quality of captured images—such simulations mimicking real-world acquisition settings could have a negative impact on the capabilities of the models trained over clean and high-quality image data. To ensure full reproducibility of this study, we made our implementation publicly available at https://github.com/danielmarek22/onboard-rock-segmentation.

Back and Forth—On Automatic Exposure of Origin and Dissemination of Files on Windows

The number of Child Sexual Abuse Material (CSAM) cases has increased dramatically in recent years. This leads to the need to automate various steps in digital forensic processing, especially for CSAM investigations. For instance, if CSAM pictures are found on a device, the investigator aim at finding traces about the origin and possible further dissemination, respectively. In this article, we address this challenge with respect to the widespread Windows operating system. We model different common scenarios of system use by CSAM offenders in the scope of file inbound and outbound transfer channels. This gives us insights about digital traces in the Windows operating system and its applications to get knowledge about origin and possible destination of a file. We review available concepts and applications to support this issue. Furthermore, we develop a recursive-based approach and provide a prototype as plugin for the open source application Autopsy. We call our prototype AutoTrack . Our evaluation against the different models of Windows system usage reveals that Autotrack is superior to existing solutions and provides support for an investigator to find digital traces about the origin and possible further dissemination of files. We publish our AutoTrack plugin and thus provide full reproducibility of our approach.

Investigating Self-Rationalizing Models for Commonsense Reasoning

The rise of explainable natural language processing spurred a bulk of work on datasets augmented with human explanations, as well as technical approaches to leverage them. Notably, generative large language models offer new possibilities, as they can output a prediction as well as an explanation in natural language. This work investigates the capabilities of fine-tuned text-to-text transfer Transformer (T5) models for commonsense reasoning and explanation generation. Our experiments suggest that while self-rationalizing models achieve interesting results, a significant gap remains: classifiers consistently outperformed self-rationalizing models, and a substantial fraction of model-generated explanations are not valid. Furthermore, training with expressive free-text explanations substantially altered the inner representation of the model, suggesting that they supplied additional information and may bridge the knowledge gap. Our code is publicly available, and the experiments were run on open-access datasets, hence allowing full reproducibility.

INNterpol: High-precision interpolation of stellar atmospheres with a deep neural network using a 1D convolutional auto encoder for feature extraction

Context. Given the widespread availability of grids of models for stellar atmospheres, it is necessary to recover intermediate atmospheric models by means of accurate techniques that go beyond simple linear interpolation and capture the intricacies of the data.Aims. Our goal is to establish a reliable, precise, lightweight, and fast method for recovering stellar model atmospheres, that is to say the stratification of mass column, temperature, gas pressure, and electronic density with optical depth given any combination of the defining atmospheric specific parameters: metallicity, effective temperature, and surface gravity, as well as the abundances of other key chemical elements.Methods. We employed a fully connected deep neural network which in turn uses a 1D convolutional auto-encoder to extract the nonlinearities of a grid using the ATLAS9 and MARCS model atmospheres.Results. This new method we call iNNterpol effectively takes into account the nonlinearities in the relationships of the data as opposed to traditional machine-learning methods, such as the light gradient boosting method (LightGBM), that are repeatedly used for their speed in well-known competitions with reduced datasets. We show a higher precision with a convolutional auto-encoder than using principal component analysis as a feature extractor. We believe it constitutes a useful tool for generating fast and precise stellar model atmospheres, mitigating convergence issues, as well as a framework for future developments. The code and data for both training and direct interpolation are available online for full reproducibility and to serve as a practical starting point for other continuous 1D data in the field and elsewhere.

An advanced method to streamline p-hacking

Recent research proposed an efficient way of streamlining p-hacking: a metric of randomly generated p-values that always indicates significant results (pointless). However, the metric’s applicability is limited, since it covers only a very small range of use cases that are relevant for p-hacking researchers. To address this issue, we present extra pointless, a new metric that allows researchers to freely specify desired p-value ranges while safeguarding full reproducibility. Our newly introduced extra pointless metric not only addresses the limitations of the original metric but also answers editors’ and reviewers’ calls for alternative and innovative statistical approaches. An R-based, interactive shiny web app helps researchers with little background in coding to apply our metric.

Discussion of “Is designed data collection still relevant in the big data era?”

Discussion of “Is designed data collection still relevant in the big data era?”

From BeyondPlanck to Cosmoglobe: Open Science, Reproducibility, and Data Longevity

The BeyondPlanck and Cosmoglobe collaborations have implemented the first integrated Bayesian end-to-end analysis pipeline for CMB experiments. The primary long-term motivation for this work is to develop a common analysis platform that supports efficient global joint analysis of complementary radio, microwave, and sub-millimeter experiments. A strict prerequisite for this to succeed is broad participation from the CMB community, and two foundational aspects of the program are therefore reproducibility and Open Science. In this paper, we discuss our efforts toward this aim. We also discuss measures toward facilitating easy code and data distribution, community-based code documentation, user-friendly compilation procedures, etc. This work represents the first publicly released end-to-end CMB analysis pipeline that includes raw data, source code, parameter files, and documentation. We argue that such a complete pipeline release should be a requirement for all major future and publicly-funded CMB experiments, noting that a full public release significantly increases data longevity by ensuring that the data quality can be improved whenever better processing techniques, complementary datasets, or more computing power become available, and thereby also taxpayers' value for money; providing only raw data and final products is not sufficient to guarantee full reproducibility in the future.

A generic methodology for the statistically uniform & comparable evaluation of Automated Trading Platform components

Introduction:Although machine learning approaches have been widely used in the field of finance, to very successful degrees, these approaches remain bespoke to specific investigations and opaque in terms of explainability, comparability, and reproducibility. Objectives:The primary objective of this research was to shed light upon this field by providing a generic methodology that was investigation-agnostic and interpretable to a financial markets’ practitioner, thus enhancing their efficiency, reducing barriers to entry, and increasing the reproducibility of experiments. The proposed methodology is showcased on two automated trading platform components. Namely, price levels, a well-known trading pattern, and a novel 2-step feature extraction method. Methods:This proposed a generic methodology, useable across markets, the methodology relies on hypothesis testing, which is widely applied in other social and scientific disciplines to effectively evaluate the concrete results beyond simple classification accuracy. The first hypothesis was formulated to evaluate whether the selected trading pattern is suitable for use in the machine learning setting. The second hypothesis allows us to systematically assess whether the proposed feature extraction method leads to any statistically significant improvement in the automated trading platform performance. Results:Experiments were conducted across, 10 contracts, 3 feature spaces, and 3 rebound configurations (for feature extraction), resulting in 90 experiments. Across the experiments we found that the use of the considered trading pattern in the machine learning setting is only partially supported by statistics, resulting in insignificant effect sizes (Rebound 7 - 0.64±1.02, Rebound 11 0.38±0.98, and rebound 15 - 1.05±1.16), but allowed the rejection of the null hypothesis based on the outcome of the statistical test. While the results of the proposed 2-step feature extraction looked promising at first sight, statistics did not support this, this demonstrated the usefulness of the proposed methodology. Additionally, we obtained SHAP values for the considered models, providing insights for adjustments to the feature space. Conclusion:We showcased the generic methodology on a US futures market instrument and provided evidence that with this methodology we could easily obtain informative metrics beyond the more traditional performance and profitability metrics. The interpretability of these results allows the practitioner to construct more effective automated trading pipelines by analysing their strategies using an intuitive and statistically sound methodology. This work is one of the first in applying this rigorous statistically-backed approach to the field of financial markets and we hope this may be a springboard for more research. A full reproducibility package is shared.

Approximation of Bayesian Hawkes process with inlabru

AbstractHawkes process are very popular mathematical tools for modeling phenomena exhibiting a self‐exciting or self‐correcting behavior. Typical examples are earthquakes occurrence, wild‐fires, drought, capture‐recapture, crime violence, trade exchange, and social network activity. The widespread use of Hawkes process in different fields calls for fast, reproducible, reliable, easy‐to‐code techniques to implement such models. We offer a technique to perform approximate Bayesian inference of Hawkes process parameters based on the use of the R‐package inlabru . The inlabru R‐package, in turn, relies on the INLA methodology to approximate the posterior of the parameters. Our Hawkes process approximation is based on a decomposition of the log‐likelihood in three parts, which are linearly approximated separately. The linear approximation is performed with respect to the mode of the parameters' posterior distribution, which is determined with an iterative gradient‐based method. The approximation of the posterior parameters is therefore deterministic, ensuring full reproducibility of the results. The proposed technique only requires the user to provide the functions to calculate the different parts of the decomposed likelihood, which are internally linearly approximated by the R‐package inlabru . We provide a comparison with the bayesianETAS R‐package which is based on an MCMC method. The two techniques provide similar results but our approach requires two to ten times less computational time to converge, depending on the amount of data.

Dynamic Curriculum Learning for Great Ape Detection in the Wild

We propose a novel end-to-end curriculum learning approach for sparsely labelled animal datasets leveraging large volumes of unlabelled data to improve supervised species detectors. We exemplify the method in detail on the task of finding great apes in camera trap footage taken in challenging real-world jungle environments. In contrast to previous semi-supervised methods, our approach adjusts learning parameters dynamically over time and gradually improves detection quality by steering training towards virtuous self-reinforcement. To achieve this, we propose integrating pseudo-labelling with curriculum learning policies and show how learning collapse can be avoided. We discuss theoretical arguments, ablations, and significant performance improvements against various state-of-the-art systems when evaluating on the Extended PanAfrican Dataset holding approx. 1.8M frames. We also demonstrate our method can outperform supervised baselines with significant margins on sparse label versions of other animal datasets such as Bees and Snapshot Serengeti. We note that performance advantages are strongest for smaller labelled ratios common in ecological applications. Finally, we show that our approach achieves competitive benchmarks for generic object detection in MS-COCO and PASCAL-VOC indicating wider applicability of the dynamic learning concepts introduced. We publish all relevant source code, network weights, and data access details for full reproducibility.

Deep learning-based quantification of NAFLD/NASH progression in human liver biopsies

Non-alcoholic fatty liver disease (NAFLD) affects about 24% of the world's population. Progression of early stages of NAFLD can lead to the more advanced form non-alcoholic steatohepatitis (NASH), and ultimately to cirrhosis or liver cancer. The current gold standard for diagnosis and assessment of NAFLD/NASH is liver biopsy followed by microscopic analysis by a pathologist. The Kleiner score is frequently used for a semi-quantitative assessment of disease progression. In this scoring system the features of active injury (steatosis, inflammation, and ballooning) and a separated fibrosis score are quantified. The procedure is time consuming for pathologists, scores have limited resolution and are subject to variation. We developed an automated deep learning method that provides full reproducibility and higher resolution. The system was established with 296 human liver biopsies and tested on 171 human liver biopsies with pathologist ground truth scores. The method is inspired by the way pathologist's analyze liver biopsies. First, the biopsies are analyzed microscopically for the relevant histopathological features. Subsequently, histopathological features are aggregated to a per-biopsy score. Scores are in the identical numeric range as the pathologist’s ballooning, inflammation, steatosis, and fibrosis scores, but on a continuous scale. Resulting scores followed a pathologist's ground truth (quadratic weighted Cohen’s κ on the test set: for steatosis 0.66, for inflammation 0.24, for ballooning 0.43, for fibrosis 0.62, and for the NAFLD activity score (NAS) 0.52. Mean absolute errors on a test set: for steatosis 0.29, for inflammation 0.53, for ballooning 0.61, for fibrosis 0.78, and for the NAS 0.77).

Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating Model‐Agnostic and Model‐Specific Configuration Steps in Applications of Large‐Domain Hydrologic Models

AbstractDespite the proliferation of computer‐based research on hydrology and water resources, such research is typically poorly reproducible. Published studies have low reproducibility due to incomplete availability of data and computer code, and a lack of documentation of workflow processes. This leads to a lack of transparency and efficiency because existing code can neither be quality controlled nor reused. Given the commonalities between existing process‐based hydrologic models in terms of their required input data and preprocessing steps, open sharing of code can lead to large efficiency gains for the modeling community. Here, we present a model configuration workflow that provides full reproducibility of the resulting model instantiations in a way that separates the model‐agnostic preprocessing of specific data sets from the model‐specific requirements that models impose on their input files. We use this workflow to create large‐domain (global and continental) and local configurations of the Structure for Unifying Multiple Modeling Alternatives (SUMMA) hydrologic model connected to the mizuRoute routing model. These examples show how a relatively complex model setup over a large domain can be organized in a reproducible and structured way that has the potential to accelerate advances in hydrologic modeling for the community as a whole. We provide a tentative blueprint of how community modeling initiatives can be built on top of workflows such as this. We term our workflow the “Community Workflows to Advance Reproducibility in Hydrologic Modeling” (CWARHM; pronounced “swarm”).

Stochastic LCA Model of Upscaling the Production of Microalgal Compounds.

Microalgae are currently being investigated for their promising metabolites but assessing the environmental impact of producing these compounds remains a challenge. Microalgae cultivation performance results from the complex interaction of biological, technological, geographical, and physical factors, which bioengineers try to optimize during the upscaling process. The path from the discovery of a microalgal compound to its industrial production is therefore highly uncertain. Nonetheless, it is key to anticipate the potential environmental impacts associated with the future production of a microalgal target compound. This is achieved in this study by developing an ex-ante, parameterized, and consequential LCA model that performs dynamic simulations of microalgae cultivation. The model is applied to calculate the environmental impacts of 9000 stochastically generated combinations of photobioreactor geometries and operational setups. The demonstration of the model is done for a fictive microalgal strain, parameterized to resemble Chlorella vulgaris, and a fictive target compound assumed to be a carbohydrate. The simulations are performed in Aalborg, Denmark, and Granada, Spain to appreciate geographical variability, which highly affects the requirements for thermoregulation. Open-source documentation allows full reproducibility and further use of the model for the ex-ante assessment of microalgal products.

A Validated Molecular Protocol to Differentiate Pure Wolves, Dogs and Wolf x Dog Hybrids through a Panel of Multiplexed Canine STR Markers

The conservation of pure wolf populations in Europe is currently threatened by anthropogenic hybridization with dogs, through potential extinction of local gene pools due to replacement with domestic gene variants. Distinction of wolf–dog hybrids from wild ancestors is essential for development and implementation of management and conservation plans. Morphological traits are rarely helpful, and a genetic approach is the most effective to distinguish admixed individuals back to old backcrosses. To provide a molecular tool to address the issue, we optimized and validated a cost-effective protocol in multiplexed PCR format based on 22 STR canine loci, which allows us to distinguish genetically pure wolves from dogs, and, when used in association with a Bayesian assignment approach, is capable of statistically assigning admixed individuals to classes of hybrids with different levels of dog ancestry. Our method demonstrated high reliability, showing full repeatability and reproducibility of data with as little as 0.125 ng of genomic DNA, and was therefore suitable for the analysis of non-invasively collected samples and degraded DNA. The application of our STR panel to the appropriate assignment procedure unambiguously defined two genetically separated clusters for wolves and dogs, and successfully identified known hybrids as admixed individuals, which eventually were classified as recent hybrids and older backcrosses. The protocol, which is described here in detail, can be adopted by various laboratories that need this kind of diagnosis; furthermore, it would be capable of producing concordant results through inter-laboratory comparisons with wolf and dog control DNAs.

Rpadrino: An R package to access and use PADRINO, an open access database of Integral Projection Models

Abstract Discrete time structured population projection models are an important tool for studying population dynamics. Within this field, integral projection models (IPMs) have become a popular method for studying populations structured by continuously distributed traits (e.g. height, weight). Databases of discrete time, discrete state structured population models, for example DATLife (life tables) and COMPADRE & COMADRE (matrix population models), have made quantitative syntheses straightforward to implement. These efforts allow researchers to address questions in both basic and applied ecology and evolutionary biology. Since their introduction in 2000, over 300 works containing IPMs have been published, offering opportunities for ecological synthesis too. We describe a novel framework to quickly reconstruct these models for subsequent analyses using Rpadrino R package, which serves as an interface to PADRINO, a new database of IPMs. We introduce an R package, Rpadrino, which enables users to download, subset, reconstruct, and extend published IPMs. Rpadrino makes use of recently created software, ipmr, to provide an engine to reconstruct a wide array of IPMs from their symbolic representations and conduct subsequent analyses. Rpadrino and ipmr are extensively documented to help users learn their usage. Rpadrino currently enables users to reconstruct 280 IPMs from 40 publications that describe the demography of 14 animal and 26 plant species. All of these IPMs are tested to ensure they reproduce published estimates. Rpadrino provides an interface to augment PADRINO with external data and modify parameter values, creating a platform to extend models beyond their original purpose while retaining full reproducibility. PADRINO and Rpadrino provide a toolbox for asking new questions and conducting syntheses with peer‐reviewed published IPMs. Rpadrino provides a user‐friendly interface so researchers do not need to worry about the database structure or syntax, and can focus on their research questions and analyses. Additionally, Rpadrino is thoroughly documented and provides numerous examples of how to perform analyses which are not included in the package's functionality.