Confidential Datasets Research Articles

Physics-guided federated learning as an enabler for digital twins

Digital twins bring the potential to increase the efficiency of assets, systems, and processes by building virtual replicas through real-time data and modeling. However, data are often confidential and distributed, high-fidelity models based on physical principles tend to be slow and require detailed knowledge about the asset design that may not be available, simplified models lack accuracy, and data-driven models are not interpretable and limited to their training space. We demonstrate the interplay of two enabling technologies, namely federated learning and hybrid analysis and modeling, to combine the strengths of physics-based and data-driven models on distributed data sets that are kept confidential by different stakeholders. Throughout the work, an ensemble of physics-guided neural networks is designed, optimized, and validated to infer the parameters of digital twin components. The physics-guided neural network is injected with output from a simplified physics-based model in intermediate layers for increased performance, and the injection layer is optimized through weight regularization. The model is then integrated into a simulation of the digital twin asset and validated there. It is shown that the model outperforms the simplified physics-based and the data-driven models both on component and on asset level. In the final step, the physics-guided neural network is trained using federated methods, which allows training on confidential data owned by different stakeholders without requiring the stakeholders to share their data. We show that federated hybrid models can – with the right training strategy – by design achieve results that are always identical and potentially superior to models trained on a central data set. The problem chosen for this demonstration is the prediction of airfoil lift and drag coefficients based on the airfoil shape and the angle of attack, which is a task crucial for simulating wind turbines. Using the physics-guided neural network, the estimation of the airfoil coefficients experiences a speedup of several orders of magnitude compared to the high-fidelity simulation, and integration into blade element momentum theory shows that the turbine thrust, torque, and mechanical power can be accurately calculated. The federated approach makes it possible to extend the training of the model to any number of distributed confidential data sets from both simulations and experiments to further increase the model’s accuracy.

Federated Learning and Unlearning as Enablers of Wind Turbine Digital Twins

Real-time capable models are paramount for the successful adaptation of digital twin technology into industries such as wind energy, but high-fidelity physics-based models cannot achieve the required speed, while data-driven and hybrid methods require large amounts of training data which is typically confidential. In this work, the combination of federated learning with hybrid modeling is proposed to train fast and reliable models across multiple confidential data sets owned by different stakeholders. The approach is demonstrated on physics-guided neural networks to estimate the lift and drag of wind turbine airfoils. A scenario is devised where multiple confidential data subsets are confined to different client devices. It is shown that the physics-guided neural networks can be trained through federated learning across those devices and data subsets and that the resulting models can recover and even surpass the accuracy of a model that is trained conventionally by merging the data subsets on a single device. The presented approach is highly scalable and can be easily adapted to many other applications. This work also discusses federated unlearning methods, which allow data owners to remove all traces of a data subset used in training if they decide to revoke their contribution.

Seed-Guided Fine-Grained Entity Typing in Science and Engineering Domains

Accurately typing entity mentions from text segments is a fundamental task for various natural language processing applications. Many previous approaches rely on massive human-annotated data to perform entity typing. Nevertheless, collecting such data in highly specialized science and engineering domains (e.g., software engineering and security) can be time-consuming and costly, without mentioning the domain gaps between training and inference data if the model needs to be applied to confidential datasets. In this paper, we study the task of seed-guided fine-grained entity typing in science and engineering domains, which takes the name and a few seed entities for each entity type as the only supervision and aims to classify new entity mentions into both seen and unseen types (i.e., those without seed entities). To solve this problem, we propose SEType which first enriches the weak supervision by finding more entities for each seen type from an unlabeled corpus using the contextualized representations of pre-trained language models. It then matches the enriched entities to unlabeled text to get pseudo-labeled samples and trains a textual entailment model that can make inferences for both seen and unseen types. Extensive experiments on two datasets covering four domains demonstrate the effectiveness of SEType in comparison with various baselines. Code and data are available at: https://github.com/yuzhimanhua/SEType.

Automatically reconciling the trade-off between prediction accuracy and earliness in prescriptive business process monitoring

Prescriptive business process monitoring provides decision support to process managers on when and how to adapt an ongoing business process to prevent or mitigate an undesired process outcome. We focus on the problem of automatically reconciling the trade-off between prediction accuracy and prediction earliness in determining when to adapt. Adaptations should happen sufficiently early to provide enough lead time for the adaptation to become effective. However, earlier predictions are typically less accurate than later predictions. This means that acting on less accurate predictions may lead to unnecessary adaptations or missed adaptations.Different approaches were presented in the literature to reconcile the trade-off between prediction accuracy and earliness. So far, these approaches were compared with different baselines, and evaluated using different data sets or even confidential data sets. This limits the comparability and replicability of the approaches and makes it difficult to choose a concrete approach in practice.We perform a comparative evaluation of the main alternative approaches for reconciling the trade-off between prediction accuracy and earliness. Using four public real-world event log data sets and two types of prediction models, we assess and compare the cost savings of these approaches. The experimental results indicate which criteria affect the effectiveness of an approach and help us state initial recommendations for the selection of a concrete approach in practice.

Industry-Scale Orchestrated Federated Learning for Drug Discovery

To apply federated learning to drug discovery we developed a novel platform in the context of European Innovative Medicines Initiative (IMI) project MELLODDY (grant n°831472), which was comprised of 10 pharmaceutical companies, academic research labs, large industrial companies and startups. The MELLODDY platform was the first industry-scale platform to enable the creation of a global federated model for drug discovery without sharing the confidential data sets of the individual partners. The federated model was trained on the platform by aggregating the gradients of all contributing partners in a cryptographic, secure way following each training iteration. The platform was deployed on an Amazon Web Services (AWS) multi-account architecture running Kubernetes clusters in private subnets. Organisationally, the roles of the different partners were codified as different rights and permissions on the platform and administrated in a decentralized way. The MELLODDY platform generated new scientific discoveries which are described in a companion paper.

Conformal efficiency as a metric for comparative model assessment befitting federated learning

In a drug discovery setting, pharmaceutical companies own substantial but confidential datasets. The MELLODDY project developed a privacy-preserving federated machine learning solution and deployed it at an unprecedented scale. Each partner built models for their own private assays that benefitted from a shared representation. Established predictive performance metrics such as AUC ROC or AUC PR are constrained to unseen labeled chemical space and cannot gage performance gains in unlabeled chemical space. Federated learning indirectly extends labeled space, but in a privacy-preserving context, a partner cannot use this label extension for performance assessment. Metrics that estimate uncertainty on a prediction can be calculated even where no label is known. Practically, the chemical space covered with predictions above an uncertainty threshold, reflects the applicability domain of a model. After establishing a link to established performance metrics, we propose the efficiency from the conformal prediction framework (‘conformal efficiency’) as a proxy to the applicability domain size. A documented extension of the applicability domain would qualify as a tangible benefit from federated learning. In interim assessments, MELLODDY partners reported a median increase in conformal efficiency of the federated over the single-partner model of 5.5% (with increases up to 9.7%). Subject to distributional conditions, that efficiency increase can be directly interpreted as the expected increase in conformal i.e. low uncertainty predictions. In conclusion, we present the first indication that privacy-preserving federated machine learning across massive drug-discovery datasets from ten pharma partners indeed extends the applicability domain of property prediction models.

Bill of lading data in international trade research with an application to the COVID‐19 pandemic

AbstractWe evaluate high‐frequency bill of lading data for international trade research. These data offer some advantages over both other publicly accessible trade data and confidential datasets, but they also have drawbacks. We analyze three aspects of trade during the COVID‐19 pandemic. First, we show how the high‐frequency data capture the within‐month collapse of trade between the United States and India that are not observable in official monthly data. Second, we demonstrate how U.S. buyers shifted their purchases across suppliers over time during the recovery. And third, we show how the data can measure vessel delivery bottlenecks in near real time.

Bill of Lading Data in International Trade Research with an Application to the COVID-19 Pandemic

<ns2:p>We evaluate high-frequency bill of lading data for its suitability in international trade research. These data offer many advantages over both other publicly accessible official trade data and confidential datasets, but they also have clear drawbacks. We provide a comprehensive overview for potential researchers to understand these strengths and weaknesses as these data become more widely available. Drawing on the strengths of the data, we analyze three aspects of trade during the COVID- 19 pandemic. First, we show how the high-frequency data capture features of the within-month collapse of trade between the United States and India that are not observable in official monthly data. Second, we demonstrate how U.S. buyers shifted their purchases across suppliers over time during the recovery. And third, we show how the data can be used to measure vessel delivery bottlenecks in near real time.</ns2:p>

Parallel encryption of input and output data for HPC applications

A methodology for protecting confidential data sets on third-party HPC systems is reported. This is based on the NIST AES algorithm and supports the common ECB, CTR and CBC modes. The methodology is built on a flexible programming model that delegates management of the encryption key to the application code. The methodology also includes a fine-grain control over which arrays on the files are encrypted. All the stages in an encrypted workflow are investigated using an established CFD code. Benchmarks are reported using the UK national supercomputer service (ARCHER) running the CFD code on up to 18,432 cores. Performance benchmarks demonstrate the importance of the way the encryption metadata is treated. Naïve treatments are shown to have a large impact on performance. However, through a more judicious treatment, the time to run the solver with encrypted input and output data is shown to be almost identical to that with plain data. A novel parallel treatment of the block chaining in AES-CBC mode allows users to benefit from the avalanche properties of this mode relative to the CTR mode, with no penalty in run-time.

Estimating the time-varying reproduction number of SARS-CoV-2 using national and subnational case counts

Background:Assessing temporal variations in transmission in different countries is essential for monitoring the epidemic, evaluating the effectiveness of public health interventions and estimating the impact of changes in policy.Methods:We use case and death notification data to generate daily estimates of the time-varying reproduction number globally, regionally, nationally, and subnationally over a 12-week rolling window. Our modelling framework, based on open source tooling, accounts for uncertainty in reporting delays, so that the reproduction number is estimated based on underlying latent infections.Results:Estimates of the reproduction number, trajectories of infections, and forecasts are displayed on a dedicated website as both maps and time series, and made available to download in tabular form.Conclusions: This decision-support tool can be used to assess changes in virus transmission both globally, regionally, nationally, and subnationally. This allows public health officials and policymakers to track the progress of the outbreak in near real-time using an epidemiologically valid measure. As well as providing regular updates on our website, we also provide an open source tool-set so that our approach can be used directly by researchers and policymakers on confidential data-sets. We hope that our tool will be used to support decisions in countries worldwide throughout the ongoing COVID-19 pandemic.

Le Pont de Londres: Interactions between monetary and prudential policies in cross‐border lending

AbstractUsing two unique confidential datasets summarizing the cross‐border lending of banks in France and the UK, we examine whether recipient‐country prudential policies can help to reduce the spillover effects of euro‐area (EA) monetary policy. We address this question from a novel angle, focused on the role of international financial centres, by considering differences in bank size and location (lending from French headquarters vs. from French affiliates located in the UK). For small French banks that lack a presence in international financial centres, the response of direct cross‐border lending from France to EA monetary policy is partially offset by recipient‐country prudential policy. For larger banks, however, the offsetting effect applies only to lending that passes through foreign affiliates located in London. This suggests the existence of a “London Bridge”: banks adapt their flows to the UK conditional on EA monetary policy and global prudential policies; and from their UK affiliates to third‐party countries in a manner that depends on local prudential settings.

Estimating the time-varying reproduction number of SARS-CoV-2 using national and subnational case counts

Background: Interventions are now in place worldwide to reduce transmission of the novel coronavirus. Assessing temporal variations in transmission in different countries is essential for evaluating the effectiveness of public health interventions and the impact of changes in policy. Methods: We use case notification data to generate daily estimates of the time-dependent reproduction number in different regions and countries. Our modelling framework, based on open source tooling, accounts for reporting delays, so that temporal variations in reproduction number estimates can be compared directly with the times at which interventions are implemented. Results: We provide three example uses of our framework. First, we demonstrate how the toolset displays temporal changes in the reproduction number. Second, we show how the framework can be used to reconstruct case counts by date of infection from case counts by date of notification, as well as to estimate the reproduction number. Third, we show how maps can be generated to clearly show if case numbers are likely to decrease or increase in different regions. Results are shown for regions and countries worldwide on our website (https://epiforecasts.io/covid/) and are updated daily. Our tooling is provided as an open-source R package to allow replication by others. Conclusions: This decision-support tool can be used to assess changes in virus transmission in different regions and countries worldwide. This allows policymakers to assess the effectiveness of current interventions, and will be useful for inferring whether or not transmission will increase when interventions are lifted. As well as providing daily updates on our website, we also provide adaptable computing code so that our approach can be used directly by researchers and policymakers on confidential datasets. We hope that our tool will be used to support decisions in countries worldwide throughout the ongoing COVID-19 pandemic.

Le Pont de Londres: Interactions Between Monetary and Prudential Policies in Cross-Border Lending

By combining analysis of two unique confidential datasets, we examine how euro-area (EA) monetary policy and recipient-country prudential policy interact to influence the cross-border lending of French banks from France and the UK. We find that monetary spillovers via cross-border lending can be partially offset by prudential measures in receiving countries. We then explore heterogeneities in that interaction, specifically the difference made by bank size and location of the affiliate (French headquarters vs. affiliates based in the UK, an international financial centre). Focusing on lending from France, we find that the response of GSIBs’ lending to EA monetary policy is less sensitive to recipient-country prudential policy than non-GSIBs’. In contrast, the response of lending to EA monetary policy from French-owned GSIB affiliates in the UK is sensitive to recipient-country prudential policy. We also find evidence that French GSIBs channel funds towards the UK in response to EA monetary policy, in a manner that is dampened by the global prudential policy setting. Together, these findings suggest the existence of a ‘London Bridge’: conditional on EA monetary policy, French GSIBs adjust their funds in the UK in response to global prudential policies and, from there, lend to third countries, responding to local prudential policies. French GSIBs’ cross-border lending from their headquarters to EA monetary policy responds differently to foreign prudential policies than their lending from international financial centres. Finally, we find evidence of a similar pattern for all EA-owned bank affiliates in the UK, suggesting a broader relevance of the London Bridge.

How Do Banking Groups React to Macroprudential Policies? Cross-Border Spillover Effects of Higher Capital Buffers on Lending, Risk-Taking and Internal Markets

We study the impact of macroprudential capital buffers on banking groups' lending and risk-taking decisions, also investigating implications for internal capital markets. For identification, we exploit heterogeneity in buffers applied to other systemically important institutions, using information from three unique confidential datasets, including information on the EBA scoring process. This policy design induces a randomized experiment in the neighborhood of the threshold, which we use to identify the effect of higher capital requirements by comparing the change in the outcome for banks just above and below the cut-off, before and after the introduction of the buffer. The analysis is implemented relying on a fuzzy regression discontinuity and on a difference-in-differences matching design. We find that, when parent banks are constrained with higher buffers, subsidiaries deleverage lending and risk-taking towards non-financial corporations and marginally expanded lending towards households, with negative effects on protability. Also, we find that parents cut down on holdings of debt and equity issued by their subsidiaries. Our findings support the hypothesis that higher capital buffers have a positive disciplinary effect by reducing banks' risk-taking while having a (temporary) adverse impact on the real economy through a decrease in affiliated banks' lending activity. Therefore, to ensure the effectiveness of macroprudential policy, it is essential that policymakers assess their potential cross-border effects. JEL Classification: E44, E51, E58, G21, G28

Evaluation-as-a-Service for the Computational Sciences

Evaluation in empirical computer science is essential to show progress and assess technologies developed. Several research domains such as information retrieval have long relied on systematic evaluation to measure progress: here, the Cranfield paradigm of creating shared test collections, defining search tasks, and collecting ground truth for these tasks has persisted up until now. In recent years, however, several new challenges have emerged that do not fit this paradigm very well: extremely large data sets, confidential data sets as found in the medical domain, and rapidly changing data sets as often encountered in industry. Crowdsourcing has also changed the way in which industry approaches problem-solving with companies now organizing challenges and handing out monetary awards to incentivize people to work on their challenges, particularly in the field of machine learning. This article is based on discussions at a workshop on Evaluation-as-a-Service (EaaS). EaaS is the paradigm of not providing data sets to participants and have them work on the data locally, but keeping the data central and allowing access via Application Programming Interfaces (API), Virtual Machines (VM), or other possibilities to ship executables. The objectives of this article are to summarize and compare the current approaches and consolidate the experiences of these approaches to outline the next steps of EaaS, particularly toward sustainable research infrastructures. The article summarizes several existing approaches to EaaS and analyzes their usage scenarios and also the advantages and disadvantages. The many factors influencing EaaS are summarized, and the environment in terms of motivations for the various stakeholders, from funding agencies to challenge organizers, researchers and participants, to industry interested in supplying real-world problems for which they require solutions. EaaS solves many problems of the current research environment, where data sets are often not accessible to many researchers. Executables of published tools are equally often not available making the reproducibility of results impossible. EaaS, however, creates reusable/citable data sets as well as available executables. Many challenges remain, but such a framework for research can also foster more collaboration between researchers, potentially increasing the speed of obtaining research results.

Do practitioner assessments agree with academic proxies for audit quality? Evidence from PCAOB and internal inspections

This study investigates the degree of concordance between fifteen measures of audit quality used in academia and two measures of audit process quality determined either by audit firms’ internal inspections or by Public Company Accounting Oversight Board inspections of individual engagements. Using two confidential datasets of these assessments of audit process quality, I find that three of the measures of audit quality used by academics have significant associations with both measures of audit process deficiencies used by auditors and regulators: (i) the propensity to restate financial statements, (ii) the propensity to meet or beat the zero earnings threshold, and (iii) audit fees. Seven academic proxies are significantly associated with only one audit process quality measure, and five have insignificant associations with both practitioner assessments. Overall, the significant associations indicate that practitioners and academics share common ground in identifying low-quality audits. These findings can provide guidance for future studies in selecting audit-quality proxies suitable for different research questions.

Asynchronous Monetary Policies and International Dollar Credit

This paper presents a theoretical model in which the supply of international dollar credit by a global bank is responsive to unconventional monetary policies (UMPs) both in the US and its home country, the functioning of the FX swap market and the bank’s default risk. The theoretical model is tested using two unique confidential datasets. The results suggest that the contractionary effect of US monetary normalisation on global liquidity would be partly offset by the expansionary effect of UMPs in Japan and the euro-area. However, a stress testing exercise shows that global liquidity would be seriously disrupted if normalisation of monetary policy in the US leads to financial market dislocation, in particular in the FX swap market. Finally, this study finds that global banks’ risk-taking attitude, credit risk exposure, and the business model of their overseas offices are important factors affecting how dollar credit supplied by international banks would respond to UMPs.

Imputation of confidential data sets with spatial locations using disease mapping models

Data that include fine geographic information, such as census tract or street block identifiers, can be difficult to release as public use files. Fine geography provides information that ill-intentioned data users can use to identify individuals. We propose to release data with simulated geographies, so as to enable spatial analyses while reducing disclosure risks. We fit disease mapping models that predict areal-level counts from attributes in the file and sample new locations based on the estimated models. We illustrate this approach using data on causes of death in North Carolina, including evaluations of the disclosure risks and analytic validity that can result from releasing synthetic geographies.

Secure Computation of the Median (and Other Elements of Specified Ranks)

We consider the problem of securely computing the kth-ranked element of the union of two or more large, confidential data sets. This is a fundamental question motivated by many practical contexts. For example, two competitive companies may wish to compute the median salary of their combined employee populations without revealing to each other the exact salaries of their employees. While protocols do exist for computing the kth-ranked element, they require time that is at least linear in the sum of the sizes of their combined inputs. This paper investigates two-party and multi-party protocols for both the semi-honest and malicious cases. In the two-party setting, we prove that the problem can be solved in a number of rounds that is logarithmic in k, where each round requires communication and computation cost that is linear in b, the number of bits needed to describe each element of the input data. In the multi-party setting, we prove that the number of rounds is linear in b, where each round has overhead proportional to b multiplied by the number of parties. The multi-party protocol can be used in the two-party case. The overhead introduced by our protocols closely match the communication complexity lower bound. Our protocols can handle a malicious adversary via simple consistency checks.

Provable Security for Outsourcing Database Operations

Database outsourcing has become popular in recent years, although it introduces substantial security and privacy risks. In many applications, users may not want to reveal their data even to a generally trusted database service provider. Several researchers have proposed encryption schemes, such as privacy homomorphisms, that allow service providers to process confidential data sets without learning too much about them. In this paper, the authors discuss serious flaws of these solutions. The authors then present a new definition of security for homomorphic database encryption schemes that avoids these flaws and show that it is difficult to build a privacy homomorphism that complies with this definition. As a practical compromise, the authors present a relaxed variant of the security definition and discuss arising security implications. They present a new method to construct encryption schemes for exact selects and prove that the resulting schemes satisfy this notion.