Pairwise Probability Research Articles

Co-occurring mental and substance use disorders in Australia 2020-2022: Prevalence, patterns, conditional probabilities and correlates in the general population.

Previous estimates from 2007 found that co-occurring mental and/or substance use disorders were a pervasive feature of Australia's mental health. Since that time there have been shifts and improvements in the conceptualisation and incorporation of co-occurring disorders in research and treatment settings. The current study provides up-to-date estimates on the prevalence of co-occurring mental and/or substance use disorders, highlights common patterns of co-occurrence, identifies significant correlates and examines any changes in the extent of co-occurring disorders since 2007. Data were from the two Australian National Surveys of Mental Health and Wellbeing conducted in 2020-2022 (N = 15,893) and 2007 (N = 8841). Descriptive statistics were estimated for the number of co-occurring conditions, correlations and pairwise conditional probabilities. Multinomial logistic and robust Poisson regressions were used to identify significant correlates and compare changes in co-occurring conditions across surveys. Approximately 46% of people with a mental or substance use disorder in the past 12 months experienced two or more diagnosable conditions. There was little evidence to suggest that the prevalence of co-occurring disorders has changed since 2007 (Prevalence Ratio (PR) = 1.08, 95% CI = 0.98-1.18). Subgroup analysis indicated that those aged 16-24 years were significantly more likely to experience any co-occurrence in 2020-2022 compared with those aged 16-24 years in 2007 (PR = 1.44, 95% CI = 1.17-1.77). Co-occurring mental and substance use disorders remain endemic in Australia. Indeed, they appear to be increasingly problematic in younger, more recent cohorts. The results suggest that continued effort is needed to develop and implement transdiagnostic interventions that target broad contextual and/or societal factors.

Unbending strategies shepherd cooperation and suppress extortion in spatial populations

Evolutionary game dynamics on networks typically consider the competition among simple strategies such as cooperation and defection in the Prisoner’s Dilemma and summarize the effect of population structure as network reciprocity. However, it remains largely unknown regarding the evolutionary dynamics involving multiple powerful strategies typically considered in repeated games, such as the zero-determinant (ZD) strategies that are able to enforce a linear payoff relationship between them and their co-players. Here, we consider the evolutionary dynamics of always cooperate (AllC), extortionate ZD (extortioners), and unbending players in lattice populations based on the commonly used death-birth updating. Out of the class of unbending strategies that can foster reciprocal cooperation and fairness among extortionate players, we consider a particular candidate, pre-optimized through the machine-learning method of particle swarm optimization (PSO), called PSO Gambler. We derive analytical results under weak selection and rare mutations, including pairwise fixation probabilities and long-term frequencies of strategies. In the absence of the third unbending type, extortioners can achieve a half-half split in equilibrium with unconditional cooperators for sufficiently large extortion factors. However, the presence of unbending players fundamentally changes the dynamics and tilts the system to favor unbending cooperation. Most surprisingly, extortioners cannot dominate at all regardless of how large their extortion factor is, and the long-term frequency of unbending players is maintained almost as a constant. Our analytical method is applicable to studying the evolutionary dynamics of multiple strategies in structured populations. Our work provides insights into the interplay between network reciprocity and direct reciprocity, revealing the role of unbending strategies in enforcing fairness and suppressing extortion.

Chainsaw: protein domain segmentation with fully convolutional neural networks.

Protein domains are fundamental units of protein structure and play a pivotal role in understanding folding, function, evolution, and design. The advent of accurate structure prediction techniques has resulted in an influx of new structural data, making the partitioning of these structures into domains essential for inferring evolutionary relationships and functional classification. This article presents Chainsaw, a supervised learning approach to domain parsing that achieves accuracy that surpasses current state-of-the-art methods. Chainsaw uses a fully convolutional neural network which is trained to predict the probability that each pair of residues is in the same domain. Domain predictions are then derived from these pairwise predictions using an algorithm that searches for the most likely assignment of residues to domains given the set of pairwise co-membership probabilities. Chainsaw matches CATH domain annotations in 78% of protein domains versus 72% for the next closest method. When predicting on AlphaFold models, expert human evaluators were twice as likely to prefer Chainsaw's predictions versus the next best method. github.com/JudeWells/Chainsaw.

Two-Stage Dimensionality Reduction for Social Media Engagement Classification

The high dimensionality of real-life datasets is one of the biggest challenges in the machine learning field. Due to the increased need for computational resources, the higher the dimension of the input data is, the more difficult the learning task will be—a phenomenon commonly referred to as the curse of dimensionality. Laying the paper’s foundation based on this premise, we propose a two-stage dimensionality reduction (TSDR) method for data classification. The first stage extracts high-quality features to a new subset by maximizing the pairwise separation probability, with the aim of avoiding overlap between individuals from different classes that are close to one another, also known as the class masking problem. The second stage takes the previous resulting subset and transforms it into a reduced final space in a way that maximizes the distance between the cluster centers of different classes while also minimizing the dispersion of instances within the same class. Hence, the second stage aims to improve the accuracy of the succeeding classifier by lowering its sensitivity to an imbalanced distribution of instances between different classes. Experiments on benchmark and social media datasets show how promising the proposed method is over some well-established algorithms, especially regarding social media engagement classification.

Opponent choice in tournaments: winning and shirking

Abstract We propose an alternative design for tournaments that use a preliminary stage, followed by several rounds of single elimination play. The conventional “bracket” design of these tournaments suffers from several deficiencies. Specifically, various reasonable performance criteria for the tournament are not satisfied, there is an unnecessary element of luck in the matchups of players, and there are situations where players have an incentive to shirk. To address all these issues, we allow higher ranked players at the single elimination stage to choose their next opponent sequentially at each round. We allow each player’s ranking either to remain static, or to improve by beating a higher ranked player (Guyon, J. 2022. “Choose your opponent”: a new knockout design for hybrid tournaments. J. Sports Anal. 8: 9–29). Using data from 2215 men’s professional tennis tournaments from 1991 to 2017, we demonstrate the reasonableness of the results obtained. We also perform sensitivity analysis for the effect of increasing irregularity in the pairwise win probability matrix on three traditional performance measures. Finally, we consider strategic shirking behavior at both the individual and group levels, and show how our opponent choice design can mitigate such behavior. Overall, the opponent choice design provides higher probabilities that the best player wins and also that the two best players meet, and reduces shirking, compared to the conventional bracket design.

ProbBreed: a novel tool for calculating the risk of cultivar recommendation in multienvironment trials.

Neglecting genotype-by-environment interactions in multienvironment trials (MET) increases the risk of flawed cultivar recommendations for growers. Recent advancements in probability theory coupled with cutting-edge software offer a more streamlined decision-making process for selecting suitable candidates across diverse environments. Here, we present the user-friendly ProbBreed package in R, which allows breeders to calculate the probability of a given genotype outperforming competitors under a Bayesian framework. This article outlines the package's basic workflow and highlights its key features, ranging from MET model fitting to estimating the per se and pairwise probabilities of superior performance and stability for selection candidates. Remarkably, only the selection intensity is required to compute these probabilities. By democratizing this complex yet efficient methodology, ProbBreed aims to enhance decision-making and ultimately contribute to more accurate cultivar recommendations in breeding programs.

MSDSE: Predicting drug-side effects based on multi-scale features and deep multi-structure neural network

Unexpected side effects may accompany the research stage and post-marketing of drugs. These accidents lead to drug development failure and even endanger patients' health. Thus, it is essential to recognize the unknown drug-side effects. Most existing methods in silico find the answer from the association network or similarity network of drugs while ignoring the drug-intrinsic attributes. The limitation is that they can only handle drugs in the maturation stage.To be suitable for early drug-side effect screening, we conceive a multi-structural deep learning framework, MSDSE, which synthetically considers the multi-scale features derived from the drug. MSDSE can jointly learn SMILES sequence-based word embedding, substructure-based molecular fingerprint, and chemical structure-based graph embedding. In the preprocessing stage of MSDSE, we project all features to the abstract space with the same dimension. MSDSE builds a bi-level channel strategy, including a convolutional neural network module with an Inception structure and a multi-head Self-Attention module, to learn and integrate multi-modal features from local to global perspectives. Finally, MSDSE regards the prediction of drug-side effects as pair-wise learning and outputs the pair-wise probability of drug-side effects through the inner product operation. MSDSE is evaluated and analyzed on benchmark datasets and performs optimally compared to other baseline models. We also set up the ablation study to explain the rationality of the feature approach and model structure. Moreover, we select model partial prediction results for the case study to reveal actual capability. The original data are available at http://github.com/yuliyi/MSDSE.

Phylourny: efficiently calculating elimination tournament win probabilities via phylogenetic methods

The prediction of knockout tournaments represents an area of large public interest and active academic as well as industrial research. Here, we show how one can leverage the computational analogies between calculating the phylogenetic likelihood score used in the area of molecular evolution to efficiently calculate, instead of approximate via simulations, the exact per-team tournament win probabilities, given a pairwise win probability matrix between all teams. We implement and make available our method as open-source code and show that it is two orders of magnitude faster than simulations and two or more orders of magnitude faster than calculating the exact per-team win probabilities naïvely, without taking into account the substantial computational savings induced by the tournament tree structure. Furthermore, we showcase novel prediction approaches that now become feasible due to this order of magnitude improvement in calculating tournament win probabilities. We demonstrate how to quantify prediction uncertainty by calculating 100,000 distinct tournament win probabilities for a tournament with 16 teams under slight variations of a reasonable pairwise win probability matrix within one minute on a standard laptop. We also conduct an analogous analysis for a tournament with 64 teams.

Causality-Aware Convolutional Neural Networks for Advanced Image Classification and Generation

Smart manufacturing uses emerging deep learning models, and particularly Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), for different industrial diagnostics tasks, e.g., classification, detection, recognition, prediction, synthetic data generation, security, etc., on the basis of image data. In spite of being efficient for these objectives, the majority of current deep learning models lack interpretability and explainability. They can discover features hidden within input data together with their mutual co-occurrence. However, they are weak at discovering and making explicit hidden causalities between the features, which could be the reason behind the particular diagnoses. In this paper, we suggest Causality-Aware CNNs (CA-CNNs) and Causality-Aware GANs (CA-GANs) to address the issue of learning hidden causalities within images. The core architecture includes an additional layer of neurons (after the last convolution-pooling and just before the dense layers), which learns pairwise conditional probabilities (aka causality estimates) for the features. Computations for these neurons are driven by the adaptive Lehmer mean function. Learned causalities are merged with the features during flattening and (via fully connected layers) influence the classification outcomes. Such causality estimates can be done for the mixed inputs where images are combined with other data. We argue that CA-CNNs not only improve the classification performance of normal CNNs but also open additional opportunities for the explainability of the models’ outcomes. We consider as an additional advantage for CA-CNNs (if used as a discriminator within CA-GANs) the possibility to generate realistically looking images with respect to the causalities.

Dependence-Robust Confidence Intervals for Capture-Recapture Surveys.

Capture-recapture (CRC) surveys are used to estimate the size of a population whose members cannot be enumerated directly. CRC surveys have been used to estimate the number of Coronavirus Disease 2019 (COVID-19) infections, people who use drugs, sex workers, conflict casualties, and trafficking victims. When k-capture samples are obtained, counts of unit captures in subsets of samples are represented naturally by a contingency table in which one element-the number of individuals appearing in none of the samples-remains unobserved. In the absence of additional assumptions, the population size is not identifiable (i.e., point identified). Stringent assumptions about the dependence between samples are often used to achieve point identification. However, real-world CRC surveys often use convenience samples in which the assumed dependence cannot be guaranteed, and population size estimates under these assumptions may lack empirical credibility. In this work, we apply the theory of partial identification to show that weak assumptions or qualitative knowledge about the nature of dependence between samples can be used to characterize a nontrivial confidence set for the true population size. We construct confidence sets under bounds on pairwise capture probabilities using two methods: test inversion bootstrap confidence intervals and profile likelihood confidence intervals. Simulation results demonstrate well-calibrated confidence sets for each method. In an extensive real-world study, we apply the new methodology to the problem of using heterogeneous survey data to estimate the number of people who inject drugs in Brussels, Belgium.

Cluster analysis via random partition distributions

AbstractHierarchical and k‐medoids clustering are deterministic clustering algorithms defined on pairwise distances. We use these same pairwise distances in a novel stochastic clustering procedure based on a probability distribution. We call our proposed method CaviarPD, a portmanteau from cluster analysis via random partition distributions. CaviarPD first samples clusterings from a distribution on partitions and then finds the best cluster estimate based on these samples using algorithms to minimize an expected loss. Using eight case studies, we show that our approach produces results as close to the truth as hierarchical and k‐medoids methods, and has the additional advantage of allowing for a probabilistic framework to assess clustering uncertainty. The method provides an intuitive graphical representation of clustering uncertainty through pairwise probabilities from partition samples. A software implementation of the method is available in the CaviarPD package for R.

Dirichlet process mixture models for regression discontinuity designs.

The regression discontinuity design is a quasi-experimental design that estimates the causal effect of a treatment when its assignment is defined by a threshold for a continuous variable. The regression discontinuity design assumes that subjects with measurements within a bandwidth around the threshold belong to a common population, so that the threshold can be seen as a randomising device assigning treatment to those falling just above the threshold and withholding it from those who fall below. Bandwidth selection represents a compelling decision for the regression discontinuity design analysis as results may be highly sensitive to its choice. A few methods to select the optimal bandwidth, mainly from the econometric literature, have been proposed. However, their use in practice is limited. We propose a methodology that, tackling the problem from an applied point of view, considers units' exchangeability, that is, their similarity with respect to measured covariates, as the main criteria to select subjects for the analysis, irrespectively of their distance from the threshold. We cluster the sample using a Dirichlet process mixture model to identify balanced and homogeneous clusters. Our proposal exploits the posterior similarity matrix, which contains the pairwise probabilities that two observations are allocated to the same cluster in the Markov chain Monte Carlo sample. Thus we include in the regression discontinuity design analysis only those clusters for which we have stronger evidence of exchangeability. We illustrate the validity of our methodology with both a simulated experiment and a motivating example on the effect of statins on cholesterol levels.

A Novel Non-Coherent SCMA With Massive MIMO

The synergistic amalgamation of sparse code multiple access (SCMA) and multiple-input multiple-output (MIMO) technologies can be exploited for improving spectral efficiency and providing enhanced wireless services to massive users. In this case, however, channel estimation is a burning issue with the increasing number of users and/or antennas. To tackle this problem, we propose a novel non-coherent transmission scheme for SCMA, referred to as NC-SCMA. In the proposed NC-SCMA, each user first maps its binary data to sparse codewords, and then perform differential modulation on the non-zero dimensions. Upon receiving all users’ signals, we leverage the channel hardening effect to carry out differential demodulation and multi-user detection without any instantaneous channel state information. In addition, the design of the sparse codebooks in the NC-SCMA system is investigated with the aid of the pair-wise probability. Numerical results demonstrate the superiority of the proposed technique over the benchmark scheme in terms of bit error rate performance.

Effect of Social Influence on a Two-Party Election: A Markovian Multiagent Model

In digital social networks, the filtering algorithms employed by the platform management to sieve the contents shared among the users can alter the social influence intensity. In this article, a Markov multiagent model of opinion dynamics is used to analyze possible opinion manipulation under apparently neutral interventions on the influence intensity. We consider a two-party election, whose voters, modeled as heterogeneous agents, are connected in a social network with an arbitrary topology. The equations describing the variance of the vote share, both in transient and steady state, are derived. The key is the solution of the second-order marginalization problem under the form of a numerically tractable characterization of pairwise joint probabilities of the voters’ opinions. In particular, these probabilities are computed by means of a Lyapunov-like matrix differential equation driven by first-order moments. This result is used to answer some important questions, like the possible nonmonotonic effect of the influence intensity on the vote volatility and the interplay of topology and individuals’ stubbornness to determine the electoral balance between two parties.

Comparison of a multitarget blood test to ultrasound and alpha-fetoprotein for hepatocellular carcinoma surveillance: Results of a network meta-analysis.

Ultrasound‐based surveillance has suboptimal sensitivity for early detection of hepatocellular carcinoma (HCC) in patients with cirrhosis. There are several emerging alternatives, including a novel multitarget HCC blood test (Mt‐HBT). We compared performance of mt‐HBT against ultrasound with or without alpha‐fetoprotein (AFP) for early HCC detection in patients with cirrhosis. Per the Preferred Reporting Items for Systematic Reviews and Meta‐Analysis (PRISMA) guidelines, two reviewers searched PubMed, Cochrane, Embase, and clinicaltrials.gov databases from January 1990 through December 2020 to identify studies reporting sensitivity and/or specificity of ultrasound and AFP for overall and early stage HCC detection in patients with cirrhosis. Mt‐HBT diagnostic performance was derived from a clinical validation study. A network meta‐analysis model was built for comparative assessment, and pooled estimates of sensitivity at a fixed specificity were estimated based on Bayesian binormal receiver operating characteristic models for each modality. Forty‐one studies (comprising 62,517 patients with cirrhosis) met inclusion criteria. Ultrasound‐alone sensitivity was 51.6% (95% credible interval [CrI], 43.3%–60.5%) for early stage HCC detection, which increased with the addition of AFP to 74.1% (95% CrI, 62.6%–82.4%); however, this was offset by decreased specificity (87.9% vs. 83.9%, respectively). With specificity fixed at 90%, mt‐HBT sensitivity for early stage HCC detection was higher than ultrasound alone (18.2%; 95% CrI, 0.2%–37.7%) and similar to ultrasound with AFP (−3.3%; 95% CrI, −22.3%–17.4%). Pairwise posterior probabilities suggested a preference for mt‐HBT over ultrasound alone in 97.4% of cases but only 36.3% of cases versus ultrasound with AFP. Conclusion: A blood‐based mt‐HBT has higher sensitivity than ultrasound alone for early stage HCC detection but similar sensitivity compared to ultrasound and AFP. Mt‐HBT could be a comparable alternative to existing methods for HCC surveillance in patients who are at risk.

The completed SDSS-IV extended Baryon Oscillation Spectroscopic Survey: measurement of the growth rate of structure from the small-scale clustering of the luminous red galaxy sample

ABSTRACT We measure the small-scale clustering of the Data Release 16 extended Baryon Oscillation Spectroscopic Survey Luminous Red Galaxy sample, corrected for fibre-collisions using Pairwise Inverse Probability weights, which give unbiased clustering measurements on all scales. We fit to the monopole and quadrupole moments and to the projected correlation function over the separation range $7-60\, h^{-1}{\rm Mpc}$ with a model based on the aemulus cosmological emulator to measure the growth rate of cosmic structure, parametrized by fσ8. We obtain a measurement of fσ8(z = 0.737) = 0.408 ± 0.038, which is 1.4σ lower than the value expected from 2018 Planck data for a flat ΛCDM model, and is more consistent with recent weak-lensing measurements. The level of precision achieved is 1.7 times better than more standard measurements made using only the large-scale modes of the same sample. We also fit to the data using the full range of scales $0.1\text{--}60\, h^{-1}{\rm Mpc}$ modelled by the aemulus cosmological emulator and find a 4.5σ tension in the amplitude of the halo velocity field with the Planck + ΛCDM model, driven by a mismatch on the non-linear scales. This may not be cosmological in origin, and could be due to a breakdown in the Halo Occupation Distribution model used in the emulator. Finally, we perform a robust analysis of possible sources of systematics, including the effects of redshift uncertainty and incompleteness due to target selection that were not included in previous analyses fitting to clustering measurements on small scales.

DGCU: A new deep directed method based on Gaussian embedding for clustering uncertain graphs

Clustering uncertain graphs, with the aim of discovering the most reliable partitions, has many applications in real-world networks. In literature, uncertainty is applied somehow to the traditional graph clustering algorithms, so that probabilistic extensions of them are achieved. Due to the difficulty of this technique per algorithm, besides the increasing noise and sparsity followed by more graph dimensions, we propose a deep learning based method to overcome these challenges. As a preprocessing, we first generate a Probabilistic Proximity Matrix (PPM) and a Probabilistic Similarity Matrix (PSM) using the topological structures of the uncertain graph. Afterward, we benefit from Gaussian embedding to capture uncertainty beyond the efficient representation of nodes in a low-dimensional space. PPM is used to embed nodes in a Gaussian space, and PSM is utilized for embedding directed clustering in which a mutually objective function is employed to learn node embedding and clustering simultaneously. Finally, the representations can be partitioned by any deterministic graph clustering algorithm. Various experiments were accomplished on four real-world datasets. The results show more effectiveness of the proposed method than the state-of-the-art approaches in detecting co-cluster objects and well-separated clusters, also indicating high reliability with regard to the pairwise connection probabilities within the clusters.

White noise space analysis and multiplicative change of measures

In this paper, we display a family of Gaussian processes, with explicit formulas and transforms. This is presented with the use of duality tools in such a way that the corresponding path-space measures are mutually singular. We make use of a corresponding family of representations of the canonical commutation relations (CCR) in an infinite number of degrees of freedom. A key feature of our construction is explicit formulas for associated transforms; these are infinite-dimensional analogs of Fourier transforms. Our framework is that of Gaussian Hilbert spaces, reproducing kernel Hilbert spaces and Fock spaces. The latter forms the setting for our CCR representations. We further show, with the use of representation theory and infinite-dimensional analysis, that our pairwise inequivalent probability spaces (for the Gaussian processes) correspond in an explicit manner to pairwise disjoint CCR representations.

A multi-category intelligent method for the evaluation of visual comfort in underground space

With the increasing use of underground spaces for various functions, improving the comfort levels in such spaces has become imperative. Although this aspect has been largely addressed from a physical standpoint, psychological problems associated with underground spaces still remain a matter of concern. Different with the questionnaire survey-based method, a multi-category measurable method has been proposed herein for evaluating overall visual comfort in underground spaces, by combing the probability statistics methods and deep learning methods, the overall comfort perception related to the color, brightness, scale, spatial form can be finely classified and quantitatively evaluated. Firstly, a multi-category labeling (MCL) method probability based on statistics method is developed, the image comfort level can be more accurate labelled by repeatedly pairwise comparison and probability distribution update, and a dataset is also established for visual comfort labeling and intelligent evaluation. Then, the Swin Transformer is selected as the visual comfort evaluation algorithm, which serves for searching the common vision comfort features of the picture shooting at areas with different functions, and makes the model with more versatility and accuracy. Finally, the Wujiaochang underground space in Shanghai, China, is undertaken as a case study. The results prove that the proposed method can effectively improve the quantification and refinement of human perception and evaluation of comfort in underground spaces. This can eventually lead to the development of more comfortable underground spaces and avoid the psychological problems associated with such spaces.

Recognition Method of Matching Error between Dance Action and Music Beat Based on Data Mining

Aiming at the problems of low error recognition accuracy and low correlation of data matching in the existing dance action and music beat matching error recognition methods, a dance action and music beat matching error recognition method based on data mining is designed. The quaternion array is used to represent the joint coordinate points of the dance movement, and the coordinates are rotated to obtain the curve change of the dance movement. The dance movement characteristics are regarded as a set of positive and negative sample data, the initial weights of different sample data are calculated, the absolute value is processed for the music beat signal, the interference factors are filtered with the help of Gauss filter, and according to the rhythm law of the music beat signal, the characteristics of music beat signal are extracted and the feature extraction of dance action matching is completed with music beat. The corresponding relationship between dance action and music beat is regarded as the corresponding model, the pairwise occurrence probability between music beat and action is determined, the matching model between dance action and music beat is discretized, and the peak point of correlation data is introduced to complete the matching between dance action and music beat. The features of matching data are extracted and segmented by short-time Fourier transform, the segmented matching data is transformed into matching data, the matching error identification model is established with the help of support vector mechanism, and the constraint conditions of error detection are set to complete the matching error identification. The experimental results show that the proposed method has high recognition accuracy and high data matching correlation.