Post-hoc Methods Research Articles

Impact of low-dose free chlorine on the conjugative transfer of antibiotic resistance genes in wastewater effluents: Identifying key environmental factors for predictive modeling.

Reclaimed water disinfection results in the coexistence of antibiotic resistance genes (ARGs) and low-dose free chlorine in receiving environments. However, the impact of low-dose free chlorine on ARGs conjugative transfer and the key factors influencing the transfer under complex environmental conditions remain unclear, hindering the establishment of an effective monitoring system for resistance pollution in reclaimed water. This study investigated ARGs conjugative transfer under the influence of free chlorine at environmentally relevant concentrations and key interactive factors using machine learning models. The results showed that low-dose free chlorine (0.05-0.3 mg/L) promoted ARGs conjugative transfer, with 0.15 mg/L having a greater promoting effect than free chlorine concentrations of 0.05 and 0.3 mg/L. Additionally, different exposure patterns of low-dose chlorine affected ARGs conjugative transfer, with intermittent exposure posing a higher risk of ARGs dissemination. SVM linear model performed best in predicting ARGs conjugative transfer (RMSE=0.012, R2=0.975), and the SHapley Additive Explanations (SHAP) method revealed that key factors such as HCO3-, SAA, NO3-, and HA had positive SHAP values, indicating a positive influence on ARGs transfer under low-dose chlorine, making them the key features for predicting the ARGs conjugative transfer under the low-dose chlorine exposure. This study also revealed potential mechanisms of ARGs transfer under continuous low-dose free chlorine exposure, including intracellular reactive oxygen species (ROS), enzyme activity, cell membrane permeability, and gene expression. The integration of the machine learning model and post-hoc interpretation methods clarified the key drivers of ARGs conjugative transfer in reclaimed water-replenished environments, providing new insights for the safe reuse of reclaimed water and the development of river monitoring indicators.

Comparative Analysis of Multiple Comparison Tests for Radiation Exposure Subgroups Among Dental Workers At Usmanu Danfodiyo University Teaching Hospital (UDUTH), Sokoto, Nigeria

Post-hoc analysis is a crucial statistical technique employed to identify specific group differences following a significant ANOVA result. This study delves into the comparative analysis of three commonly used post-hoc tests - Tukey's HSD, LSD, and Scheffe's method - to determine significant differences in radiation exposure levels among dental workers at Usman Danfodiyo University Teaching Hospital, Sokoto, Nigeria. Dental professionals are routinely exposed to ionizing radiation, primarily from X-ray machines. Excessive exposure can lead to various health risks, including cancer. Hence, accurate assessment and comparison of radiation exposure levels among different categories of dental workers are essential for effective radiation protection measures. In this study, the researchers calculated the critical values for each post-hoc test at a significance level of 0.05. The results indicated that the LSD method had the smallest critical value (0.674), followed by Tukey's HSD (1.304) and Scheffe's method (1.566). A lower critical value generally implies a higher sensitivity in detecting significant differences between groups. By conducting pairwise comparisons, the study found that the LSD method was the most effective in identifying statistically significant differences in radiation exposure levels among the dental workers. This suggests that the LSD method is a suitable choice for post-hoc analysis in this specific context. The findings of this study have significant implications for radiation protection practices in dental settings. By employing appropriate post-hoc analysis methods, healthcare institutions can accurately assess radiation exposure risks and implement targeted interventions to minimize exposure and protect the health of dental workers.

Regularized Bayesian algorithms for Q-matrix inference based on saturated cognitive diagnosis modelling.

Q-matrices are crucial components of cognitive diagnosis models (CDMs), which are used to provide diagnostic information and classify examinees according to their attribute profiles. The absence of an appropriate Q-matrix that correctly reflects item-attribute relationships often limits the widespread use of CDMs. Rather than relying on expert judgment for specification and post-hoc methods for validation, there has been a notable shift towards Q-matrix estimation by adopting Bayesian methods. Nevertheless, their dependency on Markov chain Monte Carlo (MCMC) estimation requires substantial computational burdens and their exploratory tendency is unscalable to large-scale settings. As a scalable and efficient alternative, this study introduces the partially confirmatory framework within a saturated CDM, where the Q-matrix can be partially defined by experts and partially inferred from data. To address the dual needs of accuracy and efficiency, the proposed framework accommodates two estimation algorithms-an MCMC algorithm and a Variational Bayesian Expectation Maximization (VBEM) algorithm. This dual-channel approach extends the model's applicability across a variety of settings. Based on simulated and real data, the proposed framework demonstrated its robustness in Q-matrix inference.

Forecasting Bank Default Risk with Interpretable Machine Learning: The Study of Chinese Banks

ABSTRACT Bank occupies an important position in the financial system. The stable operation of the banking industry is not only one of the important factors in achieving sustainable economic growth but also related to the stability of the entire financial system. This research collects data from 507 banks in China from 2000 to 2021, uses the non-performing loan ratio as the measurement indicator of bank risk, and selects indicators from five levels (macroeconomic environment, industry economic environment, economic policy uncertainty, financial openness and bank financial status) On this basis, we use interpretable machine learning models to predict the bank’s default risk, analyze and compare the interpretable machine learning model and the post-hoc explainable methods. The results indicate that Provision Coverage (PC), Loan Provision Coverage (LPC), Liquidity Ratio (LR), and KOF Financial Globalization Index (KOFFiGI) have strong predictive capability for bank default risk. Our research can provide a reference for banks, government and financial regulatory authorities to construct the prediction model and indicator monitoring platform for bank default risk.

Out-of-distribution detection in digital pathology: Do foundation models bring the end to reconstruction-based approaches?

Artificial intelligence (AI) has shown promising results for computational pathology tasks. However, one of the limitations in clinical practice is that these algorithms are optimised for the distribution represented by the training data. For out-of-distribution (OOD) data, they often deliver predictions with equal confidence, even though these often are incorrect. In the pursuit of OOD detection in digital pathology, this study evaluates the state-of-the-art (SOTA) in computational pathology OOD detection, based on diffusion probabilistic models, specifically by adapting the latent diffusion model (LDM) for this purpose (AnoLDM). We compare this against post-hoc methods based on the latent space of foundation models, which are SOTA in general computer vision research. The approaches are not only evaluated on data from the same medical centres as the training set, but also on several datasets with data distribution shifts. The results show that AnoLDM performs similarly well or better than diffusion model based approaches published in previous studies in computational pathology but with reduced computational costs. However, our optimal configuration of an approach based on foundation models (kang_residual) outperforms AnoLDM on OOD detection on data not experiencing any covariate shifts, with an AUROC of 96.17 versus 91.86. Interestingly, AnoLDM is more successful at handling the data distribution shifts investigated in this study. However, both AnoLDM and kang_residual suffer substantial loss in the performance under the data distribution shifts, hence future work should focus on improving the generalisation of OOD detection for computational pathology applications.

An attribution graph-based interpretable method for CNNs

Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in various domains, such as face recognition, object detection, and image segmentation. However, the lack of transparency and limited interpretability inherent in CNNs pose challenges in fields such as medical diagnosis, autonomous driving, finance, and military applications. Several studies have explored the interpretability of CNNs and proposed various post-hoc interpretable methods. The majority of these methods are feature-based, focusing on the influence of input variables on outputs. Few methods undertake the analysis of parameters in CNNs and their overall structure. To explore the structure of CNNs and intuitively comprehend the role of their internal parameters, we propose an Attribution Graph-based Interpretable method for CNNs (AGIC) which models the overall structure of CNNs as graphs and provides interpretability from global and local perspectives. The runtime parameters of CNNs and feature maps of each image sample are applied to construct attribution graphs (At-GCs), where the convolutional kernels are represented as nodes and the SHAP values between kernel outputs are assigned as edges. These At-GCs are then employed to pretrain a newly designed heterogeneous graph encoder based on Deep Graph Infomax (DGI). To comprehensively delve into the overall structure of CNNs, the pretrained encoder is used for two types of interpretable tasks: (1) a classifier is attached to the pretrained encoder for the classification of At-GCs, revealing the dependency of At-GC’s topological characteristics on the image sample categories, and (2) a scoring aggregation (SA) network is constructed to assess the importance of each node in At-GCs, thus reflecting the relative importance of kernels in CNNs. The experimental results indicate that the topological characteristics of At-GC exhibit a dependency on the sample category used in its construction, which reveals that kernels in CNNs show distinct combined activation patterns for processing different image categories, meanwhile, the kernels that receive high scores from SA network are crucial for feature extraction, whereas low-scoring kernels can be pruned without affecting model performance, thereby enhancing the interpretability of CNNs.

Enhancing Interpretability of Vertebrae Fracture Grading using Human-interpretable Prototypes

Vertebral fracture grading classifies the severity of vertebral fractures, which is a challenging task in medical imaging and has recently attracted Deep Learning (DL) models. Only a few works attempted to make such models human-interpretable despite the need for transparency and trustworthiness in critical use cases like DL-assisted medical diagnosis. Moreover, such models either rely on post-hoc methods or additional annotations. In this work, we propose a novel interpretable-by-design method, ProtoVerse, to find relevant sub-parts of vertebral fractures (prototypes) that reliably explain the model’s decision in a human-understandable way. Specifically, we introduce a novel diversity-promoting loss to mitigate prototype repetitions in small datasets with intricate semantics. We have experimented with the VerSe’19 dataset and outperformed the existing prototype-based method. Further, our model provides superior interpretability against the post-hoc method. Importantly, expert radiologists validated the visual interpretability of our results, showing clinical applicability.

Unifying Fourteen Post-Hoc Attribution Methods With Taylor Interactions.

Various attribution methods have been developed to explain deep neural networks (DNNs) by inferring the attribution/importance/contribution score of each input variable to the final output. However, existing attribution methods are often built upon different heuristics. There remains a lack of a unified theoretical understanding of why these methods are effective and how they are related. Furthermore, there is still no universally accepted criterion to compare whether one attribution method is preferable over another. In this paper, we resort to Taylor interactions and for the first time, we discover that fourteen existing attribution methods, which define attributions based on fully different heuristics, actually share the same core mechanism. Specifically, we prove that attribution scores of input variables estimated by the fourteen attribution methods can all be mathematically reformulated as a weighted allocation of two typical types of effects, i.e., independent effects of each input variable and interaction effects between input variables. The essential difference among these attribution methods lies in the weights of allocating different effects. Inspired by these insights, we propose three principles for fairly allocating the effects, which serve as new criteria to evaluate the faithfulness of attribution methods. In summary, this study can be considered as a new unified perspective to revisit fourteen attribution methods, which theoretically clarifies essential similarities and differences among these methods. Besides, the proposed new principles enable people to make a direct and fair comparison among different methods under the unified perspective.

From the Lab to the Wild: Examining Generalizability of Video-based Mind Wandering Detection

AbstractStudent’s shift of attention away from a current learning task to task-unrelated thought, also called mind wandering, occurs about 30% of the time spent on education-related activities. Its frequent occurrence has a negative effect on learning outcomes across learning tasks. Automated detection of mind wandering might offer an opportunity to assess the attentional state continuously and non-intrusively over time and hence enable large-scale research on learning materials and responding to inattention with targeted interventions. To achieve this, an accessible detection approach that performs well for various systems and settings is required. In this work, we explore a new, generalizable approach to video-based mind wandering detection that can be transferred to naturalistic settings across learning tasks. Therefore, we leverage two datasets, consisting of facial videos during reading in the lab (N = 135) and lecture viewing in-the-wild (N = 15). When predicting mind wandering, deep neural networks (DNN) and long short-term memory networks (LSTMs) achieve F$$_{1}$$ 1 scores of 0.44 (AUC-PR = 0.40) and 0.459 (AUC-PR = 0.39), above chance level, with latent features based on transfer-learning on the lab data. When exploring generalizability by training on the lab dataset and predicting on the in-the-wild dataset, BiLSTMs on latent features perform comparably to the state-of-the-art with an F$$_{1}$$ 1 score of 0.352 (AUC-PR = 0.26). Moreover, we investigate the fairness of predictive models across gender and show based on post-hoc explainability methods that employed latent features mainly encode information on eye and mouth areas. We discuss the benefits of generalizability and possible applications.

Better Understanding Differences in Attribution Methods via Systematic Evaluations.

Deep neural networks are very successful on many vision tasks, but hard to interpret due to their black box nature. To overcome this, various post-hoc attribution methods have been proposed to identify image regions most influential to the models' decisions. Evaluating such methods is challenging since no ground truth attributions exist. We thus propose three novel evaluation schemes to more reliably measure the faithfulness of those methods, to make comparisons between them more fair, and to make visual inspection more systematic. To address faithfulness, we propose a novel evaluation setting (DiFull) in which we carefully control which parts of the input can influence the output in order to distinguish possible from impossible attributions. To address fairness, we note that different methods are applied at different layers, which skews any comparison, and so evaluate all methods on the same layers (ML-Att) and discuss how this impacts their performance on quantitative metrics. For more systematic visualizations, we propose a scheme (AggAtt) to qualitatively evaluate the methods on complete datasets. We use these evaluation schemes to study strengths and shortcomings of some widely used attribution methods over a wide range of models. Finally, we propose a post-processing smoothing step that significantly improves the performance of some attribution methods, and discuss its applicability.

Enhancing Explainability in Predictive Maintenance : Investigating the Impact of Data Preprocessing Techniques on XAI Effectiveness

In predictive maintenance, the complexity of the data often requires the use of Deep Learning models. These models, called “black boxes”, have proved their worth in predicting the Remaining Useful Life (RUL) of industrial machines. However, the inherent opacity of these models requires the incorporation of post-hoc explanation methods to enhance transparency. The quality of the explanations provided is then assessed using so-called evaluation metrics. Modeling is a whole process that includes an important data preprocessing phase, with parameter selection such as time window, smoothing parameter, or rectified RUL when dealing with multivariate time series dataset. We propose to analyze the impact of these preprocessing methods on the quality of explanations provided by the local post-hoc models LIME, KernelSHAP, and L2X, utilizing six evaluation metrics: stability, consistency, congruence, selectivity, completeness, and acumen. This analysis will be based on NASA's Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset with the LSTM model. Our findings reveal that the choice of specific pre-processing parameters can significantly improve predictive performance. Furthermore, the quality of explanations depends on the selection of explicability methods. In addition, a factorial analysis of the evaluation metrics reveals that they do not all point in the same direction.Indeed, understanding the nuanced relationships between evaluation metrics is essential for a comprehensive and accurate assessment of explainability methods.

Quantifying uncertainty in graph neural network explanations.

In recent years, analyzing the explanation for the prediction of Graph Neural Networks (GNNs) has attracted increasing attention. Despite this progress, most existing methods do not adequately consider the inherent uncertainties stemming from the randomness of model parameters and graph data, which may lead to overconfidence and misguiding explanations. However, it is challenging for most of GNN explanation methods to quantify these uncertainties since they obtain the prediction explanation in a post-hoc and model-agnostic manner without considering the randomness of graph data and model parameters. To address the above problems, this paper proposes a novel uncertainty quantification framework for GNN explanations. For mitigating the randomness of graph data in the explanation, our framework accounts for two distinct data uncertainties, allowing for a direct assessment of the uncertainty in GNN explanations. For mitigating the randomness of learned model parameters, our method learns the parameter distribution directly from the data, obviating the need for assumptions about specific distributions. Moreover, the explanation uncertainty within model parameters is also quantified based on the learned parameter distributions. This holistic approach can integrate with any post-hoc GNN explanation methods. Empirical results from our study show that our proposed method sets a new standard for GNN explanation performance across diverse real-world graph benchmarks.

Weakly-supervised preclinical tumor localization associated with survival prediction from lung cancer screening Chest X-ray images

In this paper, we hypothesize that it is possible to localize image regions of preclinical tumors in a Chest X-ray (CXR) image by a weakly-supervised training of a survival prediction model using a dataset containing CXR images of healthy patients and their time-to-death label. These visual explanations can empower clinicians in early lung cancer detection and increase patient awareness of their susceptibility to the disease. To test this hypothesis, we train a censor-aware multi-class survival prediction deep learning classifier that is robust to imbalanced training, where classes represent quantized number of days for time-to-death prediction. Such multi-class model allows us to use post-hoc interpretability methods, such as Grad-CAM, to localize image regions of preclinical tumors. For the experiments, we propose a new benchmark based on the National Lung Cancer Screening Trial (NLST) dataset to test weakly-supervised preclinical tumor localization and survival prediction models, and results suggest that our proposed method shows state-of-the-art C-index survival prediction and weakly-supervised preclinical tumor localization results. To our knowledge, this constitutes a pioneer approach in the field that is able to produce visual explanations of preclinical events associated with survival prediction results.

Explaining deep learning for ECG analysis: Building blocks for auditing and knowledge discovery

Deep neural networks have become increasingly popular for analyzing ECG data because of their ability to accurately identify cardiac conditions and hidden clinical factors. However, the lack of transparency due to the black box nature of these models is a common concern. To address this issue, explainable AI (XAI) methods can be employed. In this study, we present a comprehensive analysis of post-hoc XAI methods, investigating the glocal (aggregated local attributions over multiple samples) and global (concept based XAI) perspectives. We have established a set of sanity checks to identify saliency as the most sensible attribution method. We provide a dataset-wide analysis across entire patient subgroups, which goes beyond anecdotal evidence, to establish the first quantitative evidence for the alignment of model behavior with cardiologists’ decision rules. Furthermore, we demonstrate how these XAI techniques can be utilized for knowledge discovery, such as identifying subtypes of myocardial infarction. We believe that these proposed methods can serve as building blocks for a complementary assessment of the internal validity during a certification process, as well as for knowledge discovery in the field of ECG analysis.

Post-hoc vs ante-hoc explanations: xAI design guidelines for data scientists

The growing field of explainable Artificial Intelligence (xAI) has given rise to a multitude of techniques and methodologies, yet this expansion has created a growing gap between existing xAI approaches and their practical application. This poses a considerable obstacle for data scientists striving to identify the optimal xAI technique for their needs. To address this problem, our study presents a customized decision support framework to aid data scientists in choosing a suitable xAI approach for their use-case. Drawing from a literature survey and insights from interviews with five experienced data scientists, we introduce a decision tree based on the trade-offs inherent in various xAI approaches, guiding the selection between six commonly used xAI tools. Our work critically examines six prevalent ante-hoc and post-hoc xAI methods, assessing their applicability in real-world contexts through expert interviews. The aim is to equip data scientists and policymakers with the capacity to select xAI methods that not only demystify the decision-making process, but also enrich user understanding and interpretation, ultimately advancing the application of xAI in practical settings.

DIRECT: Dual Interpretable Recommendation with Multi-aspect Word Attribution

Recommending products to users with intuitive explanations helps improve the system in transparency, persuasiveness, and satisfaction. Existing interpretation techniques include post-hoc methods and interpretable modeling. The former category could quantitatively analyze input contribution to model prediction but has limited interpretation faithfulness, while the latter could explain model internal mechanisms but may not directly attribute model predictions to input features. In this study, we propose a novel D ual I nterpretable Rec ommenda t ion model called DIRECT, which integrates ideas of the two interpretation categories to inherit their advantages and avoid limitations. Specifically, DIRECT makes use of item descriptions as explainable evidence for recommendation. First, similar to the post-hoc interpretation, DIRECT could attribute the prediction of a user preference score to textual words of the item descriptions. The attribution of each word is related to its sentiment polarity and word importance, where a word is important if it corresponds to an item aspect that the user is interested in. Second, to improve the interpretability of embedding space, we propose to extract high-level concepts from embeddings, where each concept corresponds to an item aspect. To learn discriminative concepts, we employ a concept-bottleneck layer, and maximize the coding rate reduction on word-aspect embeddings by leveraging a word-word affinity graph extracted from a pre-trained language model. In this way, DIRECT simultaneously achieves faithful attribution and usable interpretation of embedding space. We also show that DIRECT achieves linear inference time complexity regarding the length of item reviews. We conduct experiments including ablation studies on five real-world datasets. Quantitative analysis, visualizations, and case studies verify the interpretability of DIRECT. Our code is available at: https://github.com/JacksonWuxs/DIRECT .

A Test Statistic Estimation-Based Approach for Establishing Self-Interpretable CNN-Based Binary Classifiers.

Interpretability is highly desired for deep neural network-based classifiers, especially when addressing high-stake decisions in medical imaging. Commonly used post-hoc interpretability methods have the limitation that they can produce plausible but different interpretations of a given model, leading to ambiguity about which one to choose. To address this problem, a novel decision-theory-inspired approach is investigated to establish a self-interpretable model, given a pre-trained deep binary black-box medical image classifier. This approach involves utilizing a self-interpretable encoder-decoder model in conjunction with a single-layer fully connected network with unity weights. The model is trained to estimate the test statistic of the given trained black-box deep binary classifier to maintain a similar accuracy. The decoder output image, referred to as an equivalency map, is an image that represents a transformed version of the to-be-classified image that, when processed by the fixed fully connected layer, produces the same test statistic value as the original classifier. The equivalency map provides a visualization of the transformed image features that directly contribute to the test statistic value and, moreover, permits quantification of their relative contributions. Unlike the traditional post-hoc interpretability methods, the proposed method is self-interpretable, quantitative. Detailed quantitative and qualitative analyses have been performed with three different medical image binary classification tasks.

Spatiotemporal models of dengue epidemiology in the Philippines: Integrating remote sensing and interpretable machine learning

Previous dengue epidemiological analyses have been limited in spatiotemporal extent or covariate dimensions, the latter neglecting the multifactorial nature of dengue. These constraints, caused by rigid and traditional statistical tools which collapse amidst ‘Big Data’, prompt interpretable machine-learning (iML) approaches. Predicting dengue incidence and mortality in the Philippines, a data-limited yet high-burden country, the mlr3 universe of R packages was used to build and optimize ML models based on remotely sensed provincial and dekadal 3 NDVI and 9 rainfall features from 2016 to 2020. Between two tasks, models differ across four random forest-based learners and two clustering strategies. Among 16 candidates, rfsrc-year-case and ranger-year-death significantly perform best for predicting dengue incidence and mortality, respectively. Therefore, temporal clustering yields the best models, reflective of dengue seasonality. The two best models were subjected to tripartite global exploratory model analyses, which encompass model-agnostic post-hoc methods such as Permutation Feature Importance (PFI) and Accumulated Local Effects (ALE). PFI reveals that the models differ in their important explanatory aspect, rainfall for rfsrc-year-case and NDVI for ranger-year-death, among which long-term average (lta) features are most relevant. Trend-wise, ALE reveals that average incidence predictions are positively associated with ‘Rain.lta’, reflective of dengue cases peaking during the wet season. In contrast, those for mortality are negatively associated with ‘NDVI.lta’, reflective of urban spaces driving dengue-related deaths. By technologically addressing the challenges of the human-animal-ecosystem interface, this study adheres to the One Digital Health paradigm operationalized under Sustainable Development Goals (SDGs). Leveraging data digitization and predictive modeling for epidemiological research paves SDG 3, which prioritizes holistic health and well-being.

Explaining Siamese networks in few-shot learning

Machine learning models often struggle to generalize accurately when tested on new class distributions that were not present in their training data. This is a significant challenge for real-world applications that require quick adaptation without the need for retraining. To address this issue, few-shot learning frameworks, which includes models such as Siamese Networks, have been proposed. Siamese Networks learn similarity between pairs of records through a metric that can be easily extended to new, unseen classes. However, these systems lack interpretability, which can hinder their use in certain applications. To address this, we propose a data-agnostic method to explain the outcomes of Siamese Networks in the context of few-shot learning. Our explanation method is based on a post-hoc perturbation-based procedure that evaluates the contribution of individual input features to the final outcome. As such, it falls under the category of post-hoc explanation methods. We present two variants, one that considers each input feature independently, and another that evaluates the interplay between features. Additionally, we propose two perturbation procedures to evaluate feature contributions. Qualitative and quantitative results demonstrate that our method is able to identify highly discriminant intra-class and inter-class characteristics, as well as predictive behaviors that lead to misclassification by relying on incorrect features.

Augmenting the Softmax with Additional Confidence Scores for Improved Selective Classification with Out-of-Distribution Data

Detecting out-of-distribution (OOD) data is a task that is receiving an increasing amount of research attention in the domain of deep learning for computer vision. However, the performance of detection methods is generally evaluated on the task in isolation, rather than also considering potential downstream tasks in tandem. In this work, we examine selective classification in the presence of OOD data (SCOD). That is to say, the motivation for detecting OOD samples is to reject them so their impact on the quality of predictions is reduced. We show under this task specification, that existing post-hoc methods perform quite differently compared to when evaluated only on OOD detection. This is because it is no longer an issue to conflate in-distribution (ID) data with OOD data if the ID data is going to be misclassified. However, the conflation within ID data of correct and incorrect predictions becomes undesirable. We also propose a novel method for SCOD, Softmax Information Retaining Combination (SIRC), that augments a softmax-based confidence score with a secondary class-agnostic feature-based score. Thus, the ability to identify OOD samples is improved without sacrificing separation between correct and incorrect ID predictions. Experiments on a wide variety of ImageNet-scale datasets and convolutional neural network architectures show that SIRC is able to consistently match or outperform the baseline for SCOD, whilst existing OOD detection methods fail to do so. Interestingly, we find that the secondary scores investigated for SIRC do not consistently improve performance on all tested OOD datasets. To address this issue, we further extend SIRC to incorporate multiple secondary scores (SIRC+). This further improves SCOD performance, both generally, and in terms of consistency over diverse distribution shifts. Code is available at https://github.com/Guoxoug/SIRC.