Trustworthy Systems Research Articles

UNK-VQA: A Dataset and a Probe Into the Abstention Ability of Multi-Modal Large Models.

Teaching Visual Question Answering (VQA) models to refrain from answering unanswerable questions is necessary for building a trustworthy AI system. Existing studies, though have explored various aspects of VQA but somewhat ignored this particular attribute. This paper aims to bridge the research gap by contributing a comprehensive dataset, called UNK-VQA. The dataset is specifically designed to address the challenge of questions that models do not know. To this end, we first augment the existing data via deliberate perturbations on either the image or question. In specific, we carefully ensure that the question-image semantics remain close to the original unperturbed distribution. By this means, the identification of unanswerable questions becomes challenging, setting our dataset apart from others that involve mere image replacement. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models and discover their significant limitations when applied to our dataset. Additionally, we also propose a straightforward method to tackle these unanswerable questions. This dataset, we believe, will serve as a valuable benchmark for enhancing the abstention capability of VQA models, thereby leading to increased trustworthiness of AI systems. We have made the dataset available to facilitate further exploration in this area.

A Trust-Based Framework for Mitigating Poisoning Attacks in Federated Learning

Federated Learning (FL) is a decentralized machine learning approach that allows collaborative model training across multiple devices while preserving data privacy. However, its decentralized nature exposes it to poisoning attacks, where malicious clients corrupt model updates, compromising the global model’s integrity. This study introduces a trust-based framework that combines Krum, Differential Privacy (DP), and adaptive trust measures to counter these threats effectively. The framework dynamically assigns trust scores to clients based on their behavior, mitigating the impact of unreliable or malicious updates. Using the MNIST dataset, the framework’s robustness is tested against label flipping and gradient manipulation attacks at varying intensities. Results demonstrate the hybrid approach’s superior performance in accuracy, precision, recall, and F1-scores compared to standalone defenses, showcasing its adaptability and resilience. This research underscores the impor- tance of integrating robust, scalable defense mechanisms in FL to ensure secure, reliable, and trustworthy systems in adversarial settings

Interdependence and trust analysis (ITA): a framework for human–machine team design

ABSTRACT As machines' autonomy increases, the possibilities for collaboration between a human and a machine also increase. In particular, tasks may be performed with varying levels of interdependence, i.e. from independent to joint actions. The feasibility of each type of interdependence depends on factors that contribute to contextual trustworthiness, such as team members' competence, willingness and external factors. In this paper, we present the Interdependence and Trust Analysis (ITA) framework, which is an extension of Coactive Design's Interdependence Analysis framework (Johnson, M., J. M. Bradshaw, P. J. Feltovich, C. M. Jonker, M. Birna Van Riemsdijk, M. Sierhuis. 2014. Coactive Design: Designing Support for Interdependence in Joint Activity. Journal of Human-Robot Interaction 3 (1): 43–69. https://doi.org/10.5898/JHRI.3.1.Johnson). By including information on contextual trustworthiness, ITA can better support the design of human–machine teams, as well as task allocation and selection. Evaluated through expert interviews and a focus group involving a search and rescue scenario, ITA shows potential as a decision-making tool and a communication bridge among human and machine teammates. Our findings emphasise the need to define tasks and roles based on agent characteristics, and imply that decision-making models should align with human-centred objectives. ITA also highlights the trade-off between utility and effort when designing trustworthy systems, suggesting that guided conversations could improve the team design process. Finally, the ITA framework may improve transparency, justification, and interpretability in decision-making, contributing to appropriate trust among teammates.

Despite Contradictions, Artificial Intelligence and Block Chain Technology are Composing the Future Together

Now times, Block Chain technology and enhancements in Artificial Intelligence paradigm are facing more attention in the field of research with new advancements in many security techniques in all the sectors of IT and business. One cannot deny from the fact that the adoption of Artificial Intelligence paradigms and block-chain technology is shaping the new face of market. Beyond their contradictions both are proceeding at lightning speed. Both of them are aimed at providing something new to the world but the degree of complexity of each is quite different. With a secure, decentralized, and trustworthy system, block-chain technology did automate the bit-coin payments and provide the users an access to a shared ledger of records, transactions and data. With the useof smart contracts, a block-chain can also regulate the user interactions without the need for any central authority. In the contradiction, AI provides agents with the ability of reasoning, decision making and human-level intellect. At bottom one can say, block chain is concerned with keeping correct records, authentication and implementation while AI deals in conducting assessments, examining, and coming to a decision analyzing certain patterns and datasets, eventually giving rise to a self- directed interaction. Artificial Intelligence along with block chain allocates several descriptions which will make a seamless communication prospect certainly. AI and Block-Chain need sharing of data. The decentralized approach of database focuses on the significance of data sharing among various clients on a meticulous network. In the same way, AI depends very much on Big-Data, exclusively, for data distribution. The first section in the proposed article offers the Introduction part. The second section focuses on the majorly emphasized topics and domains about block chain technology indulging Artificial Intelligence into it, presents a detailed literature-review of many of the contributions by eminent research analysts. The third section describes the challenges in the collaborating AI with block-chain. The last section expresses a brief conclusion of the research work. Keywords- Block Chain Technology, Artificial Intelligence, AI, Machine Learning

Leveraging Tropical Algebra to Assess Trustworthy AI

Given the complexity of the application domain, the qualitative and quantifiable nature of the concepts involved, the wide heterogeneity and granularity of trustworthy attributes, and in some cases the non-comparability of the latter, assessing the trustworthiness of AI-based systems is a challenging process. In order to overcome these challenges, the Confiance.ai program proposes an innovative solution based on a Multi-Criteria Decision Aiding (MCDA) methodology. This approach involves several stages: framing trustworthiness as a set of well-defined attributes, exploring attributes to determine related Key Performance Indicators (KPI) or metrics, selecting evaluation protocols, and defining a method to aggregate multiple criteria to estimate an overall assessment of trust. This approach is illustrated by applying the RUM methodology (Robustness, Uncertainty, Monitoring) to ML context, while the focus on aggregation methods are based on Tropical Algebra.

Toward a responsible future: recommendations for AI-enabled clinical decision support.

Integrating artificial intelligence (AI) in healthcare settings has the potential to benefit clinical decision-making. Addressing challenges such as ensuring trustworthiness, mitigating bias, and maintaining safety is paramount. The lack of established methodologies for pre- and post-deployment evaluation of AI tools regarding crucial attributes such as transparency, performance monitoring, and adverse event reporting makes this situation challenging. This paper aims to make practical suggestions for creating methods, rules, and guidelines to ensure that the development, testing, supervision, and use of AI in clinical decision support (CDS) systems are done well and safely for patients. In May 2023, the Division of Clinical Informatics at Beth Israel Deaconess Medical Center and the American Medical Informatics Association co-sponsored a working group on AI in healthcare. In August 2023, there were 4 webinars on AI topics and a 2-day workshop in September 2023 for consensus-building. The event included over 200 industry stakeholders, including clinicians, software developers, academics, ethicists, attorneys, government policy experts, scientists, and patients. The goal was to identify challenges associated with the trusted use of AI-enabled CDS in medical practice. Key issues were identified, and solutions were proposed through qualitative analysis and a 4-month iterative consensus process. Our work culminated in several key recommendations: (1) building safe and trustworthy systems; (2) developing validation, verification, and certification processes for AI-CDS systems; (3) providing a means of safety monitoring and reporting at the national level; and (4) ensuring that appropriate documentation and end-user training are provided. AI-enabled Clinical Decision Support (AI-CDS) systems promise to revolutionize healthcare decision-making, necessitating a comprehensive framework for their development, implementation, and regulation that emphasizes trustworthiness, transparency, and safety. This framework encompasses various aspects including model training, explainability, validation, certification, monitoring, and continuous evaluation, while also addressing challenges such as data privacy, fairness, and the need for regulatory oversight to ensure responsible integration of AI into clinical workflow. Achieving responsible AI-CDS systems requires a collective effort from many healthcare stakeholders. This involves implementing robust safety, monitoring, and transparency measures while fostering innovation. Future steps include testing and piloting proposed trust mechanisms, such as safety reporting protocols, and establishing best practice guidelines.

The Trustworthiness Metric Model of Interface Based on Defects

The interface is a crucial element in component-based software, enabling the linkage of distinct components to facilitate interaction. Defects within the interface can significantly impact the overall trustworthiness of the system. Therefore, it is essential to assess the interface trustworthiness based on a defect-centric approach. This paper introduces a novel model for evaluating interface trustworthiness, anchored in defect analysis. First, the defect types are formalized based on interface specifications. Then, the comprehensive weight allocation method is established to characterize the importance degree of each interface defect type by combining the G1 and CRITIC methods. Subsequently, the attributes of the interface are evaluated by defect value analysis, and the trustworthiness measurement model of the interface is proposed based on these attributes. Furthermore, to evaluate the trustworthiness of the whole system, the trustworthiness measure models under different combination structure of components are established. Finally, the model’s’ applicability is demonstrated through an illustrative example. This trustworthiness evaluation from the interface view can guide interface designers to obtain high-quality interfaces and improve the trustworthiness of the entire software.

Subject-aware PET Denoising with Contrastive Adversarial Domain Generalization.

Recent advances in deep learning (DL) have greatly improved the performance of positron emission tomography (PET) denoising performance. However, DL model performance can vary a lot across subjects, due to the large variability of the count levels and spatial distributions. A generalizable DL model that mitigates the subject-wise variations is highly expected toward a reliable and trustworthy system for clinical application. In this work, we propose a contrastive adversarial learning framework for subject-wise domain generalization (DG). Specifically, we configure a contrastive discriminator in addition to the UNet-based denoising module to check the subject-related information in the bottleneck feature, while the denoising module is adversarially trained to enforce the extraction of subject-invariant features. The sampled low-count realizations from the list-mode data are used as anchor-positive pairs to be close to each other, while the other subjects are used as negative samples to be distributed far away. We evaluated on 97 18F-MK6240 tau PET studies, each having 20 noise realizations with 25% fractions of events. Training, validation, and testing were implemented using 1400, 120, and 420 pairs of 3D image volumes in a subject-independent manner. The proposed contrastive adversarial DG demonstrated superior denoising performance than conventional UNet without subject-wise DG and cross-entropy-based adversarial DG.

Building machines that learn and think with people.

What do we want from machine intelligence? We envision machines that are not just tools for thought but partners in thought: reasonable, insightful, knowledgeable, reliable and trustworthy systems that think with us. Current artificial intelligence systems satisfy some of these criteria, some of the time. In this Perspective, we show how the science of collaborative cognition can be put to work to engineer systems that really can be called 'thought partners', systems built to meet our expectations and complement our limitations. We lay out several modes of collaborative thought in which humans and artificial intelligence thought partners can engage, and we propose desiderata for human-compatible thought partnerships. Drawing on motifs from computational cognitive science, we motivate an alternative scaling path for the design of thought partners and ecosystems around their use through a Bayesian lens, whereby the partners we construct actively build and reason over models of the human and world.

A Formal Account of Trustworthiness: Connecting Intrinsic and Perceived Trustworthiness

This paper proposes a formal account of AI trustworthiness, connecting both intrinsic and perceived trustworthiness in an operational schematization. We argue that trustworthiness extends beyond the inherent capabilities of an AI system to include significant influences from observers' perceptions, such as perceived transparency, agency locus, and human oversight. While the concept of perceived trustworthiness is discussed in the literature, few attempts have been made to connect it with the intrinsic trustworthiness of AI systems. Our analysis introduces a novel schematization to quantify trustworthiness by assessing the discrepancies between expected and observed behaviors and how these affect perceived uncertainty and trust. The paper provides a formalization for measuring trustworthiness, taking into account both perceived and intrinsic characteristics. By detailing the factors that influence trust, this study aims to foster more ethical and widely accepted AI technologies, ensuring they meet both functional and ethical criteria.

Explainable deep learning approach for advanced persistent threats (APTs) detection in cybersecurity: a review

In recent years, Advanced Persistent Threat (APT) attacks on network systems have increased through sophisticated fraud tactics. Traditional Intrusion Detection Systems (IDSs) suffer from low detection accuracy, high false-positive rates, and difficulty identifying unknown attacks such as remote-to-local (R2L) and user-to-root (U2R) attacks. This paper addresses these challenges by providing a foundational discussion of APTs and the limitations of existing detection methods. It then pivots to explore the novel integration of deep learning techniques and Explainable Artificial Intelligence (XAI) to improve APT detection. This paper aims to fill the gaps in the current research by providing a thorough analysis of how XAI methods, such as Shapley Additive Explanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME), can make black-box models more transparent and interpretable. The objective is to demonstrate the necessity of explainability in APT detection and propose solutions that enhance the trustworthiness and effectiveness of these models. It offers a critical analysis of existing approaches, highlights their strengths and limitations, and identifies open issues that require further research. This paper also suggests future research directions to combat evolving threats, paving the way for more effective and reliable cybersecurity solutions. Overall, this paper emphasizes the importance of explainability in enhancing the performance and trustworthiness of cybersecurity systems.

Automated anomaly detection for categorical data by repurposing a form filling recommender system

Data quality is crucial in modern software systems, like data-driven decision support systems. However, data quality is affected by data anomalies, which represent instances that deviate from most of the data. These anomalies affect the reliability and trustworthiness of software systems, and may propagate and cause more issues. Although many anomaly detection approaches have been proposed, they mainly focus on numerical data. Moreover, the few approaches targeting anomaly detection for categorical data do not yield consistent results across datasets. In this paper, we propose a novel anomaly detection approach for categorical data named LAFF-AD (LAFF-based Anomaly Detection), which takes advantage of the learning ability of a state-of-the-art form filling tool (LAFF) to perform value inference on suspicious data. LAFF-AD runs a variant of LAFF that predicts the possible values of a suspicious categorical field in the suspicious instance. LAFF-AD then compares the output of LAFF to the recorded values in the suspicious instance, and uses a heuristic-based strategy to detect categorical data anomalies. We evaluated LAFF-AD by assessing its effectiveness and efficiency on six datasets. Our experimental results show that LAFF-AD can accurately determine a high range of data anomalies, with recall values between 0.6 and 1 and a precision value of at least 0.808. Furthermore, LAFF-AD is efficient, taking at most 7000 s and 735 ms to perform training and prediction, respectively.

HINTs: Sensemaking on large collections of documents with Hypergraph visualization and INTelligent agents.

Sensemaking on a large collection of documents (corpus) is a challenging task often found in fields such as market research, legal studies, intelligence analysis, political science, or computational linguistics. Previous works approach this problem from topic- and entity-based perspectives, but the capability of the underlying NLP model limits their effectiveness. Recent advances in prompting with LLMs present opportunities to enhance such approaches with higher accuracy and customizability. However, poorly designed prompts and visualizations could mislead users into falsely interpreting the visualizations and hinder the system's trustworthiness. In this paper, we address this issue by taking into account the user analysis tasks and visualization goals in the prompt-based data extraction stage, thereby extending the concept of Model Alignment. We present HINTs, a VA system for supporting sensemaking on large collections of documents, combining previous entity-based and topic-based approaches. The visualization pipeline of HINTs consists of three stages. First, entities and topics are extracted from the corpus with prompts. Then, the result is modeled as a hypergraph and hierarchically clustered. Finally, an enhanced space-filling curve layout is applied to visualize the hypergraph for interactive exploration. The system further integrates an LLM-based intelligent chatbot agent in the interface to facilitate the sensemaking of interested documents. To demonstrate the generalizability and effectiveness of the HINTs system, we present two case studies on different domains and a comparative user study. We report our insights on the behavior patterns and challenges when intelligent agents are used to facilitate sensemaking. We find that while intelligent agents can address many challenges in sensemaking, the visual hints that visualizations provide are still necessary. We discuss limitations and future work for combining interactive visualization and LLMs more profoundly to better support corpus analysis.

Explainable AI in Manufacturing and Industrial Cyber–Physical Systems: A Survey

This survey explores applications of explainable artificial intelligence in manufacturing and industrial cyber–physical systems. As technological advancements continue to integrate artificial intelligence into critical infrastructure and industrial processes, the necessity for clear and understandable intelligent models becomes crucial. Explainable artificial intelligence techniques play a pivotal role in enhancing the trustworthiness and reliability of intelligent systems applied to industrial systems, ensuring human operators can comprehend and validate the decisions made by these intelligent systems. This review paper begins by highlighting the imperative need for explainable artificial intelligence, and, subsequently, classifies explainable artificial intelligence techniques systematically. The paper then investigates diverse explainable artificial-intelligence-related works within a wide range of industrial applications, such as predictive maintenance, cyber-security, fault detection and diagnosis, process control, product development, inventory management, and product quality. The study contributes to a comprehensive understanding of the diverse strategies and methodologies employed in integrating explainable artificial intelligence within industrial contexts.

Ensuring fundamental rights compliance and trustworthiness of law enforcement AI systems: the ALIGNER Fundamental Rights Impact Assessment

Artificial intelligence systems can expand the capabilities and enhance the efficiency of law enforcement agencies preventing, investigating, detecting, and prosecuting criminal offences in the European Union. At the same time, the deployment of artificial intelligence in the security domain often raises numerous legal and ethical concerns. The ALIGNER Fundamental Rights Impact Assessment is an operational tool, rooted in fundamental rights and in the principles of AI ethics, ready to be integrated in the AI governance measures of European law enforcement agencies to inform their decision-making processes and ensure compliance with the recently adopted Artificial Intelligence Act. This paper first introduces the main tensions between law enforcement AI and fundamental rights, as enshrined in the Charter of Fundamental Rights of the European Union; then, it gives an overview of the main developments and best practices in AI governance and their relationship with fundamental rights as well as AI ethics; and finally, it describes the structure of the ALIGNER Fundamental Rights Impact Assessment.

Towards explainable fake news detection and automated content credibility assessment: Polish internet and digital media use-case

Detecting fake news and disinformation is a challenging and complex task. With the proliferation of the transformer architectures, researchers and practitioners have started actively using them to combat this phenomenon. In that regard, different approaches have been proposed by the community. Unfortunately, a significant number of solutions have started to treat misinformation in a similar way as other natural language processing challenges (e.g., sentiment classification). However, the reality is more complex, and the fake news detection problem often require additional context that may be essential to provide a complete assessment. Another pitfall in combating fake news is treating it as a binary classification challenge (e.g., fake vs. legitimate news). Combining such a simplified response with a deep black-box neural system effectively impacts the interpretability and eventual trustworthiness of the detection system. In this paper, the phenomenon of fake news is approached from a multi-factor perspective. Therefore, the proposed innovative approach does not limit news content rating to a one-dimensional binary response. In particular, the authors utilize a vast spectrum of processing techniques, including text embedding, machine translation, and various users’ preferences to provide content assessment. The results are presented using a Polish digital media use-case.

Perceptions of multi-cancer early detection tests among communities facing barriers to health care.

Marginalized racial and ethnic groups and rural and lower income communities experience significant cancer inequities. Blood-based multi-cancer early detection tests (MCEDs) provide a simple and less invasive method to screen for multiple cancers at a single access point and may be an important strategy to reduce cancer inequities. In this qualitative study, we explored barriers and facilitators to MCED adoption among communities facing health care access barriers in Alaska, California, and Oregon. We used reflexive thematic analysis to analyze general barriers to cancer screening, MCED-specific barriers, facilitators of MCED adoption, and MCED communication strategies. We found barriers and facilitators to MCED adoption across 4 levels of the social-ecological model: (1) individual, (2) interpersonal, (3) health care system, and (4) societal. These included adverse psychological impacts, positive perceptions of MCEDs, information and knowledge about cancer screening, the quality of the patient-provider relationship, a lack of health care system trustworthiness, logistical accessibility, patient supports, and financial accessibility. Optimal MCED communication strategies included information spread through the medical environment and the community. These findings underscore the importance of understanding and addressing the multilevel factors that may influence MCED adoption among communities facing health care access barriers to advance health equity.

Trustworthiness for an Ultra-Wideband Localization Service.

Trustworthiness assessment is an essential step to assure that interdependent systems perform critical functions as anticipated, even under adverse conditions. In this paper, a holistic trustworthiness assessment framework for ultra-wideband self-localization is proposed, including the attributes of reliability, security, privacy, and resilience. Our goal is to provide guidance for evaluating a system's trustworthiness based on objective evidence, i.e., so-called trustworthiness indicators. These indicators are carefully selected through the threat analysis of the particular system under evaluation. Our approach guarantees that the resulting trustworthiness indicators correspond to chosen real-world threats. Moreover, experimental evaluations are conducted to demonstrate the effectiveness of the proposed method. While the framework is tailored for this specific use case, the process itself serves as a versatile template, which can be used in other applications in the domains of the Internet of Things or cyber-physical systems.

Assessing Digital Solutions Adoption in Small and Medium-Sized Enterprises in DRC

Purpose: This study assesses the adoption of digital solutions in small and medium-sized enterprises (SMEs) in the Democratic Republic of Congo. Methodology: A Kobocollect survey was conducted in 13 cities across the DRC over a one-month period, collecting data from 459 SMEs. The sample comprised 18.53% of individual SMEs led by women, 60.13% owned by men, and 21.35% led by associations. Findings: The results show that only 15.03% of respondents consider ICT and digitization to be a new topic, while 75.16% of companies' systems are not digitized. However, 68.85% of respondents are willing to integrate new technology into their business control systems by digitizing their accounting systems, with no significant difference observed across income ranges (p-value=0.107). Digital literacy of users is identified as a key factor influencing the adoption of digital solutions, which is facilitated by adapted solutions that meet SMEs' specific needs. Capacity building and system trustworthiness are also found to be important factors influencing adoption. Furthermore, belonging to a specific income range is found to influence the adoption of daily reporting systems and market pricing (p-value=0.107 and p-value=0.06363, respectively). Unique Contribution to Theory, Policy and Practice: The study highlights the importance of investing in SME digitization and the need for adapted solutions that meet management needs and price qualifications.

Predictive analytics for financial compliance: Machine learning concepts for fraudulent transaction identification

Predictive analytics has emerged as a pivotal tool in financial compliance, offering sophisticated methods for identifying fraudulent transactions through the application of machine learning (ML) concepts. As financial institutions grapple with increasingly complex fraud schemes and stringent regulatory requirements, the integration of predictive analytics with ML provides a proactive approach to fraud detection and prevention. Machine learning algorithms excel in analyzing vast datasets, identifying hidden patterns, and making real-time predictions. In the realm of financial compliance, supervised learning models such as logistic regression, decision trees, and random forests are commonly used to classify transactions as legitimate or fraudulent. These models are trained on historical transaction data, learning to recognize the subtle indicators of fraud by identifying correlations between various features and fraudulent outcomes. This allows for high-accuracy predictions on new, unseen data. Unsupervised learning techniques, such as clustering and anomaly detection, are equally critical in predictive analytics for financial compliance. These methods do not require labeled data and are adept at uncovering novel fraud patterns by detecting outliers and irregularities that deviate from normal transactional behavior. Anomaly detection algorithms, including k-means clustering and isolation forests, can identify transactions that exhibit unusual characteristics, flagging them for further investigation. The integration of predictive analytics with real-time data processing capabilities enhances the agility of fraud detection systems. Streaming analytics and real-time scoring enable the continuous monitoring of transactions, ensuring that suspicious activities are identified and addressed promptly. This real-time aspect is crucial for minimizing the impact of fraudulent transactions and ensuring compliance with regulatory standards. Despite the advancements, implementing predictive analytics for financial compliance involves challenges such as ensuring data quality, addressing privacy concerns, and maintaining model transparency. Financial institutions must navigate these challenges by employing robust data governance practices, leveraging secure data processing techniques, and adopting explainable AI models that provide insights into their decision-making processes. In conclusion, predictive analytics, powered by machine learning concepts, offers a robust framework for identifying fraudulent transactions and enhancing financial compliance. By leveraging advanced ML algorithms and real-time data processing, financial institutions can proactively detect and prevent fraud, thereby safeguarding their operations and ensuring adherence to regulatory mandates. This approach not only mitigates financial losses but also strengthens the overall integrity and trustworthiness of the financial system.