Logic-based Test Research Articles

Drowzee: Metamorphic Testing for Fact-Conflicting Hallucination Detection in Large Language Models

Large language models (LLMs) have revolutionized language processing, but face critical challenges with security, privacy, and generating hallucinations — coherent but factually inaccurate outputs. A major issue is fact-conflicting hallucination (FCH), where LLMs produce content contradicting ground truth facts. Addressing FCH is difficult due to two key challenges: 1) Automatically constructing and updating benchmark datasets is hard, as existing methods rely on manually curated static benchmarks that cannot cover the broad, evolving spectrum of FCH cases. 2) Validating the reasoning behind LLM outputs is inherently difficult, especially for complex logical relations. To tackle these challenges, we introduce a novel logic-programming-aided metamorphic testing technique for FCH detection. We develop an extensive and extensible framework that constructs a comprehensive factual knowledge base by crawling sources like Wikipedia, seamlessly integrated into Drowzee. Using logical reasoning rules, we transform and augment this knowledge into a large set of test cases with ground truth answers. We test LLMs on these cases through template-based prompts, requiring them to provide reasoned answers. To validate their reasoning, we propose two semantic-aware oracles that assess the similarity between the semantic structures of the LLM answers and ground truth. Our approach automatically generates useful test cases and identifies hallucinations across six LLMs within nine domains, with hallucination rates ranging from 24.7% to 59.8%. Key findings include LLMs struggling with temporal concepts, out-of-distribution knowledge, and lack of logical reasoning capabilities. The results show that logic-based test cases generated by Drowzee effectively trigger and detect hallucinations. To further mitigate the identified FCHs, we explored model editing techniques, which proved effective on a small scale (with edits to fewer than 1000 knowledge pieces). Our findings emphasize the need for continued community efforts to detect and mitigate model hallucinations.

Enhancing logic-based testing with EvoDomain: A search-based domain-oriented test suite generation approach

ContextEffective software testing requires test adequacy criteria. MC/DC, a widely used logic-based testing criterion, struggles to detect domain errors caused by incorrect arithmetic operations. Domain errors occur when test requirement boundaries shift or tilt, causing unpredictable behavior and system crashes. ObjectiveTo address the inadequacy of MC/DC in detecting domain errors, we present EvoDomain, a search-based testing technique. MethodEvoDomain uses a memetic algorithm combining genetic and hill-climbing algorithms, along with the DBSCAN clustering algorithm to select diversified boundary test data. The memetic algorithm is designed to efficiently enhance the search process for covering boundary test data. We compared EvoDomain with two logic-based testing approaches, a domain-oriented test suite generation approach, and random testing. ResultsEvaluations on 30 case studies show EvoDomain increases fault detection by 74.44% over MC/DC and 65.06% over RoRG. Additionally, EvoDomain improves support for different fault types by up to 68.89% for MC/DC and 66.33% for RoRG. Compared to COSMOS, which uses static analysis, EvoDomain improves the convergence effectiveness of identifying feasible subdomains by 32%. It offers high accuracy (0.99-1) and F1-score (0.99-1). EvoDomain finds the subdomains in less than 1/3 the time of Random search. ConclusionEvoDomain effectively generates domain-oriented test suites, enhancing the accuracy and effectiveness of fault detection.

Analytical Models for the Evaluation of Resistive Short Defect Detectability in Presence of Process Variations: Application to 28nm Bulk and FDSOI Technologies

This paper deals with the analysis of the impact of process variations on the detection of resistive short defects, in the context of a logic-based test. Two types of short defects are considered for our investigation, i.e. resistive short to either ground terminal (GND) or power supply terminal (VDD). For both defect types, simple analytical models are proposed that permit to evaluate the robust detectability range in presence of process variations. These models rely on a pre-characterization of the gate library through Monte-Carlo simulation and permit to evaluate the detectability range of a given defect without performing any fault simulation. These models are applied to perform a comparative analysis of 28 nm Bulk and FDSOI (Fully Depleted Silicon-On-Insulator) technologies, considering both regular-VT and low-VT devices. The influence of operating conditions on defect detectability range is also investigated.

What to expect of predicates: An empirical analysis of predicates in real world programs

One source of complexity in programs is logic expressions, i.e., predicates. Predicates define much of the functional behavior of the software. Many logic-based test criteria have been developed, including the active clause coverage (ACC) criteria and the modified condition/decision coverage (MCDC). The MCDC/ACC criteria is viewed as being expensive, which motivated us to evaluate the cost of applying these criteria using a basic proxy: the number of clauses. We looked at the frequency and percentage of predicates in 63 Java programs. Moreover, we also compared these Java programs with three programs in the safety-critical domain, in which logic-basic testing is often used. Although around 99% of the predicates within Java programs contain at most three clauses, there is a positive linear correlation between overall measures of size and the number of predicates that have more than three clauses. Furthermore, safety-critical C/C++ programs have more complex predicates than non-safety-critical programs. However, similar to the predicates in non-safety-critical programs, most predicates in safety-critical programs have up to three clauses. We conclude that non-safety-critical and safety-critical programs do not have many complex predicates. Thus, MCDC/ACC is only needed on a small fraction of the predicates.

Embedded Memory Interface Logic and Interconnect Testing

The increased size of embedded memory for system-on-chip (SoC) and multicore processors has a positive impact on performance yet poses a big challenge for chip yield, power consumption, and overall cost. Big percentage (>60%) of today’s processors and SoC area in both 2-D (planar) and 3-D technologies, such as through silicon via (TSV), are dedicated to memory. Most of today’s embedded memories are not as simple as a storage area with single interface of data, address, and control, but rather they compromise complex logic on their interface due to timing constrains and interconnect technologies (NoC and TSV). Memory core testing strategy is well understood and has mature tools and methodologies to screen for defects such as built-in-self-test (BIST). In addition, core and logic-based testing using scan and automatic test pattern generation (ATPG) tools and methodologies are intended for flop-based design. However, interface logic and complex interconnect like the one in 3-D chips are not thoroughly tested using BIST or ATPG as they are not designed for such logic. This becomes even more important for 3-D chips where a stack memory could have different testing strategies other than the base layer core which is interfacing with it. This brief presents a design for test methodology to achieve good coverage on interface logic for embedded and stack memory. The proposed approach uses modified ATPG and scan methodology to test the memory logic interface with minimum impact to existing design.

Passive performance testing of network protocols

Complementary to performance evaluation, performance testing of communicating protocols is a qualitative and quantitative test of a system, aiming at checking whether performance requirements of protocols have been satisfied under certain conditions. It raises an interesting issue of accurately formalizing specified performance requirements by taking consideration of data values of the protocol messages. In this paper, we present a novel logic-based testing approach to check protocol performance requirements through real execution traces and formally specified properties. In order to evaluate and assess our methodology, we develop a prototype and present experiments through a set of Session Initiation Protocol (SIP) properties. Finally, a performance benchmark method is proposed and relevant verdicts and discussions are provided.

Improving logic-based testing

Logic-based testers design tests from logical expressions that appear in software artifacts such as source code, design models, and requirements specifications. This paper presents three improvements to logic-based test design. First, in the context of mutation testing, we present fault hierarchies for the six relational operators. Applying the ROR mutation operator causes each relational operator to generate seven mutants per clause. The fault hierarchies show that only three of these seven mutants are needed. Second, we show how to bring the power of the ROR operator to logic-based test criteria such as the widely used Multiple Condition-Decision Coverage (MCDC) test criterion. Third, we present theoretical results supported by empirical data that show that the more recent coverage criterion of minimal-MUMCUT can find significantly more faults than MCDC. The paper has three specific recommendations: (1) Change the way the ROR mutation operator is defined in existing and future mutation systems. (2) Augment logic-based test criteria to incorporate relational operator replacement from mutation. (3) Replace the use of MCDC with minimal-MUMCUT, both in practice and in standards documents like FAA-DO178B.