MachSMT: A Machine Learning-based Algorithm Selector for SMT Solvers

Joseph Scott,Mathias Preiner,Saeed Nejati,Aina Niemetz,Vijay Ganesh

doi:10.1007/978-3-030-72013-1_16

Joseph Scott, Mathias Preiner + Show 3 more

Open Access

https://doi.org/10.1007/978-3-030-72013-1_16

Copy DOI

Publication Date: Jan 1, 2021
Citations: 13	License type: CC BY 4.0

Affiliation: Stanford University, University of Waterloo

Abstract

In this paper, we present MachSMT, an algorithm selection tool for Satisfiability Modulo Theories (SMT) solvers. MachSMT supports the entirety of the SMT-LIB language. It employs machine learning (ML) methods to construct both empirical hardness models (EHMs) and pairwise ranking comparators (PWCs) over state-of-the-art SMT solvers. Given an SMT formula mathcal {I} as input, MachSMT leverages these learnt models to output a ranking of solvers based on predicted run time on the formula mathcal {I}. We evaluate MachSMT on the solvers, benchmarks, and data obtained from SMT-COMP 2019 and 2020. We observe MachSMT frequently improves on competition winners, winning 54 divisions outright and up to a 198.4% improvement in PAR-2 score, notably in logics that have broad applications (e.g., BV, LIA, NRA, etc.) in verification, program analysis, and software engineering. The MachSMT tool is designed to be easily tuned and extended to any suitable solver application by users. MachSMT is not a replacement for SMT solvers by any means. Instead, it is a tool that enables users to leverage the collective strength of the diverse set of algorithms implemented as part of these sophisticated solvers.

Highlights

Satisfiability Modulo Theories (SMT) solvers are tools to decide the satisfiability of formulas over first-order theories such as bit-vectors, floating-point arithmetic, integers, reals, strings, arrays, and their combinations [44,9,24,18,47,20,46]
Our results demonstrate that the MachSMT algorithm selector is highly effective, in that it outperforms the competition winners on the majority of tracks from the SMT-COMP in 2019 and 2020
We observe that MachSMT improves on competition winners in 54 divisions, with up to 198.4% improvement in performance for the QF BVFPLRA SQ ’20 and up to 191.1% for the QF BVFP SQ ’20 division

Summary

Introduction

Satisfiability Modulo Theories (SMT) solvers are tools to decide the satisfiability of formulas over first-order theories such as bit-vectors, floating-point arithmetic, integers, reals, strings, arrays, and their combinations [44,9,24,18,47,20,46]. SMT solvers have had a revolutionary impact on applications in software engineering (broadly construed), such as software testing [17,48] and verification [23,15,27,39], as well as in sub-fields of AI [53,35,30]. Researchers have long known that given a set of different algorithms and implementations for the same specification or problem, it is often the case that one of these implementations may perform poorly on a given class of inputs while another might perform very well. This is especially true for problems believed to be computationally hard (e.g., NP-hard). Greedy algorithm selection can be sub-optimal when the best empirical algorithm has deficiencies relative to other algorithms on certain families of inputs

Methods

Findings

Discussion

Conclusion