Abstract
Recurrent neural networks (RNNs) have been applied to a broad range of applications, including natural language processing, drug discovery, and video recognition. Their vulnerability to input perturbation is also known. Aligning with a view from software defect detection, this article aims to develop a coverage-guided testing approach to systematically exploit the internal behavior of RNNs, with the expectation that such testing can detect defects with high possibility. Technically, the long short-term memory network (LSTM), a major class of RNNs, is thoroughly studied. A family of three test metrics are designed to quantify not only the values but also the temporal relations (including both stepwise and bounded-length) exhibited when LSTM processing inputs. A genetic algorithm is applied to efficiently generate test cases. The test metrics and test case generation algorithm are implemented into a tool <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> , which is then evaluated on a set of LSTM benchmarks. Experiments confirm that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> has advantages over the state-of-the-art tool DeepStellar and attack-based defect detection methods, owing to its working with finer temporal semantics and the consideration of the naturalness of input perturbation. Furthermore, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">testRNN</small> enables meaningful information to be collected and exhibited for users to understand the testing results, which is an important step toward interpretable neural network testing.
Highlights
Feedforward neural networks (FNNs), notably convolutional neural networks (CNNs), are vulnerable in various safety and security scenarios, subject to adversarial attack [42], backdoor attack [17], data poisoning attack [36], privacy issues [37], etc
We focus on long short-term memory networks (LSTMs), which is the most important class of recurrent neural networks (RNNs), and design three long short term memory network (LSTM) structural coverage metrics, namely boundary coverage (BC), step-wise coverage (SC) and temporal coverage (TC)
We evaluate our TESTRNN approach with an extensive set of experiments from the following aspects: (1) the diversity of test cases generated under the guidance of the coverage metrics (Section VII-B), (2) the ability of detecting RNN defects (Section VII-C), (3) the effectiveness of the test case generation algorithms (Section VII-D), (4) the advantages over state-of-the-art attack tool [31] (Section VII-E), (5) the difference from state-of-the-art RNN testing tool DeepStellar [12] (Section VII-F), and (6) the exhibition of LSTM internal working mechanism (Section VII-G)
Summary
Feedforward neural networks (FNNs), notably convolutional neural networks (CNNs), are vulnerable in various safety and security scenarios, subject to adversarial attack [42], backdoor attack [17], data poisoning attack [36], privacy issues [37], etc. As stated in [5], a (software) program with high test coverage has more of its source code executed during testing, which suggests it has a lower chance of containing undetected software defects compared to a program with low test coverage We concur with this view, and suggest that, instead of identifying a particular type of defects such as adversarial samples, coverage-guided testing is to generate a set of test cases as diversified as possible while preserving the naturalness, in order to exploit the internal behaviour of the neural networks that has real operational impact.
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
