New metrics for prioritized interaction test suites
Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in prac...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Published: |
Institute of Electronics, Information and Communication Engineers
2014
|
| Subjects: | |
| Online Access: | https://eprints.nottingham.ac.uk/47283/ |
| _version_ | 1848797507692265472 |
|---|---|
| author | Huang, Rubing Towey, Dave Chen, Jinfu Lu, Yansheng |
| author_facet | Huang, Rubing Towey, Dave Chen, Jinfu Lu, Yansheng |
| author_sort | Huang, Rubing |
| building | Nottingham Research Data Repository |
| collection | Online Access |
| description | Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCCλ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S. However, APCCλ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence - which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCCλ (NAPCC) to replace the APCCλ Additionally, to overcome the second drawback, we propose three new metrics: the Average Percentage of Strengths Satisfied (APSS); the Average Percentage of Weighted Multiple Interaction Coverage (APWMIC); and the Normalized APWMIC (NAPWMIC). These metrics comprehensively assess a given interaction test sequence by considering different interaction coverage at different strengths. Empirical studies show that the proposed metrics can be used to distinguish different interaction test sequences, and hence can be used to compare different test prioritization strategies. |
| first_indexed | 2025-11-14T20:04:59Z |
| format | Article |
| id | nottingham-47283 |
| institution | University of Nottingham Malaysia Campus |
| institution_category | Local University |
| last_indexed | 2025-11-14T20:04:59Z |
| publishDate | 2014 |
| publisher | Institute of Electronics, Information and Communication Engineers |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | nottingham-472832020-05-04T16:43:40Z https://eprints.nottingham.ac.uk/47283/ New metrics for prioritized interaction test suites Huang, Rubing Towey, Dave Chen, Jinfu Lu, Yansheng Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCCλ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S. However, APCCλ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence - which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCCλ (NAPCC) to replace the APCCλ Additionally, to overcome the second drawback, we propose three new metrics: the Average Percentage of Strengths Satisfied (APSS); the Average Percentage of Weighted Multiple Interaction Coverage (APWMIC); and the Normalized APWMIC (NAPWMIC). These metrics comprehensively assess a given interaction test sequence by considering different interaction coverage at different strengths. Empirical studies show that the proposed metrics can be used to distinguish different interaction test sequences, and hence can be used to compare different test prioritization strategies. Institute of Electronics, Information and Communication Engineers 2014-04-01 Article PeerReviewed Huang, Rubing, Towey, Dave, Chen, Jinfu and Lu, Yansheng (2014) New metrics for prioritized interaction test suites. IEICE Transactions on Information and Systems, E97.D (4). pp. 830-841. ISSN 1745-1361 combinatorial interaction testing; test case prioritization; prioritized interaction test suite (or interaction test sequence); interaction coverage; metrics https://www.jstage.jst.go.jp/article/transinf/E97.D/4/E97.D_830/_article doi:10.1587/transinf.E97.D.830 doi:10.1587/transinf.E97.D.830 |
| spellingShingle | combinatorial interaction testing; test case prioritization; prioritized interaction test suite (or interaction test sequence); interaction coverage; metrics Huang, Rubing Towey, Dave Chen, Jinfu Lu, Yansheng New metrics for prioritized interaction test suites |
| title | New metrics for prioritized interaction test suites |
| title_full | New metrics for prioritized interaction test suites |
| title_fullStr | New metrics for prioritized interaction test suites |
| title_full_unstemmed | New metrics for prioritized interaction test suites |
| title_short | New metrics for prioritized interaction test suites |
| title_sort | new metrics for prioritized interaction test suites |
| topic | combinatorial interaction testing; test case prioritization; prioritized interaction test suite (or interaction test sequence); interaction coverage; metrics |
| url | https://eprints.nottingham.ac.uk/47283/ https://eprints.nottingham.ac.uk/47283/ https://eprints.nottingham.ac.uk/47283/ |