How effectively does metamorphic testing alleviate the oracle problem?
In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, p...
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Published: |
Institute of Electrical and Electronics Engineers
2014
|
| Subjects: | |
| Online Access: | https://eprints.nottingham.ac.uk/51787/ |
| _version_ | 1848798574833303552 |
|---|---|
| author | Liu, Huai Kuo, Fei-Ching Towey, Dave Chen, Tsong Yueh |
| author_facet | Liu, Huai Kuo, Fei-Ching Towey, Dave Chen, Tsong Yueh |
| author_sort | Liu, Huai |
| building | Nottingham Research Data Repository |
| collection | Online Access |
| description | In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, properties of the software under test represented in the form of relations among inputs and outputs of multiple executions, to help verify the correctness of a program. This paper presents new empirical evidence to support this approach, which has been used to alleviate the oracle problem in various applications and to enhance several software analysis and testing techniques. It has been observed that identification of a sufficient number of appropriate metamorphic relations for testing, even by inexperienced testers, was possible with a very small amount of training. Furthermore, the cost-effectiveness of the approach could be enhanced through the use of more diverse metamorphic relations. The empirical studies presented in this paper clearly show that a small number of diverse metamorphic relations, even those identified in an ad hoc manner, had a similar fault-detection capability to a test oracle, and could thus effectively help alleviate the oracle problem. |
| first_indexed | 2025-11-14T20:21:56Z |
| format | Article |
| id | nottingham-51787 |
| institution | University of Nottingham Malaysia Campus |
| institution_category | Local University |
| last_indexed | 2025-11-14T20:21:56Z |
| publishDate | 2014 |
| publisher | Institute of Electrical and Electronics Engineers |
| recordtype | eprints |
| repository_type | Digital Repository |
| spelling | nottingham-517872020-05-04T16:41:05Z https://eprints.nottingham.ac.uk/51787/ How effectively does metamorphic testing alleviate the oracle problem? Liu, Huai Kuo, Fei-Ching Towey, Dave Chen, Tsong Yueh In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, properties of the software under test represented in the form of relations among inputs and outputs of multiple executions, to help verify the correctness of a program. This paper presents new empirical evidence to support this approach, which has been used to alleviate the oracle problem in various applications and to enhance several software analysis and testing techniques. It has been observed that identification of a sufficient number of appropriate metamorphic relations for testing, even by inexperienced testers, was possible with a very small amount of training. Furthermore, the cost-effectiveness of the approach could be enhanced through the use of more diverse metamorphic relations. The empirical studies presented in this paper clearly show that a small number of diverse metamorphic relations, even those identified in an ad hoc manner, had a similar fault-detection capability to a test oracle, and could thus effectively help alleviate the oracle problem. Institute of Electrical and Electronics Engineers 2014-01-31 Article PeerReviewed Liu, Huai, Kuo, Fei-Ching, Towey, Dave and Chen, Tsong Yueh (2014) How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering, 40 (1). pp. 4-22. ISSN 0098-5589 Software testing; test oracle; oracle problem; metamorphic testing; metamorphic relation https://ieeexplore.ieee.org/document/6613484/ doi:10.1109/tse.2013.46 doi:10.1109/tse.2013.46 |
| spellingShingle | Software testing; test oracle; oracle problem; metamorphic testing; metamorphic relation Liu, Huai Kuo, Fei-Ching Towey, Dave Chen, Tsong Yueh How effectively does metamorphic testing alleviate the oracle problem? |
| title | How effectively does metamorphic testing alleviate the oracle problem? |
| title_full | How effectively does metamorphic testing alleviate the oracle problem? |
| title_fullStr | How effectively does metamorphic testing alleviate the oracle problem? |
| title_full_unstemmed | How effectively does metamorphic testing alleviate the oracle problem? |
| title_short | How effectively does metamorphic testing alleviate the oracle problem? |
| title_sort | how effectively does metamorphic testing alleviate the oracle problem? |
| topic | Software testing; test oracle; oracle problem; metamorphic testing; metamorphic relation |
| url | https://eprints.nottingham.ac.uk/51787/ https://eprints.nottingham.ac.uk/51787/ https://eprints.nottingham.ac.uk/51787/ |