SOFTWARE UNDER TEST DALAM PENELITIAN SOFTWARE TESTING: SEBUAH REVIEW

Ani - Rahmani, Joe Lian Min, S Suprihanto

Abstract


Software under Test (SUT) is an essential aspect of software testing research activities. Preparation of the SUT is not simple. It requires accuracy, completeness and will affect the quality of the research conducted. Currently, there are several ways to utilize an SUT in software testing research: building an own SUT, utilization of open source to build an SUT, and SUT from the repository utilization. This article discusses the results of SUT identification in many software testing studies. The research is conducted in a systematic literature review (SLR) using the Kitchenham protocol. The review process is carried out on 86 articles published in 2017-2020. The article was selected after two selection stages: the Inclusion and Exclusion Criteria and the quality assessment. The study results show that the trend of using open source is very dominant. Some researchers use open source as the basis for developing SUT, while others use SUT from a repository that provides ready-to-use SUT. In this context, utilization of the SUT from the software infrastructure repository (SIR) and Defect4J are the most significant choice of researchers.

Full Text:

PDF

References


Abbas, M., Inayat, I., Saadatmand, M., Jan, N., 2019. Requirements dependencies-based test case prioritization for extra-functional properties. Proc. - 2019 IEEE 12th Int. Conf. Softw. Testing, Verif. Valid. Work. ICSTW 2019 159–163.

Abdur, M., Abu, M., Saeed, M., 2018. Prioritizing Dissimilar Test Cases in Regression Testing using Historical Failure Data. Int. J. Comput. Appl. 180, 1–8.

Akour, M., Abuwardih, L., Alhindawi, N., Alshboul, A., 2018. Test Case Minimization using Genetic Algorithm: Pilot Study. 2018 8th Int. Conf. Comput. Sci. Inf. Technol. CSIT 2018 66–70.

Al-Sabbagh, K.W., Staron, M., Ochodek, M., Hebig, R., Meding, W., 2020. Selective Regression Testing based on Big Data: Comparing Feature Extraction Techniques. Proc. - 2020 IEEE 13th Int. Conf. Softw. Testing, Verif. Valid. Work. ICSTW 2020 322–329.

Alfieri, A., Castiglione, C., Pastore, E., 2020. A multi-objective tabu search algorithm for product portfolio selection: A case study in the automotive industry. Comput. Ind. Eng. 142, 106382.

Ali, S., Li, Y., Yue, T., Zhang, M., 2017. An empirical evaluation of mutation and crossover operators for multi-objective uncertainty-wise test minimization. Proc. - 2017 IEEE/ACM 10th Int. Work. Search-Based Softw. Testing, SBST 2017 21–27.

Anderson, J., Azizi, M., Salem, S., Do, H., 2019. On the use of usage patterns from telemetry data for test case prioritization. Inf. Softw. Technol. 113, 110–130.

Andrianto, I., Liem, M.M.I., Asnar, Y.D.W., 2018. Web application fuzz testing. Proc. 2017 Int. Conf. Data Softw. Eng. ICoDSE 2017 2018-Janua, 1–6.

Arrieta, A., Wang, S., Markiegi, U., Sagardui, G., Etxeberria, L., 2018. Employing Multi-Objective Search to Enhance Reactive Test Case Generation and Prioritization for Testing Industrial Cyber-Physical Systems. IEEE Trans. Ind. Informatics 14, 1055–1066.

Awasthi, U., Palmer, K.A., Bollas, G.M., 2020. Optimal test and sensor selection for active fault diagnosis using integer programming. J. Process Control 92, 202–211.

Azizi, M., Do, H., 2018a. A collaborative filtering recommender system for test case prioritization in web applications. Proc. ACM Symp. Appl. Comput. 1560–1567.

Azizi, M., Do, H., 2018b. Graphite: A Greedy Graph-Based Technique for Regression Test Case Prioritization. Proc. - 29th IEEE Int. Symp. Softw. Reliab. Eng. Work. ISSREW 2018 245–251.

Ba-quttayyan, B., Mohd, H., Yusof, Y., 2019. Regression Test Case Prioritization Frameworks: Challenges and Future Directions. Int. J. Recent Technol. Eng. 8, 8457–8462.

Bertolino, A., Miranda, B., Pietrantuono, R., Russo, S., 2017. Adaptive Coverage and Operational Profile-Based Testing for Reliability Improvement. Proc. - 2017 IEEE/ACM 39th Int. Conf. Softw. Eng. ICSE 2017 541–551.

Bosmans, S., Mercelis, S., Denil, J., Hellinckx, P., 2019. Testing IoT systems using a hybrid simulation based testing approach. Computing 101, 857–872.

Butool, R., Nadeem, A., Sindhu, M., Zaman, O.U., 2019. Improving requirements coverage in test case prioritization for regression testing. Proc. - 22nd Int. Multitopic Conf. INMIC 2019.

Chen, B., Song, J., Xu, P., Hu, X., Jiang, Z.M., 2018. An automated approach to estimating code coverage measures via execution logs. ASE 2018 - Proc. 33rd ACM/IEEE Int. Conf. Autom. Softw. Eng. 305–316.

Chi, J., Qu, Y., Zheng, Q., Yang, Z., Jin, W., Cui, D., Liu, T., 2020. Relation-based test case prioritization for regression testing. J. Syst. Softw. 163.

Choudhary, A., Agrawal, A.P., Kaur, A., 2018. An effective approach for regression test case selection using pareto based multi-objective harmony search. Proc. - Int. Conf. Softw. Eng. 13–20.

Chuaychoo, N., Kansomkeat, S., 2017. Path coverage test case generation using genetic algorithms. J. Telecommun. Electron. Comput. Eng. 9, 115–119.

Dadkhah, M., Araban, S., Paydar, S., 2020. A systematic literature review on semantic web enabled software testing. J. Syst. Softw. 162, 110485.

Di Nucci, D., Panichella, A., Zaidman, A., De Lucia, A., 2020. A Test Case Prioritization Genetic Algorithm Guided by the Hypervolume Indicator. IEEE Trans. Softw. Eng. 46, 674–696.

Eldrandaly, K., Ellatif, M.A., Zaki, N., 2020. A proposed framework for test suite prioritization and reduction using the clustering data mining technique. J. Theor. Appl. Inf. Technol. 98, 290–307.

Fu, W., Yu, H., Fan, G., Ji, X., Pei, X., 2018. A Regression Test Case Prioritization Algorithm Based on Program Changes and Method Invocation Relationship. Proc. - Asia-Pacific Softw. Eng. Conf. APSEC 2017-Decem, 169–178.

Garousi, V., Felderer, M., Kılıçaslan, F.N., 2019. A survey on software testability. Inf. Softw. Technol. 108, 35–64.

Hajri, I., Goknil, A., Pastore, F., Briand, L.C., 2020. Automating system test case classification and prioritization for use case-driven testing in product lines. Empir. Softw. Eng. 25, 3711–3769.

Halim, S.A., Jawawi, D.N.A., Sahak, M., 2019. Similarity distance measure and prioritization algorithm for test case prioritization in software product line testing. J. Inf. Commun. Technol. 18, 57–75.

Hettiarachchi, C., Do, H., 2019. A Systematic Requirements and Risks-Based Test Case Prioritization Using a Fuzzy Expert System. Proc. - 19th IEEE Int. Conf. Softw. Qual. Reliab. Secur. QRS 2019 374–385.

Hilton, M., Bell, J., Marinov, D., 2018. A large-scale study of test coverage evolution. ASE 2018 - Proc. 33rd ACM/IEEE Int. Conf. Autom. Softw. Eng. 53–63.

Huang, R., Sun, W., Chen, T.Y., Towey, D., Chen, J., Zong, W., Zhou, Y., 2020a. Abstract Test Case Prioritization Using Repeated Small-Strength Level-Combination Coverage. IEEE Trans. Reliab. 69, 349–372.

Huang, R., Zhang, Q., Towey, D., Sun, W., Chen, J., 2020b. Regression test case prioritization by code combinations coverage. J. Syst. Softw. 169, 110712.

Hynninen, T., Kasurinen, J., Knutas, A., Taipale, O., 2018. Software testing: Survey of the industry practices. 2018 41st Int. Conv. Inf. Commun. Technol. Electron. Microelectron. MIPRO 2018 - Proc. 1449–1454.

Jahan, H., Feng, Z., Mahmud, S.M.H., 2020. Risk-Based Test Case Prioritization by Correlating System Methods and Their Associated Risks. Arab. J. Sci. Eng. 45, 6125–6138.

Ji, S., Li, B., Zhang, P., 2019. Test Case Selection for All-Uses Criterion-Based Regression Testing of Composite Service. IEEE Access 7, 174438–174464.

Khanna, M., Chaudhary, A., Toofani, A., Pawar, A., 2019a. Performance Comparison of Multi-objective Algorithms for Test Case Prioritization During Web Application Testing. Arab. J. Sci. Eng. 44, 9599–9625.

Khanna, M., Chauhan, N., Sharma, D.K., 2019b. Search for prioritized test cases during web application testing. Int. J. Appl. Metaheuristic Comput. 10, 1–26.

Khatibsyarbini, M., Isa, M.A., Jawawi, D.N.A., 2017. A hybrid weight-based and string distances using particle swarm optimization for prioritizing test cases. J. Theor. Appl. Inf. Technol. 95, 2723–2732.

Khatibsyarbini, M., Isa, M.A., Jawawi, D.N.A., Hamed, H.N.A., Mohamed Suffian, M.D., 2019. Test Case Prioritization Using Firefly Algorithm for Software Testing. IEEE Access 7, 132360–132373.

Khatibsyarbini, M., Isa, M.A., Jawawi, D.N.A., Tumeng, R., 2018. Test case prioritization approaches in regression testing: A systematic literature review. Inf. Softw. Technol. 93, 74–93.

Kitchenham, B., Brereton, P., 2013. A systematic review of systematic review process research in software engineering. Inf. Softw. Technol. 55, 2049–2075.

Lei Xiao, Huaikou, Weiwei Zhuang, S.C., 2017. An Empirical Study on Clustering Approach Combining Fault Prediction for Test Case Prioritization. In: 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS). pp. 815–820.

Li, Y., Wang, J., Yang, Y., Wang, Q., 2020. An extensive study of class-level and method-level test case selection for continuous integration. J. Syst. Softw. 167, 110614.

Lin, J.W., Jabbarvand, R., Garcia, J., Malek, S., 2018. Nemo: Multi-criteria test-suite minimization with integer nonlinear programming. Proc. - Int. Conf. Softw. Eng. 1039–1049.

Lun, L., Wang, S., Chi, X., Xu, H., 2017. Automatic generation of basis component path coverage for software architecture testing. Comput. Informatics 36, 386–404.

Luo, L., 2000. Software Testing Techniques. In: Software Testing Techniques Technology Maturation and Research Strategy. Institute for Software Research International, Carnegie Mellon University.

Magalhães, C., Andrade, J., Perrusi, L., Mota, A., Barros, F., Maia, E., 2020. HSP: A hybrid selection and prioritisation of regression test cases based on information retrieval and code coverage applied on an industrial case study. J. Syst. Softw. 159.

Mahdieh, M., Mirian-Hosseinabadi, S.H., Etemadi, K., Nosrati, A., Jalali, S., 2020. Incorporating fault-proneness estimations into coverage-based test case prioritization methods. Inf. Softw. Technol. 121, 106269.

Marcozzi, M., Bardin, S., Kosmatov, N., Papadakis, M., Prevosto, V., Correnson, L., 2017. Freeing Testers from Polluting Test Objectives.

Matinnejad, R., Nejati, S., Briand, L.C., 2017. Automated testing of hybrid simulink/stateflow controllers: Industrial case studies. Proc. ACM SIGSOFT Symp. Found. Softw. Eng. Part F1301, 938–943.

Matinnejad, R., Nejati, S., Briand, L.C., Bruckmann, T., 2019. Test Generation and Test Prioritization for Simulink Models with Dynamic Behavior. IEEE Trans. Softw. Eng. 45, 919–944.

Min, J.L., Rajabi, N., Rahmani, A., 2021. Comprehensive study of SIR: Leading SUT repository for software testing. J. Phys. Conf. Ser. 1869.

Miranda, B., Cruciani, E., Verdecchia, R., Bertolino, A., 2018. FAST approaches to scalable similarity-based test case prioritization. Proc. - Int. Conf. Softw. Eng. 2018-Janua, 222–232.

Mishra, D.B., Mishra, R., Acharya, A.A., Das, K.N., 2019. Test case optimization and prioritization based on multi-objective genetic algorithm, Advances in Intelligent Systems and Computing. Springer Singapore.

Mohd-Shafie, M.L., Wan-Kadir, W.M.N., Khatibsyarbini, M., Isa, M.A., 2020. Model-based test case prioritization using selective and even-spread count-based methods with scrutinized ordering criterion. PLoS One 15, 1–27.

Mondal, S., Nasre, R., 2021. Summary of Hansie: Hybrid and consensus regression test prioritization. Proc. - 2021 IEEE 14th Int. Conf. Softw. Testing, Verif. Validation, ICST 2021 278–280.

Nayak, S., Kumar, C., Tripathi, S., 2017. Enhancing Efficiency of the Test Case Prioritization Technique by Improving the Rate of Fault Detection. Arab. J. Sci. Eng. 42, 3307–3323.

Ouriques, J.F.S., Cartaxo, E.G., Machado, P.D.L., 2018. Test case prioritization techniques for model-based testing: a replicated study. Softw. Qual. J. 26, 1451–1482.

Özener, O.Ö., Sözer, H., 2020. An effective formulation of the multi-criteria test suite minimization problem. J. Syst. Softw. 168, 110632.

Öztürk, M.M., 2017. A bat-inspired algorithm for prioritizing test cases. Vietnam J. Comput. Sci.

Pachariya, M.K., 2020. Building Ant System for Multi-Faceted Test Case Prioritization: An Empirical Study. Int. J. Softw. Innov. 8, 23–37.

Pradhan, D., Wang, S., Ali, S., Yue, T., Liaaen, M., 2019. Employing rule mining and multi-objective search for dynamic test case prioritization. J. Syst. Softw. 153, 86–104.

Rahmani, A., Min, J.L., Maspupah, A., 2020. An Evaluation of Code Coverage Adequacy in Automatic Testing using Control Flow Graph Visualization. ISCAIE 2020 - IEEE 10th Symp. Comput. Appl. Ind. Electron. 239–244.

Rahmani, A., Min, J.L., Maspupah, A., 2021. An empirical study of regression testing techniques. J. Phys. Conf. Ser. 1869.

Riskiawan, H.Y., Azhari, 2017. Automated software testing system using multi-agent system characteristics approach. Adv. Sci. Lett. 23, 2389–2391.

Shin, D., Yoo, S., Papadakis, M., Bae, D.H., 2019. Empirical evaluation of mutation-based test case prioritization techniques. Softw. Test. Verif. Reliab. 29, 1–28.

Shin, K.W., Lim, D.J., 2020. Model-based test case prioritization using an alternating variable method for regression testing of a UML-based model. Appl. Sci. 10, 1–23.

Shrivathsan, A.D., Ravichandran, K.S., Krishankumar, R., Sangeetha, V., Kar, S., Ziemba, P., Jankowski, J., 2019. Novel fuzzy clustering methods for test case prioritization in Software Projects. Symmetry (Basel). 11, 1–22.

Silva, D.S., Rabelo, R., Neto, P.S., Britto, R., Oliveira, P.A., 2019. A test case prioritization approach based on software component metrics. Conf. Proc. - IEEE Int. Conf. Syst. Man Cybern. 2019-Octob, 2939–2945.

Sinaga, A.M., 2017. Case study on testing of web-based application: Del’s students information system. J. Telecommun. Electron. Comput. Eng. 9, 1–5.

Singh, M., Srivastava, V.M., 2018. Extended firm mutation testing: A cost reduction technique for mutation testing. 2017 4th Int. Conf. Image Inf. Process. ICIIP 2017 2018-Janua, 604–609.

SIR Repository [WWW Document], n.d.

Someoliayi, K.E., Jalali, S., Mahdieh, M., Mirian-Hosseinabadi, S.H., 2019. Program State Coverage: A Test Coverage Metric Based on Executed Program States. SANER 2019 - Proc. 2019 IEEE 26th Int. Conf. Softw. Anal. Evol. Reengineering 584–588.

Su, W., Li, Z., Wang, Z., Yang, D., 2020. A Meta-heuristic Test Case Prioritization Method Based on Hybrid Model. Proc. - 2020 Int. Conf. Comput. Eng. Appl. ICCEA 2020 430–435.

Sun, Z., Zhang, J.M., Harman, M., Papadakis, M., Zhang, L., 2020. Automatic testing and improvement of machine translation. Proc. - Int. Conf. Softw. Eng. 974–985.

Taneja, D., Singh, R., Singh, A., Malik, H., 2020. A Novel technique for test case minimization in object oriented testing. Procedia Comput. Sci. 167, 2221–2228.

Tran, H.M., Le, S.T., Nguyen, S. Van, Ho, P.T., 2020. An Analysis of Software Bug Reports Using Machine Learning Techniques. SN Comput. Sci. 1.

Ulewicz, S., Vogel-heuser, B., Member, S., 2018. Prioritization in Production Automation 1–13.

Vasanthapriyan, S., Tian, J., Zhao, D., Xiong, S., Xiang, J., 2017. An ontology-based knowledge management system for software testing. Proc. Int. Conf. Softw. Eng. Knowl. Eng. SEKE 230–235.

Venugopal, Y., Quang-Ngoc, P., Eunseok, L., 2020. Modification point aware test prioritization and sampling to improve patch validation in automatic program repair. Appl. Sci. 10, 1–14.

Vescan, A., Şerban, C., Chisǎliţǎ-Creţu, C., Dioşan, L., 2017. Requirement dependencies-based formal approach for test case prioritization in regression testing. Proc. - 2017 IEEE 13th Int. Conf. Intell. Comput. Commun. Process. ICCP 2017 181–188.

Wang, H., Yang, M., Jiang, L., Xing, J., Yang, Q., Yan, F., 2020. Test Case Prioritization for Service-Oriented Workflow Applications: A Perspective of Modification Impact Analysis. IEEE Access 8, 101260–101273.

Wang, R., Li, Z., Jiang, S., Tao, C., 2020. Regression Test Case Prioritization Based on Fixed Size Candidate Set ART Algorithm. Int. J. Softw. Eng. Knowl. Eng. 30, 291–320.

Wang, Y., Zhu, Z., Yang, B., Guo, F., Yu, H., 2018. Using reliability risk analysis to prioritize test cases. J. Syst. Softw. 139, 14–31.

Wongwuttiwat, J., Lawanna, A., 2018. Performance improvement model of regression test selection. ICIIBMS 2017 - 2nd Int. Conf. Intell. Informatics Biomed. Sci. 2018-Janua, 49–53.

Yadav, D.K., Dutta, S., 2020. Regression test case selection and prioritization for object oriented software. Microsyst. Technol. 26, 1463–1477.

Yu, Z., Fahid, F., Menzies, T., Rothermel, G., Patrick, K., Cherian, S., 2019. TERMINATOR: Better automated UI test case prioritization. ESEC/FSE 2019 - Proc. 2019 27th ACM Jt. Meet. Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng. 883–894.

Zhang, T., Wang, X., Wei, D., Fang, J., 2018. Test Case Prioritization Technique Based on Error Probability and Severity of UML Models 28, 831–844.

Zhang, W., Qi, Y., Zhang, X., Wei, B., Zhang, M., Dou, Z., 2019. On test case prioritization using ant colony optimization algorithm. Proc. - 21st IEEE Int. Conf. High Perform. Comput. Commun. 17th IEEE Int. Conf. Smart City 5th IEEE Int. Conf. Data Sci. Syst. HPCC/SmartCity/DSS 2019 2767–2773.




DOI: https://doi.org/10.31884/jtt.v7i2.362

Refbacks

  • There are currently no refbacks.


Copyright (c) 2021 JTT (Jurnal Teknologi Terapan)

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

View Stats

 

 Creative Common Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)