Please use this identifier to cite or link to this item:
http://bura.brunel.ac.uk/handle/2438/29248
Full metadata record
DC Field | Value | Language |
---|---|---|
dc.contributor.author | Yao, J | - |
dc.contributor.author | Shepperd, M | - |
dc.date.accessioned | 2024-06-22T19:08:39Z | - |
dc.date.available | 2021-06-17 | - |
dc.date.available | 2024-06-22T19:08:39Z | - |
dc.date.issued | 2021-06-17 | - |
dc.identifier | ORCiD: Martin Shepperd https://orcid.org/0000-0003-1874-6145 | - |
dc.identifier | 106664 | - |
dc.identifier.citation | Yao, J. and . (2021) 'The impact of using biased performance metrics on software defect prediction research', Information and Software Technology, 139, 106664, pp. 1 - 14. doi: 10.1016/j.infsof.2021.106664. | en_US |
dc.identifier.issn | 0950-5849 | - |
dc.identifier.uri | https://bura.brunel.ac.uk/handle/2438/29248 | - |
dc.description | The article archived on this institutional repository is the accepted manuscript [v4], made available at arXiv:2103.10201v4 [cs.SE], https://arxiv.org/abs/2103.10201v4 under a CC BY-NC-SA license on Tue, 22 Jun 2021 09:50:19 UTC (3,233 KB). Comments: Accepted by the journal Information & Software Technology. It is a greatly extended version of "Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters" presented at EASE 2020. | - |
dc.description.abstract | Context: Software engineering researchers have undertaken many experiments investigating the potential of software defect prediction algorithms. Unfortunately some widely used performance metrics are known to be problematic, most notably F1, but nevertheless F1 is widely used. Objective: To investigate the potential impact of using F1 on the validity of this large body of research. Method: We undertook a systematic review to locate relevant experiments and then extract all pairwise comparisons of defect prediction performance using F1 and the unbiased Matthews correlation coefficient (MCC). Results: We found a total of 38 primary studies. These contain 12,471 pairs of results. Of these comparisons, 21.95% changed direction when the MCC metric is used instead of the biased F1 metric. Unfortunately, we also found evidence suggesting that F1 remains widely used in software defect prediction research. Conclusion: We reiterate the concerns of statisticians that the F1 is a problematic metric outside of an information retrieval context, since we are concerned about both classes (defect-prone and not defect-prone units). This inappropriate usage has led to a substantial number (more than one fifth) of erroneous (in terms of direction) results. Therefore we urge researchers to (i) use an unbiased metric and (ii) publish detailed results including confusion matrices such that alternative analyses become possible. | en_US |
dc.description.sponsorship | Jingxiu Yao wishes to acknowledge the support of the China Scholarship Council . | en_US |
dc.format.extent | 1 - 14 | - |
dc.format.medium | Print-Electronic | - |
dc.language | English | - |
dc.language.iso | en_US | en_US |
dc.publisher | Elsevier | en_US |
dc.relation.uri | https://arxiv.org/abs/2103.10201v4 | - |
dc.rights | Copyright © 2021 The Author(s). This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (https://creativecommons.org/licenses/by-nc-sa/4.0/). | - |
dc.rights.uri | https://creativecommons.org/licenses/by-nc-sa/4.0/ | - |
dc.subject | software engineering | en_US |
dc.subject | machine learning | en_US |
dc.subject | software defect prediction | en_US |
dc.subject | computational experiment | en_US |
dc.subject | classification metrics | en_US |
dc.title | The impact of using biased performance metrics on software defect prediction research | en_US |
dc.type | Article | en_US |
dc.identifier.doi | https://doi.org/10.1016/j.infsof.2021.106664 | - |
dc.relation.isPartOf | Information and Software Technology | - |
pubs.publication-status | Published | - |
pubs.volume | 139 | - |
dc.identifier.eissn | 1873-6025 | - |
dc.rights.license | https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode.en | - |
dc.rights.holder | The Author(s) | - |
Appears in Collections: | Dept of Computer Science Research Papers |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Preprint.pdf | 2.88 MB | Adobe PDF | View/Open |
This item is licensed under a Creative Commons License