Please use this identifier to cite or link to this item:
http://bura.brunel.ac.uk/handle/2438/12596
Title: | Realistic assessment of software effort estimation models |
Authors: | Sigweni, B Shepperd, M Turchi, T |
Keywords: | Software engineering experimentation;Software effort estimation;Cross vaidation approaches |
Issue Date: | 2016 |
Publisher: | ACM |
Citation: | 20th International Conference on Evaluation and Assessment in Software Engineering (EASE 2016), Limerick, Ireland, (1 - 3 June 2016) |
Abstract: | Context: It is unclear that current approaches to evaluating or comparing competing software cost or effort models give a realistic picture of how they would perform in actual use. Specifically, we’re concerned that the usual practice of using all data with some holdout strategy is at variance with the reality of a data set growing as projects complete. Objective: This study investigates the impact of using unrealistic, though possibly convenient to the researchers, ways to compare models on commercial data sets. Our questions are does this lead to different conclusions in terms of the comparisons and if so,are the results biased e.g., more optimistic than those that might realistically be achieved in practice. Method: We compare a traditional approach based on leave one out cross-validation with growing the data set chronologically using the Finnish and Desharnais data sets. Results: Our realistic, time-based approach to validation is significantly more conservative than leave-one-out cross-validation (LOOCV) for both data sets. Conclusion: If we want our research to lead to actionable findings it’s incumbent upon the researchers to evaluate their models in realistic ways. This means a departure from LOOCV techniques, while further investigation is needed for other validation techniques, such as k-fold validation. |
URI: | http://bura.brunel.ac.uk/handle/2438/12596 |
DOI: | http://dx.doi.org/10.1145/2915970.2916005 |
ISBN: | 978-1-4503-3691-8/16/06 |
Appears in Collections: | Dept of Computer Science Research Papers |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Fulltext.pdf | File is embargoed until 06/06/2016 | 258.59 kB | Adobe PDF | View/Open |
Items in BURA are protected by copyright, with all rights reserved, unless otherwise indicated.