Please use this identifier to cite or link to this item:
http://bura.brunel.ac.uk/handle/2438/11593
Title: | Using blind analysis for software engineering experiments |
Authors: | Sigweni, B Shepperd, M |
Keywords: | Researcher Bias;Blind analysis;Software engineering experimentation;Software e ort estimation |
Issue Date: | 2015 |
Publisher: | ACM |
Citation: | EASE '15 Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 32, Nanjing, China, (April 27 - 29, 2015) |
Abstract: | Context: In recent years there has been growing concern about conflicting experimental results in empirical software engineering. This has been paralleled by awareness of how bias can impact research results. Objective: To explore the practicalities of blind analysis of experimental results to reduce bias. Method : We apply blind analysis to a real software engineering experiment that compares three feature weighting approaches with a na ̈ıve benchmark (sample mean) to the Finnish software effort data set. We use this experiment as an example to explore blind analysis as a method to reduce researcher bias. Results: Our experience shows that blinding can be a relatively straightforward procedure. We also highlight various statistical analysis decisions which ought not be guided by the hunt for statistical significance and show that results can be inverted merely through a seemingly inconsequential statistical nicety (i.e., the degree of trimming). Conclusion: Whilst there are minor challenges and some limits to the degree of blinding possible, blind analysis is a very practical and easy to implement method that supports more objective analysis of experimental results. Therefore we argue that blind analysis should be the norm for analysing software engineering experiments. |
URI: | http://dl.acm.org/citation.cfm?id=2745832 http://bura.brunel.ac.uk/handle/2438/11593 |
DOI: | http://dx.doi.org/10.1145/2745802.2745832 |
ISBN: | 978-1-4503-3350-4 |
Appears in Collections: | Computer Science Dept of Computer Science Research Papers |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Fulltext.pdf | 263.8 kB | Adobe PDF | View/Open |
Items in BURA are protected by copyright, with all rights reserved, unless otherwise indicated.