gSched: A resource aware Hadoop scheduler for heterogeneous cloud computing environments

Abstract

MapReduce has become a major programming model for data-intensive applications in cloud computing environments. Hadoop, an open source implementation of MapReduce, has been adopted by an increasingly wide user community. However, Hadoop suffers from task scheduling performance degradation in heterogeneous contexts because of its homogeneous design focus. This paper presents gSched, a resource-aware Hadoop scheduler that takes into account both the heterogeneity of computing resources and provisioning charges in task allocation in cloud computing environments. gSched is initially evaluated in an experimental Hadoop cluster and demonstrates enhanced performance compared with the default Hadoop scheduler. Further evaluations are conducted on the Amazon EC2 cloud that demonstrates the effectiveness of gSched in task allocation in heterogeneous cloud computing environments.