The Comprehensive Performance Rating for Hadoop Clusters on Cloud Computing Platform
Keywords:
Hadoop, mapreduce, data locality, principal component analysis.Abstract
Virtualization platform solutions throughout the IT infrastructure are one important type of Green IT services in cloud data center. The Hadoop clusters composed of on-demand virtual infrastructure are used as an effective implementation of MapReduce for developing data intensive applications in cloud computing. Deploying Hadoop clusters on large numbers of data center virtual machines (VMs) can significantly increase its productivity and reduce both energy and resource consumption. However, the interference between the VMs is complicated and changing with the growth of data size. On the
other hand, it would also decrease the performance of Map and
Reduce tasks while using Hadoop clusters on virtual machines. In this paper, a Comprehensive Performance Rating (CPR) scheme is presented to probe the root causes of these problems, and solutions for VMs interference are introduced using data locality and excessive configuration parameters. Unlike previous solutions by customizing Hadoop native job scheduler, the proposed CPR scheme uses Hadoop configuration metrics revealed through Principal Component Analysis (PCA) method to guide the performance tuning work. The experimental data is resulted on a 20-node virtual cluster demonstration. The proposed CPR scheme performance is close to the measured execution time in different data size, cluster size and map tasks ratio.
Downloads
Downloads
Published
Issue
Section
License
You are free to:
- Share — copy and redistribute the material in any medium or format for any purpose, even commercially.
- Adapt — remix, transform, and build upon the material for any purpose, even commercially.
- The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms:
- Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
Notices:
You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation .
No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.
