TY - GEN
T1 - Cloud versus in-house cluster
T2 - State of the Practice Reports, SC'11
AU - Zhai, Yan
AU - Liu, Mingliang
AU - Zhai, Jidong
AU - Ma, Xiaosong
AU - Chen, Wenguang
PY - 2011
Y1 - 2011
N2 - The emergence of cloud services brings new possibilities for constructing and using HPC platforms. However, while cloud services provide the exibility and convenience of customized, pay-as-you-go parallel computing, multiple previous studies in the past three years have indicated that cloudbased clusters need a significant performance boost to become a competitive choice, especially for tightly coupled parallel applications. In this work, we examine the feasibility of running HPC applications in clouds. This study distinguishes itself from existing investigations in several ways: 1) We carry out a comprehensive examination of issues relevant to the HPC community, including performance, cost, user experience, and range of user activities. 2) We compare an Amazon EC2-based platform built upon its newly available HPCoriented virtual machines with typical local cluster and supercomputer options, using benchmarks and applications with scale and problem size unprecedented in previous cloud HPC studies. 3) We perform detailed performance and scalability analysis to locate the chief limiting factors of the state-of-the-art cloud based clusters. 4) We present a case study on the impact of per-application parallel I/O system configuration uniquely enabled by cloud services. Our results reveal that though the scalability of EC2-based virtual clusters still lags behind traditional HPC alternatives, they are rapidly gaining in overall performance and cost-effectiveness, making them feasible candidates for performing tightly coupled scientific computing. In addition, our detailed benchmarking and profiling discloses and analyzes several problems regarding the performance and performance stability on EC2.
AB - The emergence of cloud services brings new possibilities for constructing and using HPC platforms. However, while cloud services provide the exibility and convenience of customized, pay-as-you-go parallel computing, multiple previous studies in the past three years have indicated that cloudbased clusters need a significant performance boost to become a competitive choice, especially for tightly coupled parallel applications. In this work, we examine the feasibility of running HPC applications in clouds. This study distinguishes itself from existing investigations in several ways: 1) We carry out a comprehensive examination of issues relevant to the HPC community, including performance, cost, user experience, and range of user activities. 2) We compare an Amazon EC2-based platform built upon its newly available HPCoriented virtual machines with typical local cluster and supercomputer options, using benchmarks and applications with scale and problem size unprecedented in previous cloud HPC studies. 3) We perform detailed performance and scalability analysis to locate the chief limiting factors of the state-of-the-art cloud based clusters. 4) We present a case study on the impact of per-application parallel I/O system configuration uniquely enabled by cloud services. Our results reveal that though the scalability of EC2-based virtual clusters still lags behind traditional HPC alternatives, they are rapidly gaining in overall performance and cost-effectiveness, making them feasible candidates for performing tightly coupled scientific computing. In addition, our detailed benchmarking and profiling discloses and analyzes several problems regarding the performance and performance stability on EC2.
UR - http://www.scopus.com/inward/record.url?scp=83055184887&partnerID=8YFLogxK
U2 - 10.1145/2063348.2063363
DO - 10.1145/2063348.2063363
M3 - Conference contribution
AN - SCOPUS:83055184887
SN - 9781450311397
T3 - State of the Practice Reports, SC'11
BT - State of the Practice Reports, SC'11
Y2 - 12 November 2011 through 18 November 2011
ER -