• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文会议>IEEE International Symposium on Computer Architecture and High Performance Computing >MR-Advisor: A Comprehensive Tuning Tool for Advising HPC Users to Accelerate MapReduce Applications on Supercomputers
【24h】

MR-Advisor: A Comprehensive Tuning Tool for Advising HPC Users to Accelerate MapReduce Applications on Supercomputers

机译：MR-Advisor：一个综合的调优工具，可建议HPC用户加速超级计算机上的MapReduce应用程序

获取原文
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

MapReduce is the most popular parallel computing framework for big data processing which allows massive scalability across distributed computing environment. Advanced RDMA-based design of Hadoop MapReduce has been proposed that alleviates the performance bottlenecks in default Hadoop MapReduce by leveraging the benefits from RDMA. On the other hand, data processing engine, Spark, provides fast execution of MapReduce applications through in-memory processing. Performance optimization for these contemporary big data processing frameworks on modern High-Performance Computing (HPC) systems is a formidable task because of the numerous configuration possibilities in each of them. In this paper, we propose MR-Advisor, a comprehensive tuning tool for MapReduce. MR-Advisor is generalized to provide performance optimizations for Hadoop, Spark, and RDMA-enhanced Hadoop MapReduce designs over different file systems such as HDFS, Lustre, and Tachyon. Performance evaluations reveal that, with MR-Advisor's suggested values, the job execution performance can be enhanced by a maximum of 58% over the current best-practice values for user-level configuration parameters. To the best of our knowledge, this is the first tool that supports tuning for both Apache Hadoop and Spark, as well as the RDMA and Lustre-based advanced designs.
机译：MapReduce是用于大数据处理的最流行的并行计算框架，该框架允许在分布式计算环境中实现大规模的可伸缩性。已经提出了基于高级RDMA的Hadoop MapReduce设计，该设计通过利用RDMA的好处缓解了默认Hadoop MapReduce中的性能瓶颈。另一方面，数据处理引擎Spark通过内存处理提供了MapReduce应用程序的快速执行。这些现代大数据处理框架在现代高性能计算（HPC）系统上的性能优化是一项艰巨的任务，因为它们每个中都有许多配置可能性。在本文中，我们提出了MR-Advisor，它是MapReduce的综合调整工具。 MR-Advisor可以通用化，从而为HDFS，Lustre和Tachyon等不同文件系统上的Hadoop，Spark和RDMA增强的Hadoop MapReduce设计提供性能优化。性能评估表明，使用MR-Advisor的建议值，与用户级别配置参数的当前最佳做法值相比，作业执行性能最多可以提高58％。据我们所知，这是第一个支持对Apache Hadoop和Spark以及基于RDMA和Lustre的高级设计进行调整的工具。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号