• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>IEEE Transactions on Parallel and Distributed Systems >Energy Efficiency Aware Task Assignment with DVFS in Heterogeneous Hadoop Clusters
【24h】

Energy Efficiency Aware Task Assignment with DVFS in Heterogeneous Hadoop Clusters

机译：异构Hadoop集群中使用DVFS的能效感知任务分配

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

While Hadoop ecosystems become increasingly important for practitioners of large-scale data analysis, they also incur tremendous energy cost. This trend is driving up the need for designing energy-efficient Hadoop clusters in order to reduce the operational costs and the carbon emission associated with its energy consumption. However, despite extensive studies of the problem, existing approaches for energy efficiency have not fully considered the heterogeneity of both workload and machine hardware found in production environments. In this paper, we find that heterogeneity-oblivious task assignment approaches are detrimental to both performance and energy efficiency of Hadoop clusters. Our observation shows that even heterogeneity-aware techniques that aim to reduce the job completion time do not guarantee a reduction in energy consumption of heterogeneous machines. We propose a heterogeneity-aware task assignment approach, E-Ant, that aims to improve the overall energy consumption in a heterogeneous Hadoop cluster without sacrificing job performance. It adaptively schedules heterogeneous workloads on energy-efficient machines, without a priori knowledge of the workload properties. E-Ant employs an ant colony optimization approach that generates task assignment solutions based on the feedback of each task's energy consumption reported by Hadoop TaskTrackers in an agile way. Furthermore, we integrate DVFS technique with E-Ant to further improve the energy efficiency of heterogeneous Hadoop clusters. It relies on a DVFS controller to dynamically scale the CPU frequency of each slave machine in response to time-varying resource demands. Experimental results on a heterogeneous cluster with varying hardware capabilities show that E-Ant with DVFS improves the overall energy savings for a synthetic workload from Microsoft by 23 and 17 percent compared to Fair Scheduler and Tarazu, respectively.
机译：尽管Hadoop生态系统对于大规模数据分析的实践者变得越来越重要，但它们也带来了巨大的能源成本。这种趋势推动了对设计节能型Hadoop集群的需求，以降低运营成本以及与其能耗相关的碳排放。但是，尽管对该问题进行了广泛的研究，但是现有的能源效率方法尚未完全考虑生产环境中工作负载和机器硬件的异质性。在本文中，我们发现，忽略异构性的任务分配方法不利于Hadoop集群的性能和能效。我们的观察结果表明，即使旨在减少作业完成时间的异构感知技术也不能保证降低异构机器的能耗。我们提出了一种异构感知任务分配方法E-Ant，其目的是在不牺牲作业性能的情况下改善异构Hadoop集群中的总体能耗。它可以在节能机器上自适应地调度异构工作负载，而无需事先了解工作负载属性。 E-Ant采用蚁群优化方法，该方法基于Hadoop TaskTrackers报告的每个任务能耗的反馈以敏捷方式生成任务分配解决方案。此外，我们将DVFS技术与E-Ant集成在一起，以进一步提高异构Hadoop集群的能效。它依靠DVFS控制器动态地缩放每个从属计算机的CPU频率，以响应随时间变化的资源需求。在具有不同硬件功能的异构群集上的实验结果表明，与Fair Scheduler和Tarazu相比，带有DVFS的E-Ant将Microsoft的合成工作负载的总能耗节省了23％和17％。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号