• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>Services Computing, IEEE Transactions on >An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing
【24h】

An Adaptive Efficiency-Fairness Meta-Scheduler for Data-Intensive Computing

机译：数据密集型计算的自适应效率-公平元调度程序

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

In data-intensive cluster computing platforms such as Hadoop YARN, efficiency and fairness are two important factors for system design and optimizations. Previous studies are either for efficiency or for fairness solely, without considering the tradeoff between efficiency and fairness. Recent studies observe that there is a tradeoff between efficiency and fairness because of resource contention between users/jobs. By leveraging the existing schedulers, a meta-scheduler is able to dynamically choose one of them for job/task scheduling at runtime. In this paper, we propose a meta-scheduler called FLEX to realize the tradeoff between system efficiency and fairness in Hadoop YARN. FLEX combines multiple existing schedulers into a single aggregated view without any modification on the original schedulers. Equipped with these candidate schedulers, FLEX utilizes machine learning approach to adaptively choose the most proper scheduler according to the characteristic of current running workload and user-defined Service Level Agreement (SLA). We implement FLEX in Hadoop YARN. We conduct experiments with real deployment in a local cluster and perform simulation studies with production traces. Experimental results show that the FLEX outperforms the state-of-the-art approach in two aspects: 1) Given a predefined threshold on the fairness loss, the FLEX reduces the makespan by up to 22 and 24 percent in real deployment and the large-scale simulation, respectively; 2) Given the predefined threshold on the makespan reduction, the FLEX reduces the fairness loss by up to 75 and 73 percent in real deployment and the large-scale simulation, respectively.
机译：在诸如Hadoop YARN之类的数据密集型集群计算平台中，效率和公平性是系统设计和优化的两个重要因素。先前的研究要么只是为了效率，要么是为了公平，而没有考虑效率与公平之间的权衡。最近的研究发现，由于用户/工作之间的资源争用，效率和公平之间存在权衡。通过利用现有的调度程序，元调度程序能够在运行时动态地选择其中一个进行作业/任务调度。在本文中，我们提出了一个称为FLEX的元调度程序，以实现Hadoop YARN中系统效率与公平性之间的折衷。 FLEX将多个现有调度程序合并到一个聚合视图中，而无需对原始调度程序进行任何修改。配备了这些候选调度程序后，FLEX根据当前运行的工作负载和用户定义的服务水平协议（SLA）的特征，利用机器学习方法来自适应地选择最合适的调度程序。我们在Hadoop YARN中实现FLEX。我们在本地集群中进行实际部署的实验，并使用生产跟踪进行模拟研究。实验结果表明，FLEX在两个方面都优于最新方法：1）给定公平损失阈值，在实际部署中，FLEX可将制造时间最多减少22％和24％，而在大型部署中，比例模拟2）给定预定义的制造期降低阈值，在实际部署和大规模仿真中，FLEX分别将公平性损失降低了75％和73％。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号