• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>IEEE Transactions on Parallel and Distributed Systems >Cooper: Expedite Batch Data Dissemination in Computer Clusters with Coded Gossips
【24h】

Cooper: Expedite Batch Data Dissemination in Computer Clusters with Coded Gossips

机译：Cooper：使用编码八卦加快计算机群集中的批处理数据分发

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

Data transfers happen frequently in server clusters for software and application deployment, and in parallel computing clusters to transmit intermediate results in batches among servers between computation stages. This paper presents Cooper, an optimized prototype system to speedup multi-batch data transfers among a cluster of servers, leveraging a theoretically proven optimal algorithm called “coded permutation gossip,” which employs a simple random topology control scheme to best utilize bandwidth and decentralized random linear network coding to maximize the useful information transmitted. On a process-level coding-transfer pipeline, we investigate the best block division, batch division and inter-batch scheduling strategies to minimize the broadcast finish time in a realistic setting. For batch-based transfers, we propose a scheduling algorithm with low overhead that overlaps the transfers of consecutive batches and temporarily prioritizes later batches, to further reduce the broadcast finish time. We describe an asynchronous and distributed implementation of Cooper and have deployed it on Amazon EC2 for evaluation. Based on results from real experiments, we show that Cooper can almost double the speed of data transfers in computing clusters, as compared to state-of-the-art content distribution tools like BitTorrent, at a low CPU overhead.
机译：数据传输经常在服务器群集中进行软件和应用程序部署，而在并行计算群集中则经常在计算阶段之间的服务器之间批量传输中间结果。本文介绍了Cooper，这是一种优化的原型系统，可利用理论上证明有效的称为“编码置换八卦”的最佳算法来加速服务器集群之间的多批数据传输，该算法采用简单的随机拓扑控制方案来最佳利用带宽和分散式随机线性网络编码以最大化有用信息的传输。在过程级别的编码传输管道上，我们研究了最佳的块划分，批处理划分和批间调度策略，以在实际设置中最大程度地减少广播结束时间。对于基于批处理的传输，我们提出了一种开销较低的调度算法，该算法与连续批处理的传输重叠，并临时对以后的批处理进行优先级排序，以进一步减少广播结束时间。我们描述了Cooper的异步和分布式实现，并将其部署在Amazon EC2上进行评估。根据实际实验的结果，我们显示，与最新的内容分发工具（例如BitTorrent）相比，Cooper可以在计算集群中将数据传输速度提高近一倍，而CPU开销却很低。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号