• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>IEEE Transactions on Parallel and Distributed Systems >Breaking (Global) Barriers in Parallel Stochastic Optimization With Wait-Avoiding Group Averaging
【24h】

Breaking (Global) Barriers in Parallel Stochastic Optimization With Wait-Avoiding Group Averaging

机译：通过等待避免组平均分离并行随机优化中的（全球性）障碍

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

Deep learning at scale is dominated by communication time. Distributing samples across nodes usually yields the best performance, but poses scaling challenges due to global information dissemination and load imbalance across uneven sample lengths. State-of-the-art decentralized optimizers mitigate the problem, but require more iterations to achieve the same accuracy as their globally-communicating counterparts. We present Wait-Avoiding Group Model Averaging (WAGMA) SGD, a wait-avoiding stochastic optimizer that reduces global communication via subgroup weight exchange. The key insight is a combination of algorithmic changes to the averaging scheme and the use of a group allreduce operation. We prove the convergence of WAGMA-SGD, and empirically show that it retains convergence rates similar to Allreduce-SGD. For evaluation, we train ResNet-50 on ImageNet; Transformer for machine translation; and deep reinforcement learning for navigation at scale. Compared with state-of-the-art decentralized SGD variants, WAGMA-SGD significantly improves training throughput (e.g., 2.1x on 1,024 GPUs for reinforcement learning), and achieves the fastest time-to-solution (e.g., the highest score using the shortest training time for Transformer).
机译：尺度深入学习是由通信时间的主导。在节点上分布样本通常会产生最佳性能，但由于全球信息传播和在不均匀样本长度上的负载不平衡，造成缩放挑战。最先进的分散优化器减轻了这个问题，但需要更多的迭代来实现与全球通信对方相同的准确性。我们呈现等待避免的小组模型平均（WAGMA）SGD，等待避免随机优化器，通过子组权重交换来降低全局通信。关键洞察力是对平均方案的算法改变的组合，以及使用组已复发操作。我们证明了Wagma-SGD的融合，并经验证明它保留了类似于ALLREDUCE-SGD的收敛速率。对于评估，我们在想象中训练Reset-50;机器翻译变压器;和深度加强在规模上导航学习。与最先进的分散的SGD变体相比，WAGMA-SGD显着提高了培训吞吐量（例如，在1,024杆GPU上进行加固学习的2.1倍），并实现最快的解决时间（例如，使用最高分数变压器最短的培训时间）。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号