• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文会议>International Joint Conference on Neural Networks >A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity
【24h】

A Novel Stochastic Stratified Average Gradient Method: Convergence Rate and Its Complexity

机译：一种新型随机分层平均梯度法：收敛速率及其复杂性

获取原文
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

SGD (Stochastic Gradient Descent) is a popular algorithm for large scale optimization problems due to its low iterative cost. However, SGD can not achieve linear convergence rate as FGD (Full Gradient Descent) because of the inherent gradient variance. To attack the problem, mini-batch SGD was proposed to get a trade-off in terms of convergence rate and iteration cost. In this paper, a general CVI (ConvergenceVariance Inequality) equation is presented to state formally the interaction of convergence rate and gradient variance. Then a novel algorithm named SSAG (Stochastic Stratified Average Gradient) is introduced to reduce gradient variance based on two techniques, stratified sampling and averaging over iterations that is a key idea in SAG (Stochastic Average Gradient). Furthermore, SSAG can achieve linear convergence rate of O((1 - μ/8C L)k) at smaller storage and iterative costs, where C ≥ 2 is the category number of training data. This convergence rate depends mainly on the variance between classes, but not on the variance within the classes. In the case of C ? N (N is the training data size), SSAG's convergence rate is much better than SAG's convergence rate of O((1 - μ/8N L)k). Our experimental results show SSAG outperforms SAG and many other algorithms.
机译：SGD（随机梯度下降）是一种由于其低迭代成本而具有大规模优化问题的流行算法。然而，由于固有的梯度方差，SGD不能实现线性会聚速率（完全梯度下降）。为了攻击问题，提出了迷你批处理SGD，以便在收敛速度和迭代成本方面获得权衡。在本文中，呈现了一般的CVI（收敛variance）方程式以正式的趋同率和梯度方差的相互作用。然后引入名为SSAG（随机分层平均梯度）的新颖算法以基于两种技术，分层采样和在凹陷中的关键思想（随机平均梯度）上的迭代的平均来降低梯度方差。此外，SSAG可以实现O的线性收敛速率（（1 - μ/ 8cl） k ）以较小的存储和迭代成本，其中C≥2是培训数据的类别数量。这种收敛速度主要取决于类之间的差异，但不是类内的方差。在C的情况下？ n（n是培训数据大小），SSAG的收敛速度远比SAG的o（（1 - μ/ 8nL）的收敛速度好 k ）。我们的实验结果表明SSAG优于SAG和许多其他算法。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号