• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文期刊>Neural Networks and Learning Systems, IEEE Transactions on >Bayesian Recurrent Neural Network for Language Modeling
【24h】

Bayesian Recurrent Neural Network for Language Modeling

机译：贝叶斯递归神经网络的语言建模

获取原文
获取原文并翻译 | 示例
           
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

A language model (LM) is calculated as the probability of a word sequence that provides the solution to word prediction for a variety of information systems. A recurrent neural network (RNN) is powerful to learn the large-span dynamics of a word sequence in the continuous space. However, the training of the RNN-LM is an ill-posed problem because of too many parameters from a large dictionary size and a high-dimensional hidden layer. This paper presents a Bayesian approach to regularize the RNN-LM and apply it for continuous speech recognition. We aim to penalize the too complicated RNN-LM by compensating for the uncertainty of the estimated model parameters, which is represented by a Gaussian prior. The objective function in a Bayesian classification network is formed as the regularized cross-entropy error function. The regularized model is constructed not only by calculating the regularized parameters according to the maximum criterion but also by estimating the Gaussian hyperparameter by maximizing the marginal likelihood. A rapid approximation to a Hessian matrix is developed to implement the Bayesian RNN-LM (BRNN-LM) by selecting a small set of salient outer-products. The proposed BRNN-LM achieves a sparser model than the RNN-LM. Experiments on different corpora show the robustness of system performance by applying the rapid BRNN-LM under different conditions.
机译：语言模型（LM）被计算为单词序列的概率，它为各种信息系统的单词预测提供了解决方案。递归神经网络（RNN）强大，可以学习连续空间中单词序列的大跨度动态。但是，由于来自大字典大小和高维隐藏层的参数太多，因此RNN-LM的训练是一个不适的问题。本文提出了一种贝叶斯方法来规范化RNN-LM并将其应用于连续语音识别。我们旨在通过补偿估计的模型参数的不确定性来惩罚过于复杂的RNN-LM，该不确定性由高斯先验表示。贝叶斯分类网络中的目标函数形成为正则化的交叉熵误差函数。不仅通过根据最大准则计算正则化参数来构造正则化模型，而且还通过最大化边际可能性来估计高斯超参数来构造正则化模型。通过选择少量显着的外部乘积，开发出了对Hessian矩阵的快速近似以实现贝叶斯RNN-LM（BRNN-LM）。提出的BRNN-LM比RNN-LM实现了稀疏模型。在不同语料库上的实验通过在不同条件下应用快速BRNN-LM显示了系统性能的鲁棒性。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号