• 主题
高级搜索  >
历史搜索 清空历史
首页> 外文会议>IEEE International Conference on Acoustics, Speech and Signal Processing >Making Punctuation Restoration Robust and Fast with Multi-Task Learning and Knowledge Distillation
【24h】

Making Punctuation Restoration Robust and Fast with Multi-Task Learning and Knowledge Distillation

机译：用多任务学习和知识蒸馏制作标点恢复速度和快速

获取原文
页面导航
  • 摘要
  • 著录项
  • 相似文献
  • 相关主题

摘要

In punctuation restoration, we try to recover the missing punctuation from automatic speech recognition output to improve understandability. Currently, large pre-trained transformers such as BERT set the benchmark on this task but there are two main drawbacks to these models. First, the pre-training data does not match the output data from speech recognition that contains errors. Second, the large number of model parameters increases inference time. To address the former, we use a multi-task learning framework with ELECTRA, a recently proposed improvement on BERT, that has a generator-discriminator structure. The generator allows us to inject errors into the training data and, as our experiments show, this improves robustness against speech recognition errors during inference. To address the latter, we investigate knowledge distillation and parameter pruning of ELECTRA. In our experiments on the IWSLT 2012 benchmark data, a model with less than 11% the size of BERT achieved better performance while having an 82% faster inference time.
机译：在标点符号恢复中，我们尝试从自动语音识别输出中恢复缺少的标点符号，以提高可易于的可理解性。目前，大型预训练的变压器如BERT在这项任务上设置了基准，但这些模型有两个主要缺点。首先，预训练数据与包含错误的语音识别的输出数据与输出数据不匹配。其次，大量模型参数增加了推理时间。为了解决前者，我们使用具有电力的多任务学习框架，最近提出的伯特改进，具有发电机鉴别器结构。发电机允许我们将错误注入培训数据，并且作为我们的实验表明，这改善了在推理期间对语音识别误差的鲁棒性。为了解决后者，我们研究了电力的知识蒸馏和参数灌注。在我们对IWSLT 2012基准数据的实验中，BERT大小尺寸小于11％的模型可以实现更好的性能，同时具有82％的推理时间。

著录项

相似文献

  • 外文文献
  • 中文文献
  • 专利
获取原文

客服邮箱：kefu@zhangqiaokeyan.com

京公网安备：11010802029741号 ICP备案号：京ICP备15016152号-6 六维联合信息科技 (北京) 有限公司©版权所有

  • 客服微信

  • 服务号