This paper aims to compare different regularization strategies to address a common phenomenon, severe overrating, in embedding-based neural networks for NLP. We chose two widely studied neural models and tasks as our testbed. We tried several frequently applied or newly proposed regularization strategies, including penalizing weights (embeddings excluded), penalizing embeddings, re-embedding words, and dropout. We also emphasized on incremental hyperparame-ter tuning, and combining different regu-larizations. The results provide a picture on tuning hyperparameters for neural NLP models.
展开▼
