This paper experimentally reports on predictive uncertainty for real-world text classification
tasks. We define a straightforward protocol to evaluate the quality of Deep Learning
uncertainty estimation. We report on a Monte Carlo Dropout-based model and data
uncertainties using 1-D convolutional neural networks on multi-class news topic and
sentiment classification datasets. We find that our protocol effectively enables to test for
novelty detection robustness showing that Bayesian quantities underestimate uncertainty …

… For novelty detection, shows easy separable classes. Multi-label annotations ensure class
separability information. Then why would … We investigate the reliability of Monte Carlo
Dropout-based uncertainty estimates for unsupervised detection of novel class data in text
classification and find that the studied methods underestimate uncertainty. • We experimentally
demonstrate on real-world text classification datasets that uncertainty modelling with Bayesian
DL methods does not guarantee performance increase on classification and calibration metrics …