In this paper, we present a review of Shannon and differential entropy rate estimation
techniques. Entropy rate, which measures the average information gain from a stochastic …

We give a brief survey of the literature on the empirical estimation of entropy, differential
entropy, relative entropy, mutual information and related information measures. While those …

This survey provides an exposition of a suite of techniques based on the theory of
polynomials, collectively referred to as polynomial methods, which have recently been …

Transformer-based models in Neural Machine Translation (NMT) rely heavily on multi-head
attention for capturing dependencies within and across source and target sequences. In …

We study the following learning problem with dependent data: Given a trajectory of length $
n $ from a stationary Markov chain with $ k $ states, the goal is to predict the distribution of …

The intrinsic predictability of a given time series indicates how well an (ideal) algorithm
could potentially predict it when trained on the time series data. Being able to compute the …

Neural language models have drawn a lot of attention for their strong ability to predict
natural language text. In this paper, we estimate the entropy rate of natural language with …

We show through case studies that it is easier to estimate the fundamental limits of data
processing than to construct the explicit algorithms to achieve those limits. Focusing on …

For any Markov source, there exist universal codes whose normalized codelength
approaches the Shannon limit asymptotically as the number of samples goes to infinity. This …

We study the following learning problem with dependent data: Observing a trajectory of
length from a stationary Markov chain with states, the goal is to predict the next state. For …