This article presents a mathematical and empirical verification of computational constancy
measures for natural language text. A constancy measure characterizes a given text by …

It was observed that large language models exhibit a power-law decay of cross entropy with
respect to the number of parameters and training tokens. When extrapolated literally, this …

As we discuss, a stationary stochastic process is nonergodic when a random persistent topic
can be detected in the infinite random text sampled from the process, whereas we call the …

This paper presents a new interpretation for Zipf–Mandelbrot's law in natural language
which rests on two areas of information theory. Firstly, we construct a new class of grammar …

We present two examples of finite-alphabet, infinite excess entropy processes generated by
stationary hidden Markov models (HMMs) with countable state sets. The first, simpler …

We present a hypothetical argument against finite-state processes in statistical language
modeling that is based on semantics rather than syntax. In this theoretical model, we …

We review recent progress in understanding the meaning of mutual information in natural
language. Let us define words in a text as strings that occur sufficiently often. In a few …

As it is known, universal codes, which estimate the entropy rate consistently, exist for
stationary ergodic sources over finite alphabets but not over countably infinite ones. We …

Neural language models have drawn a lot of attention for their strong ability to predict
natural language text. In this paper, we estimate the entropy rate of natural language with …

We construct mixing processes over an infinite alphabet and ergodic processes over a finite
alphabet for which Shannon mutual information between adjacent blocks of length n grows …