It was observed that large language models exhibit a power-law decay of cross entropy with
respect to the number of parameters and training tokens. When extrapolated literally, this …

As we discuss, a stationary stochastic process is nonergodic when a random persistent topic
can be detected in the infinite random text sampled from the process, whereas we call the …

We present a hypothetical argument against finite-state processes in statistical language
modeling that is based on semantics rather than syntax. In this theoretical model, we …

We review recent progress in understanding the meaning of mutual information in natural
language. Let us define words in a text as strings that occur sufficiently often. In a few …

The article discusses two mutually-incompatible hypotheses about the stochastic
mechanism of the generation of texts in natural language, which could be related to entropy …

We investigate stationary hidden Markov processes for which mutual information between
the past and the future is infinite. It is assumed that the number of observable states is finite …

The relaxed Hilberg conjecture states that the mutual information between two adjacent
blocks of text in natural language grows as a power of the block length. The present paper …

A regular Hilberg process is a stationary process that satisfies both a hyperlogarithmic
growth of maximal repetition and a power-law growth of topological entropy, which are a …

We supply corrected proofs of the invariance of completion and the chain rule for the
Shannon information measures of arbitrary fields, as stated by Dębowski in 2009. Our …

This paper concerns the rates of power law growth of mutual information computed for a
stationary measure or for a universal code. The rates are called Hilberg exponents, and four …