Creating learning models that can exhibit sophisticated reasoning abilities is one of the
greatest challenges in deep learning research, and mathematics is rapidly becoming one of …

Autoregressive large language models (LLMs) compress knowledge from their training data
through next-token conditional distributions. This limits tractable querying of this knowledge …

Transformers are the dominant architecture for sequence modeling, but there is growing
interest in models that use a fixed-size latent state that does not depend on the sequence …

Previous studies have typically assumed that large language models are unable to
accurately perform arithmetic operations, particularly multiplication of> 8 digits, and …

Length generalization, defined as the ability to extrapolate from shorter training sequences
to longer test ones, is a significant challenge for language models. This issue persists even …

Transformers, central to the successes in modern Natural Language Processing, often falter
on arithmetic tasks despite their vast capabilities--which paradoxically include remarkable …

In an era where symbolic mathematical equations are indispensable for modeling complex
natural phenomena, scientific inquiry often involves collecting observations and translating …

It has been observed in recent years that transformers have problems with length
generalization for certain types of reasoning and arithmetic tasks. In particular, the …

The poor performance of transformers on arithmetic tasks seems to stem in large part from
their inability to keep track of the exact position of each digit inside of a large span of digits …

Large Language Models (LLMs) have revolutionized the field of Natural Language
Processing thanks to their ability to reuse knowledge acquired on massive text corpora on a …