Towards understanding how transformer perform multi-step reasoning with matching operation

Z Wang, Y Wang, Z Zhang, Z Zhou, H Jin, T Hu… - arXiv preprint arXiv …, 2024 - arxiv.org
Large language models have consistently struggled with complex reasoning tasks, such as
mathematical problem-solving. Investigating the internal reasoning mechanisms of these …

Initialization is Critical to Whether Transformers Fit Composite Functions by Inference or Memorizing

Z Zhang, P Lin, Z Wang, Y Zhang, ZQJ Xu - arXiv preprint arXiv …, 2024 - arxiv.org
Transformers have shown impressive capabilities across various tasks, but their
performance on compositional problems remains a topic of debate. In this work, we …

Anchor Attention, Small Cache: Code Generation with Large Language Models

X Zhang, Y Zhou, G Yang, HC Gall, T Chen - arXiv preprint arXiv …, 2024 - arxiv.org
The development of large language models (LLMs) has revolutionized automated code
generation. However, their high demand of computation resources has hindered a broader …

Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling

M Wang - arXiv preprint arXiv:2402.00522, 2024 - arxiv.org
We conduct a systematic study of the approximation properties of Transformer for sequence
modeling with long, sparse and complicated memory. We investigate the mechanisms …