Large language models (LLMs) have garnered unprecedented advancements across
diverse fields, ranging from natural language processing to computer vision and beyond …

Structured pruning of modern large language models (LLMs) has emerged as a way of
decreasing their high computational needs. Width pruning reduces the size of projection …

ABSTRACT Large Language Models (LLMs) have demonstrated remarkable capabilities
across various fields, from natural language understanding to text generation. Compared to …

We introduce MoMa, a novel modality-aware mixture-of-experts (MoE) architecture designed
for pre-training mixed-modal, early-fusion language models. MoMa processes images and …

Data curation is an essential component of large-scale pretraining. In this work, we
demonstrate that jointly selecting batches of data is more effective for learning than selecting …

The rapid development of large language models (LLMs) has significantly transformed the
field of artificial intelligence, demonstrating remarkable capabilities in natural language …

Diffusion Transformers have emerged as the preeminent models for a wide array of
generative tasks, demonstrating superior performance and efficacy across various …

Large Language Models (LLMs) are not only resource-intensive to train but even more
costly to deploy in production. Therefore, recent work has attempted to prune blocks of LLMs …

A well-known dilemma in large vision-language models (eg, GPT-4, LLaVA) is that while
increasing the number of vision tokens generally enhances visual understanding, it also …

Despite the significant progress in multimodal large language models (MLLMs), their high
computational cost remains a barrier to real-world deployment. Inspired by the mixture of …