On-device language models: A comprehensive review
The advent of large language models (LLMs) revolutionized natural language processing
applications, and running LLMs on edge devices has become increasingly attractive for …
applications, and running LLMs on edge devices has become increasingly attractive for …
Llm inference serving: Survey of recent advances and opportunities
This survey offers a comprehensive overview of recent advancements in Large Language
Model (LLM) serving systems, focusing on research since the year 2023. We specifically …
Model (LLM) serving systems, focusing on research since the year 2023. We specifically …
Loongserve: Efficiently serving long-context large language models with elastic sequence parallelism
The context window of large language models (LLMs) is rapidly increasing, leading to a
huge variance in resource usage between different requests as well as between different …
huge variance in resource usage between different requests as well as between different …
A survey on efficient inference for large language models
Large Language Models (LLMs) have attracted extensive attention due to their remarkable
performance across various tasks. However, the substantial computational and memory …
performance across various tasks. However, the substantial computational and memory …
Mooncake: A kvcache-centric disaggregated architecture for llm serving
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It
features a KVCache-centric disaggregated architecture that separates the prefill and …
features a KVCache-centric disaggregated architecture that separates the prefill and …
Memserve: Context caching for disaggregated llm serving with elastic memory pool
Large language model (LLM) serving has transformed from stateless to stateful systems,
utilizing techniques like context caching and disaggregated inference. These optimizations …
utilizing techniques like context caching and disaggregated inference. These optimizations …
Taming throughput-latency tradeoff in llm inference with sarathi-serve
Each LLM serving request goes through two phases. The first is prefill which processes the
entire input prompt to produce one output token and the second is decode which generates …
entire input prompt to produce one output token and the second is decode which generates …
Neo: Saving gpu memory crisis with cpu offloading for online llm inference
Online LLM inference powers many exciting applications such as intelligent chatbots and
autonomous agents. Modern LLM inference engines widely rely on request batching to …
autonomous agents. Modern LLM inference engines widely rely on request batching to …
Taming {Throughput-Latency} Tradeoff in {LLM} Inference with {Sarathi-Serve}
Each LLM serving request goes through two phases. The first is prefill which processes the
entire input prompt and produces the first output token and the second is decode which …
entire input prompt and produces the first output token and the second is decode which …
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models
The rapid development of large language models (LLMs) has significantly transformed the
field of artificial intelligence, demonstrating remarkable capabilities in natural language …
field of artificial intelligence, demonstrating remarkable capabilities in natural language …