1

Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP

Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, …

Yilong Zhao, Xiaonan Nie, Kan Zhu, Shuang Ma, Zhichao Lai, Hongxiang Hao, Yang Zhou, Baris Kasikci, Ion Stoica

Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding

Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) …

Yilong Zhao, Jiaming Tang, Kan Zhu, Zihao Ye, Chi-Chih Chang, Chaofan Lin, Jongseok Park, Guangxuan Xiao, Mohamed S. Abdelfattah, Mingyu Gao, Baris Kasikci, Song Han, Ion Stoica

TeleRAG: Efficient Retrieval-Augmented Generation Inference with Lookahead Retrieval

Retrieval-augmented generation (RAG) extends large language models (LLMs) with external data sources to enhance factual correctness and …

Chien-Yu Lin, Keisuke Kamahori, Yiyu Liu, Xiaoxiang Shi, Madhav Kashyap, Yile Gu, Rulin Shao, Zihao Ye, Kan Zhu, Rohan Kadekodi, Stephanie Wang, Arvind Krishnamurthy, Luis Ceze, Baris Kasikci

Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs

Selected tokens by a target fraction of cumulative attention scores instead of a fixed token budget
Estimated token importance efficiently with clustering-based sorting and distribution fitting

Kan Zhu, Tian Tang, Qinyu Xu, Zhan Jin, Yile Gu, Zhichen Zeng, Rohan Kadekodi, Liangyu Zhao, Ang Li, Arvind Krishnamurthy, Baris Kasikci

BlendServe: Optimizing Offline Inference with Resource-aware Batching

Offline batch inference is gaining popularity as a cost-effective solution for latency-insensitive tasks, such as model evaluation and …

Yilong Zhao, Shuo Yang, Kan Zhu, Lianmin Zheng, Baris Kasikci, Yifan Qiao, Yang Zhou, Jiarong Xing, Ion Stoica

NanoFlow: Towards Optimal Large Language Model Serving Throughput

Identified the underutilization of resources within a single device in existing serving systems due to sequentially executing operations with various resource requirements
Overlapping the utilization of different resources within a single device through operation co-scheduling for higher resource utilization

Kan Zhu, Yufei Gao, Yilong Zhao, Liangyu Zhao, Gefei Zuo, Yile Gu, Dedong Xie, Tian Tang, Qinyu Xu, Zihao Ye, Keisuke Kamahori, Chien-Yu Lin, Ziren Wang, Stephanie Wang, Arvind Krishnamurthy, Baris Kasikci

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

Identified the sparsity in the attention mechanism of long-context LLMs
Dynamically choose critical tokens based on the query

Jiaming Tang, Yilong Zhao, Kan Zhu, Guangxuan Xiao, Baris Kasikci, Song Han

Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

Use activation and weight low-bit quantization to improve throughput.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, Baris Kasikci

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

From Optimal to Practical: Efficient Micro-op Cache Replacement Policies for Data Center Applications

Understand the limitations of state-of-the-art replacement policy and the uniqueness of micro-op cache.
Proposed and evaluated counter-based, profile-guided replacement policy

Kan Zhu, Yilong Zhao, Yufei Gao, Peter Braun, Tanvir Ahmed Khan, Heiner Litz, Baris Kasikci, Shuwen Deng

From Optimal to Practical: Efficient Micro-op Cache Replacement Policies for Data Center Applications

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models

We propose Fiddler, a resource-efficient inference engine with CPU-GPU orchestration for MoE models.

Keisuke Kamahori, Tian Tang, Yile Gu, Kan Zhu, Baris Kasikci

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models