NanoFlow

NanoFlow is a throughput-oriented serving framework for large language models, published at OSDI 2025. End-to-end LLM serving is compute-bound for most common workloads, yet existing engines execute compute, memory, and network operations sequentially within a device. NanoFlow instead exploits intra-device parallelism: it splits inputs into nano-batches and overlaps these heterogeneous operations on a single GPU.

Impact

NanoFlow’s asynchronous scheduling, which prepares the next batch on the CPU while the GPU runs the current one, is integrated into SGLang as its zero-overhead batch scheduler, enabled by default since SGLang v0.4. Similar ideas of overlapping CPU scheduling, computation, and communication are now part of most mainstream serving engines and systems, including vLLM (async scheduling), TensorRT-LLM (overlap scheduler), and TokenWeave (compute–communication overlap).

The code is available on GitHub.