Skip to content
vLLM
policies
Initializing search
GitHub
Home
User Guide
Developer Guide
Benchmarking
API Reference
CLI Reference
Community
vLLM
GitHub
Home
User Guide
User Guide
Getting Started
Getting Started
Quickstart
Installation
Installation
GPU
CPU
TPU
Examples
Examples
Applications
Applications
Chatbot
Rag
Basic
Basic
Offline Inference
Online Serving
Deployment
Deployment
Async LLM Streaming
Helm Charts
LLM Engine Example
Sagemaker-Entrypoint
Disaggregated
Disaggregated
Disaggregated Encoder
Disaggregated Serving
Ec Both Encoder
Disaggregated Prefill V1
Flexkv Connector
KV Load Failure Recovery Test
LMCache Examples
Mooncake Connector
Features
Features
Automatic Prefix Caching
Batch Invariance
Context Extension
Data Parallel
Kv Events
Logging Configuration
Custom Logits Processors
LoRA
Offline Inference with the OpenAI Batch file format
Pause Resume
Profiling
Prompt Embed
Reset Kv
Sharded State
Speculative Decoding
Structured Outputs
Tensorize vLLM Model
Torchrun
Generate
Generate
Batched Chat Completions Online
Multimodal
Qwen 1M Offline
Token Generation Client
Observability
Observability
Monitoring Dashboards
Metrics
Setup OpenTelemetry POC
Prometheus and Grafana
Pooling
Pooling
Classify
Embed
Plugin
Reward
Score
Token Classify
Token Embed
Ray Serving
Ray Serving
Batch LLM Inference
Elastic Ep
Multi-Node-Serving
Ray Serve Deepseek
Run Cluster
Reasoning
Reasoning
OpenAI Chat Completion Tool Calls With Reasoning
OpenAI Chat Completion With Reasoning
OpenAI Chat Completion With Reasoning Streaming
OpenAI Responses Client
RL
RL
RLHF Async New APIs
RLHF Http IPC
RLHF Http NCCL
RLHF IPC
RLHF IPC Fsdp Ep
RLHF NCCL
RLHF NCCL Fsdp Ep
RLHF Sparse NCCL
Routed Experts E2E
Skip Loading Weights In Engine Init
Speech To Text
Speech To Text
Lid
OpenAI
Realtime
Tool Calling
Tool Calling
Chat With Tools Offline
OpenAI Chat Completion Client With Tools
OpenAI Chat Completion Client With Tools Required
OpenAI Chat Completion Client With Tools Xlam
OpenAI Chat Completion Client With Tools Xlam Streaming
OpenAI Responses Client With Mcp Tools
OpenAI Responses Client With Tools
General
General
vLLM V1
Frequently Asked Questions
Production Metrics
Reproducibility
Security
Troubleshooting
Usage Stats Collection
Inference and Serving
Inference and Serving
Offline Inference
Online Serving
Online Serving
Generative Scoring
OpenAI-Compatible Server
Renderer APIs
Speech to Text APIs
Context Parallel Deployment
Data Parallel Deployment
Troubleshooting distributed deployments
Expert Parallel Deployment
Parallelism and Scaling
Integrations
Integrations
Claude Code
Codex
LangChain
LlamaIndex
Deployment
Deployment
Using Docker
Using Kubernetes
Using Nginx
Frameworks
Frameworks
Anyscale
AnythingLLM
AutoGen
BentoML
Cerebrium
Chatbox
Dify
dstack
Haystack
Helm
Hugging Face Inference Endpoints
LiteLLM
Lobe Chat
LWS
Modal
Open WebUI
Retrieval-Augmented Generation
RunPod
SkyPilot
Streamlit
NVIDIA Triton
Integrations
Integrations
AIBrix
NVIDIA Dynamo
KAITO
KServe
Kthena
KubeAI
KubeRay
Llama Stack
llm-d
llmaz
Production stack
Training
Training
Async Reinforcement Learning
What is Layerwise (Re)loading?
Reinforcement Learning from Human Feedback
Transformers Reinforcement Learning
Weight Transfer
Weight Transfer
Base Class and Custom Engines
IPC Engine
NCCL Engine
Configuration
Configuration
Conserving Memory
Engine Arguments
Environment Variables
Model Resolution
Optimization and Tuning
Server Arguments
TPU
Models
Models
Supported Models
Generative Models
Pooling Models
Pooling Models
Classification Usages
Embedding Usages
Reward Usages
Scoring Usages
Specific Model Examples
Token Classification Usages
Token Embedding Usages
Extensions
Extensions
Loading model weights with fastsafetensors
Loading Model Weights with InstantTensor
Loading models with Run:ai Model Streamer
Loading models with CoreWeave's Tensorizer
Hardware Supported Models
Hardware Supported Models
CPU - Intel® Xeon®
XPU - Intel® GPUs
TPU
Features
Features
Automatic Prefix Caching
Batch Invariance
Context Extension
Custom Arguments
Custom Logits Processors
Disaggregated Encoder
Disaggregated Prefilling (experimental)
IndexCache
Interleaved Thinking
LoRA Adapters
MooncakeConnector Usage Guide
MooncakeStoreConnector Usage Guide
MoRIIOConnector Usage Guide
Multimodal Inputs
NixlConnector Compatibility Matrix
NixlConnector Usage Guide
Prompt Embedding Inputs
Reasoning Outputs
Sleep Mode
Structured Outputs
Tool Calling
Quantization
Quantization
AutoAWQ
BitsAndBytes
FP8 ViT Encoder Attention
GGUF
GPTQModel
Intel Quantization Support
NVIDIA Model Optimizer
Online Quantization
Quantized KV Cache
AMD Quark
TorchAO
LLM Compressor
LLM Compressor
FP8 W8A8
INT4 W4A16
INT8 W4A8
INT8 W8A8
Speculative Decoding
Speculative Decoding
Draft Models
EAGLE Draft Models
Hidden State Extraction
MLP Draft Models
MTP (Multi-Token Prediction)
N-Gram Speculation
Parallel Draft Models
vLLM-Project/Speculators
Suffix Decoding
Developer Guide
Developer Guide
General
General
Deprecation Policy
Dockerfile
Editing Agent Instructions
Incremental Compilation Workflow
Profiling vLLM
Vulnerability Management
Model Implementation
Model Implementation
Basic Model
Registering a Model
Unit Testing
Multi-Modal Support
Speech-to-Text (Transcription/Translation) Support
CI
CI
CI Failures
Nightly Builds of vLLM Wheels
Update PyTorch version on vLLM OSS CI/CD
Design Documents
Design Documents
Plugins
Plugins
IO Processor Plugins
LoRA Resolver Plugins
Plugin System
Architecture Overview
Attention Backend Feature Support
CUDA Graphs
Vision Encoder (ViT) CUDA Graphs
CustomOp
Dual Batch Overlap
How to debug the vLLM-torch.compile integration
Fused MoE Modular Kernel
Fusion torch.compile passes
Integration with Hugging Face
Hybrid KV Cache Manager
Logits Processors
Metrics
Multi-Modal Data Processing
Model Runner V2 Design Document
Fused MoE Kernel Features
Python Multiprocessing
NIXL KV Cache Lease Renewal
Optimization Levels
Paged Attention
Automatic Prefix Caching
torch.compile integration
torch.compile with Multimodal Encoders
vLLM IR: Functional Intermediate Representation
Benchmarking
Benchmarking
Benchmark CLI
Parameter Sweeps
Performance Dashboard
API Reference
API Reference
vllm
vllm
collect_env
connections
env_override
envs
exceptions
forward_context
logger
logits_process
logprobs
model_inspection
outputs
pooling_params
sampling_params
scalar_type
scripts
sequence
tasks
version
assets
assets
audio
base
image
video
benchmarks
benchmarks
latency
mm_processor
plot
serve
startup
throughput
datasets
datasets
create_txt_slices_dataset
datasets
utils
lib
lib
endpoint_request_func
ready_checker
utils
sweep
sweep
cli
param_sweep
plot
plot_pareto
serve
serve_workload
server
startup
utils
compilation
compilation
backends
base_static_graph
breakable_cudagraph
caching
codegen
compiler_interface
counter
cuda_graph
decorators
monitor
partition_rules
piecewise_backend
wrapper
passes
passes
fx_utils
inductor_pass
pass_manager
vllm_inductor_pass
fusion
fusion
act_quant_fusion
allreduce_rms_fusion
attn_quant_fusion
collective_fusion
matcher_utils
mla_attn_quant_fusion
mla_rope_kvcache_cat_fusion
qk_norm_rope_fusion
rms_quant_fusion
rocm_aiter_fusion
rope_kvcache_fusion
sequence_parallelism
ir
ir
clone_elimination
inplace_functionalization
lowering_pass
utils
utility
utility
fix_functionalization
noop_elimination
post_cleanup
scatter_split_replace
split_coalescing
config
config
attention
cache
compilation
device
ec_transfer
kernel
kv_events
kv_transfer
load
lora
mamba
model
model_arch
multimodal
observability
offload
parallel
pooler
profiler
quantization
reasoning
scheduler
speculative
speech_to_text
structured_outputs
utils
vllm
weight_transfer
cute_utils
cute_utils
cvt
device_allocator
device_allocator
cumem
xpumem
distributed
distributed
communication_op
kv_events
nixl_utils
parallel_state
stateless_coordinator
utils
device_communicators
device_communicators
all2all
all_reduce_utils
base_device_communicator
cpu_communicator
cuda_communicator
cuda_wrapper
custom_all_reduce
flashinfer_all_reduce
mnnvl_compat
pynccl
pynccl_allocator
pynccl_wrapper
quick_all_reduce
ray_communicator
shm_broadcast
shm_object_storage
symm_mem
xpu_communicator
ec_transfer
ec_transfer
ec_transfer_state
ec_connector
ec_connector
base
example_connector
factory
elastic_ep
elastic_ep
elastic_execute
elastic_state
standby_state
eplb
eplb
async_worker
eplb_communicator
eplb_state
eplb_utils
rebalance_execute
policy
policy
abstract
default
kv_transfer
kv_transfer
kv_transfer_state
kv_connector
kv_connector
base
factory
utils
v1
v1
base
decode_bench_connector
example_connector
example_hidden_states_connector
flexkv_connector
lmcache_connector
lmcache_mp_connector
metrics
multi_connector
offloading_connector
simple_cpu_offload_connector
ssm_conv_transfer_utils
hf3fs
hf3fs
hf3fs_client
hf3fs_connector
hf3fs_metadata_server
utils
utils
common
gather_scatter_helper
hf3fs_mock_client
lmcache_integration
lmcache_integration
multi_process_adapter
utils
vllm_v1_adapter
mooncake
mooncake
mooncake_connector
mooncake_utils
rdma_utils
stats
store
store
connector
coordinator
data
metrics
protocol
scheduler
worker
moriio
moriio
moriio_common
moriio_connector
moriio_engine
nixl
nixl
connector
metadata
scheduler
stats
tp_mapping
utils
worker
offloading
offloading
common
metrics
scheduler
worker
weight_transfer
weight_transfer
base
factory
ipc_engine
nccl_engine
packed_tensor
engine
engine
arg_utils
async_llm_engine
llm_engine
protocol
entrypoints
entrypoints
api_server
chat_utils
grpc_server
launcher
llm
offline_utils
anthropic
anthropic
api_router
protocol
serving
cli
cli
collect_env
launch
main
openai
run_batch
serve
types
benchmark
benchmark
base
latency
main
mm_processor
serve
startup
sweep
throughput
generate
generate
api_router
factories
beam_search
beam_search
offline
online
utils
generative_scoring
generative_scoring
api_router
serving
mcp
mcp
tool
tool_server
openai
openai
api_server
cli_args
dp_supervisor
run_batch
chat_completion
chat_completion
api_router
batch_serving
protocol
serving
stream_harmony
completion
completion
api_router
protocol
serving
engine
engine
protocol
serving
models
models
api_router
protocol
serving
parser
parser
harmony_utils
responses_parser
responses
responses
api_router
context
harmony
protocol
serving
streaming_events
utils
pooling
pooling
factories
offline
typing
utils
base
base
io_processor
protocol
serving
classify
classify
api_router
io_processor
protocol
serving
embed
embed
api_router
io_processor
protocol
serving
pooling
pooling
api_router
io_processor
protocol
serving
scoring
scoring
api_router
io_processor
protocol
serving
typing
utils
serve
serve
dev
dev
cache
cache
api_router
rlhf
rlhf
api_router
rpc
rpc
api_router
server_info
server_info
api_router
sleep
sleep
api_router
disagg
disagg
api_router
mm_serde
protocol
serving
elastic_ep
elastic_ep
api_router
middleware
instrumentator
instrumentator
basic
health
metrics
offline_docs
lora
lora
api_router
protocol
profile
profile
api_router
render
render
api_router
serving
sagemaker
sagemaker
api_router
tokenize
tokenize
api_router
protocol
serving
utils
utils
api_utils
constants
error_response
fingerprint
orca_metrics
request_logger
server_utils
ssl
tool_calls_utils
speech_to_text
speech_to_text
factories
base
base
protocol
serving
realtime
realtime
api_router
connection
metrics
protocol
serving
transcription
transcription
api_router
protocol
serving
translation
translation
api_router
protocol
serving
inputs
inputs
engine
llm
preprocess
ir
ir
op
tolerances
util
ops
ops
layernorm
kernels
kernels
aiter_ops
oink_ops
vllm_c
xpu_ops
helion
helion
case_key
config_manager
register
utils
ops
ops
silu_mul_fp8
triton
triton
qkv_padded_fp8_quant
logging_utils
logging_utils
access_log_filter
dump_input
formatter
lazy
log_time
torch_tensor
lora
lora
lora_model
lora_weights
model_manager
peft_helper
request
resolver
utils
worker_manager
layers
layers
base
base_linear
column_parallel_linear
fused_moe
logits_processor
replicated_linear
row_parallel_linear
utils
vocal_parallel_embedding
ops
ops
torch_ops
torch_ops
lora_ops
triton_ops
triton_ops
fp8_kernel_utils
fused_moe_lora_fp8_op
fused_moe_lora_op
kernel_utils
lora_expand_fp8_op
lora_expand_op
lora_kernel_metadata
lora_shrink_fp8_op
lora_shrink_op
utils
xpu_ops
xpu_ops
lora_ops
punica_wrapper
punica_wrapper
punica_base
punica_cpu
punica_gpu
punica_selector
punica_xpu
utils
model_executor
model_executor
custom_op
parameter
utils
kernels
kernels
linear
linear
base
zentorch_utils
mixed_precision
mixed_precision
allspark
conch
cpu
cutlass
dynamic_4bit
exllama
humming
MPLinearKernel
machete
marlin
rdna3_w4a16
triton_w4a16
xpu
zentorch
mxfp4
mxfp4
base
flashinfer
marlin
xpu
mxfp8
mxfp8
emulation
flashinfer
Mxfp8LinearKernel
marlin
xpu
nvfp4
nvfp4
base
cutlass
emulation
fbgemm
flashinfer
marlin
scaled_mm
scaled_mm
aiter
BlockScaledMMLinearKernel
cpu
cutlass
deep_gemm
flashinfer
marlin
pytorch
rocm
ScaledMMLinearKernel
triton
xpu
zentorch
mhc
mhc
aiter
tilelang
tilelang_kernels
torch
triton
layers
layers
activation
attention_layer_base
batch_invariant
conv
layernorm
lightning_attn
linear
logits_processor
mhc
mla
resampler
sparse_attn_indexer
utils
vocab_parallel_embedding
attention
attention
attention
chunked_local_attention
cross_attention
encoder_only_attention
kv_transfer_utils
mla_attention
mm_encoder_attention
static_sink_attention
fla
fla
ops
ops
chunk
chunk_delta_h
chunk_o
chunk_scaled_dot_kkt
cumsum
fused_gdn_prefill_post_conv
fused_recurrent
fused_sigmoid_gating
index
kda
l2norm
layernorm_guard
op
solve_tril
utils
wy_fast
fused_moe
fused_moe
activation
all2all_utils
config
cpu_fused_moe
deep_gemm_utils
eep_reconfigure
expert_map_manager
fused_moe
fused_moe_method_base
fused_moe_modular_method
layer
modular_kernel
moe_align_block_size
moe_fused_mul_sum
moe_permute_unpermute
routed_experts
routed_experts_capturer
topk_weight_and_reduce
unquantized_fused_moe_method
utils
experts
experts
aiter_mxfp4_w4a8_moe
batched_deep_gemm_moe
cpu_int4_moe
cpu_moe
cutlass_moe
deep_gemm_moe
fallback
flashinfer_b12x_moe
flashinfer_cutedsl_batched_moe
flashinfer_cutedsl_moe
flashinfer_cutlass_moe
fused_batched_moe
fused_humming_moe
gpt_oss_triton_kernels_moe
lora_context
lora_experts_mixin
marlin_moe
nvfp4_emulation_moe
ocp_mx_emulation_moe
rocm_aiter_moe
triton_cutlass_moe
triton_deep_gemm_moe
triton_moe
trtllm_bf16_moe
trtllm_fp8_moe
trtllm_mxfp4_moe
trtllm_mxint4_moe
trtllm_nvfp4_moe
xpu_moe
oracle
oracle
fp8
int8
int_wna16
mxfp4
mxfp8
nvfp4
unquantized
w4a8
w4a8_int8
prepare_finalize
prepare_finalize
batched
deepep_ht
deepep_ll
deepep_v2
flashinfer_nvlink_one_sided
flashinfer_nvlink_two_sided
mori
naive_dp_ep
nixl_ep
no_dp_ep
router
router
aiter_shared_routed_fused_moe_router
base_router
custom_routing_router
fused_moe_router
fused_topk_bias_router
fused_topk_router
gate_linear
grouped_topk_router
router_factory
routing_simulator_router
zero_expert_router
runner
runner
moe_runner
moe_runner_interface
shared_experts
mamba
mamba
abstract
mamba_mixer
mamba_mixer2
mamba_utils
short_conv
gdn
gdn
base
kimi_gdn_linear_attn
olmo_gdn_linear_attn
qwen_gdn_linear_attn
linear
linear
bailing_linear_attn
base
minimax_linear_attn
ops
ops
causal_conv1d
layernorm_gated
mamba_ssm
ssd_bmm
ssd_chunk_scan
ssd_chunk_state
ssd_combined
ssd_state_passing
ssu_dispatch
triton_helpers
cpu
cpu
causal_conv1d
gdn_attention
gdn_chunk_cutedsl
gdn_chunk_cutedsl
kernel_h
kernel_kkt_inv_uw
kernel_o
minimax_rms_norm
minimax_rms_norm
lamport_workspace
rms_norm_tp
pooler
pooler
abstract
activations
common
special
seqwise
seqwise
heads
methods
poolers
tokwise
tokwise
heads
methods
poolers
quantization
quantization
auto_gptq
awq
awq_marlin
awq_triton
base_config
bitsandbytes
experts_int8
fbgemm_fp8
fp8
fp_quant
gguf
humming
inc
input_quant_fp8
kv_cache
modelopt
moe_wna16
mxfp4
qutlass_utils
torchao
compressed_tensors
compressed_tensors
compressed_tensors
compressed_tensors_embedding
triton_scaled_mm
utils
compressed_tensors_moe
compressed_tensors_moe
compressed_tensors_moe
compressed_tensors_moe_w4a4_mxfp4
compressed_tensors_moe_w4a4_nvfp4
compressed_tensors_moe_w4a8_fp8
compressed_tensors_moe_w4a8_int8
compressed_tensors_moe_w8a8_fp8
compressed_tensors_moe_w8a8_int8
compressed_tensors_moe_w8a8_mxfp8
compressed_tensors_moe_wna16
compressed_tensors_moe_wna16_marlin
compressed_tensors_moe_wna16_rdna3
rocm_moe
schemes
schemes
compressed_tensors_scheme
compressed_tensors_w4a4_mxfp4
compressed_tensors_w4a4_nvfp4
compressed_tensors_w4a8_fp8
compressed_tensors_w4a8_int
compressed_tensors_w8a8_fp8
compressed_tensors_w8a8_int8
compressed_tensors_w8a8_mxfp8
compressed_tensors_w8a16_fp8
compressed_tensors_wNa8o8
compressed_tensors_wNa16
transform
transform
linear
module
utils
schemes
schemes
linear_qutlass_nvfp4
online
online
base
fp8
int8
moe_base
mxfp8
quark
quark
quark
quark_moe
utils
schemes
schemes
quark_nvfp4
quark_ocp_mx
quark_scheme
quark_w4a8_mxfp4_fp8
quark_w8a8_fp8
quark_w8a8_int8
turboquant
turboquant
centroids
config
utils
utils
allspark_utils
flashinfer_fp4_moe
flashinfer_mxint4_moe
flashinfer_utils
fp8_utils
gptq_utils
humming_utils
int8_utils
layer_utils
machete_utils
marlin_utils
marlin_utils_fp4
marlin_utils_fp8
marlin_utils_test
mxfp4_utils
mxfp6_utils
mxfp8_utils
nvfp4_emulation_utils
nvfp4_utils
ocp_mx_utils
quant_utils
w8a8_utils
rotary_embedding
rotary_embedding
base
common
deepseek_scaling_rope
dual_chunk_rope
dynamic_ntk_alpha_rope
dynamic_ntk_scaling_rope
ernie45_vl_rope
fope
gemma4_rope
linear_scaling_rope
llama3_rope
llama4_vision_rope
mrope
mrope_interleaved
ntk_scaling_rope
phi3_long_rope_scaled_rope
telechat3_scaling_rope
xdrope
yarn_scaling_rope
model_loader
model_loader
base_loader
bitsandbytes_loader
default_loader
dummy_loader
ep_weight_filter
gguf_loader
modelexpress_loader
runai_streamer_loader
sharded_state_loader
tensorizer
tensorizer_loader
utils
weight_utils
reload
reload
layerwise
meta
sanitize
torchao_decorator
types
utils
models
models
AXK1
adapters
afmoe
aimv2
apertus
arcee
arctic
aria
audioflamingo3
aya_vision
bagel
baichuan
bailing_moe
bailing_moe_linear
bamba
bee
bert
bert_with_rope
blip
blip2
bloom
chameleon
chatglm
cheers
clip
cohere2_moe
cohere2_vision
cohere_asr
cohere_eagle
colbert
colmodernvbert
colpali
colqwen3
colqwen3_5
commandr
config
conformer_encoder
cosmos3
dbrx
deepencoder
deepencoder2
deepseek_eagle
deepseek_eagle3
deepseek_mtp
deepseek_ocr
deepseek_ocr2
deepseek_v2
deepseek_vl2
dots1
dots_ocr
eagle2_5_vl
ernie
ernie45
ernie45_moe
ernie45_vl
ernie45_vl_moe
ernie_mtp
exaone
exaone4
exaone4_5
exaone4_5_mtp
exaone_moe
exaone_moe_mtp
extract_hidden_states
fairseq2_llama
falcon
falcon_h1
fireredasr2
fireredlid
flex_olmo
funasr
funaudiochat
fuyu
gemma
gemma2
gemma3
gemma3_mm
gemma3n
gemma3n_audio_utils
gemma3n_mm
gemma4
gemma4_mm
gemma4_mtp
gemma4_unified
glm
glm4
glm4_1v
glm4_moe
glm4_moe_lite
glm4_moe_lite_mtp
glm4_moe_mtp
glm4v
glm_ocr
glm_ocr_mtp
glmasr
glmasr_utils
gpt2
gpt_bigcode
gpt_j
gpt_neox
gpt_oss
granite
granite4_vision
granite_speech
granite_speech_plus
granitemoe
granitemoehybrid
granitemoeshared
gritlm
grok1
h2ovl
hunyuan_v1
hunyuan_vision
hy_v3
hy_v3_mtp
hyperclovax
hyperclovax_vision
hyperclovax_vision_v2
idefics2_vision_model
idefics3
interfaces
interfaces_base
intern_vit
internlm2
internlm2_ve
interns1
interns1_pro
interns1_vit
interns2_preview
internvl
iquest_loopcoder
isaac
jais2
jamba
jina
jina_vl
kanana_v
keye
keye_vl1_5
kimi_audio
kimi_k25
kimi_k25_vit
kimi_linear
kimi_vl
laguna
lfm2
lfm2_moe
lfm2_siglip2
lfm2_vl
lightonocr
llama
llama4
llama4_eagle
llama_eagle
llama_eagle3
llava
llava_next
llava_next_video
llava_onevision
longcat_flash
longcat_flash_mtp
mamba
mamba2
medusa
mellum
midashenglm
mimo
mimo_audio
mimo_mtp
mimo_v2
mimo_v2_mtp
mimo_v2_omni
minicpm
minicpm3
minicpm_eagle
minicpmo
minicpmv
minicpmv4_6
minimax_m2
minimax_text_01
minimax_vl_01
mistral
mistral3
mistral_eagle
mistral_large_3
mistral_large_3_eagle
mixtral
mllama4
mlp_speculator
modernbert
module_mapping
molmo
molmo2
moondream3
moonvit
mpt
musicflamingo
nano_nemotron_vl
nemotron
nemotron_h
nemotron_h_mtp
nemotron_nas
nemotron_parse
nemotron_vl
nvlm_d
olmo
olmo2
olmo_hybrid
olmoe
opencua
openpangu
openpangu_mtp
openpangu_vl
openvla
opt
orion
ouro
ovis
ovis2_5
paddleocr_vl
paligemma
parakeet
param2moe
persimmon
phi
phi3
phi3v
phi4mm
phi4mm_audio
phi4mm_utils
phi4siglip
phimoe
pixtral
plamo2
plamo3
qianfan_ocr
qwen
qwen2
qwen2_5_omni_thinker
qwen2_5_vl
qwen2_audio
qwen2_moe
qwen2_rm
qwen2_vl
qwen3
qwen3_5
qwen3_5_mtp
qwen3_asr
qwen3_asr_forced_aligner
qwen3_asr_realtime
qwen3_dflash
qwen3_moe
qwen3_next
qwen3_next_mtp
qwen3_omni_moe_thinker
qwen3_vl
qwen3_vl_moe
qwen_vl
radio
registry
rnj1
roberta
rvl
sarvam
seed_oss
siglip
siglip2navit
skyworkr1v
smolvlm
solar
stablelm
starcoder2
step1
step3_text
step3_vl
step3p5
step3p5_mtp
step3p7
step_vl
tarsier
telechat2
teleflm
terratorch
ultravox
utils
vision
voxtral
voxtral_realtime
voyage
whisper
whisper_causal
whisper_utils
zamba2
transformers
transformers
base
causal
legacy
moe
multimodal
pooling
utils
offloader
offloader
base
prefetch
prefetch_ops
uva
warmup
warmup
deep_gemm_warmup
kernel_warmup
models
models
deepseek_v4
deepseek_v4
attention
compressor
quant_config
sparse_mla
amd
amd
model
mtp
rocm
common
common
rope
ops
ops
cache_utils
fused_compress_quant_cache
fused_indexer_q
fused_inv_rope_fp8_quant
fused_mtp_input_rmsnorm
fused_qk_rmsnorm
save_partial_states
nvidia
nvidia
flashinfer_sparse
flashmla
model
mtp
ops
ops
dequant_gather_k_cutedsl
fused_indexer_q_cutedsl
o_proj
prepare_megamoe
sparse_attn_compress_cutedsl
xpu
xpu
model
mtp
xpu_qnorm_rope_kv_fp8_insert
xpu_sparse
xpu_sparse_decode_fp8
multimodal
multimodal
audio
cache
encoder_budget
evs
hasher
image
inputs
parse
registry
utils
video
media
media
audio
base
connector
image
video
processing
processing
context
dummy_inputs
inputs
processor
parser
parser
abstract_parser
parser_manager
platforms
platforms
cpu
cuda
interface
rocm
tpu
xpu
zen_cpu
plugins
plugins
io_processors
io_processors
interface
lora_resolvers
lora_resolvers
filesystem_resolver
hf_hub_resolver
profiler
profiler
layerwise_profile
utils
wrapper
ray
ray
lazy_utils
ray_env
reasoning
reasoning
abs_reasoning_parsers
basic_parsers
cohere_command_reasoning_parser
deepseek_r1_reasoning_parser
deepseek_v3_reasoning_parser
ernie45_reasoning_parser
gemma4_reasoning_parser
gemma4_utils
gptoss_reasoning_parser
granite_reasoning_parser
hunyuan_a13b_reasoning_parser
hy_v3_reasoning_parser
identity_reasoning_parser
kimi_k2_reasoning_parser
minimax_m2_reasoning_parser
mistral_reasoning_parser
nemotron_v3_reasoning_parser
olmo3_reasoning_parser
poolside_v1_reasoning_parser
qwen3_reasoning_parser
seedoss_reasoning_parser
step3_reasoning_parser
step3p5_reasoning_parser
renderers
renderers
base
deepseek_v4
deepseek_v32
embed_utils
grok2
hf
mistral
params
registry
terratorch
inputs
inputs
preprocess
tokenize
tokenizers
tokenizers
deepseek_v4
deepseek_v4_encoding
deepseek_v32
deepseek_v32_encoding
detokenizer_utils
fastokens
grok2
hf
kimi_audio
mistral
protocol
qwen_vl
registry
tool_parsers
tool_parsers
abstract_tool_parser
apertus_tool_parser
cohere_command_tool_parser
deepseekv3_tool_parser
deepseekv4_tool_parser
deepseekv31_tool_parser
deepseekv32_tool_parser
ernie45_tool_parser
functiongemma_tool_parser
gemma4_tool_parser
gemma4_utils
gigachat3_tool_parser
glm4_moe_tool_parser
glm47_moe_tool_parser
granite4_tool_parser
granite_20b_fc_tool_parser
granite_tool_parser
hermes_tool_parser
hunyuan_a13b_tool_parser
hy_v3_tool_parser
internlm2_tool_parser
jamba_tool_parser
kimi_k2_tool_parser
lfm2_tool_parser
llama4_pythonic_tool_parser
llama_tool_parser
longcat_tool_parser
minicpm5xml_tool_parser
minimax_m2_tool_parser
minimax_tool_parser
mistral_tool_parser
olmo3_tool_parser
openai_tool_parser
phi4mini_tool_parser
poolside_v1_tool_parser
pythonic_tool_parser
qwen3coder_tool_parser
qwen3xml_tool_parser
seed_oss_tool_parser
step3_tool_parser
step3p5_tool_parser
streaming
structural_tag_registry
utils
xlam_tool_parser
tracing
tracing
otel
utils
transformers_utils
transformers_utils
config
config_parser_base
dynamic_module
gguf_utils
model_arch_config_convertor
processor
repo_utils
runai_utils
s3_utils
utils
chat_templates
chat_templates
registry
triton_utils
triton_utils
allocation
importing
jit_monitor
usage
usage
usage_lib
utils
utils
argparse_utils
async_utils
cache
collection_utils
counter
cpu_resource_utils
cpu_triton_utils
deep_gemm
flashinfer
func_utils
gc_utils
hashing
import_utils
jsontree
math_utils
mem_constants
mem_utils
mistral
multi_stream_utils
nccl
network_utils
numa_utils
nvtx_pytorch_hooks
ompmultiprocessing
platform_utils
print_utils
registry
serial_utils
system_utils
tensor_schema
torch_utils
tqdm_utils
v1
v1
cudagraph_dispatcher
kv_cache_interface
kv_cache_spec_registry
outputs
request
serial_utils
utils
attention
attention
backend
selector
backends
backends
cpu_attn
fa_utils
flash_attn
flash_attn_diffkv
flashinfer
flex_attention
gdn_attn
linear_attn
mamba1_attn
mamba2_attn
mamba_attn
registry
rocm_aiter_fa
rocm_aiter_unified_attn
rocm_attn
short_conv_attn
triton_attn
turboquant_attn
utils
mla
mla
aiter_triton_mla
compressor_utils
cutlass_mla
flashattn_mla
flashinfer_mla
flashinfer_mla_sparse
flashmla
flashmla_sparse
indexer
rocm_aiter_mla
rocm_aiter_mla_sparse
sparse_swa
sparse_utils
tokenspeed_mla
triton_mla
xpu_mla_sparse
prefill
prefill
base
flash_attn
flashinfer
registry
selector
tokenspeed_mla
trtllm_ragged
ops
ops
chunked_prefill_paged_decode
common
dcp_alltoall
flashmla
merge_attn_states
paged_attn
prefix_prefill
rocm_aiter_mla_sparse
triton_attention_helpers
triton_decode_attention
triton_merge_attn_states
triton_prefill_attention
triton_reshape_and_cache_flash
triton_turboquant_decode
triton_turboquant_store
triton_unified_attention
vit_attn_wrappers
xpu_mla_sparse
core
core
block_pool
encoder_cache_manager
kv_cache_coordinator
kv_cache_manager
kv_cache_metrics
kv_cache_utils
single_type_kv_cache_manager
sched
sched
async_scheduler
interface
output
request_queue
scheduler
utils
engine
engine
async_llm
coordinator
core
core_client
detokenizer
exceptions
input_processor
llm_engine
logprobs
output_processor
parallel_sampling
tensor_ipc
utils
executor
executor
abstract
multiproc_executor
ray_env_utils
ray_executor
ray_executor_v2
ray_utils
uniproc_executor
vllm_net_devices
kv_offload
kv_offload
base
factory
file_mapper
cpu
cpu
common
gpu_worker
manager
shared_offload_region
spec
swap_blocks_triton
policies
policies
arc
base
lru
tiering
tiering
base
factory
manager
spec
example
example
manager
fs
fs
io
manager
thread_pool
obj
obj
config
manager
worker
worker
worker
metrics
metrics
loggers
perf
prometheus
ray_wrappers
reader
stats
utils
pool
pool
late_interaction
metadata
sample
sample
metadata
rejection_sampler
sampler
thinking_budget_state
logits_processor
logits_processor
builtin
interface
state
ops
ops
bad_words
logprobs
penalties
topk_topp_sampler
topk_topp_triton
simple_kv_offload
simple_kv_offload
copy_backend
cuda_mem_ops
manager
metadata
worker
spec_decode
spec_decode
custom_class_proposer
dflash
draft_model
eagle
extract_hidden_states
gemma4
llm_base_proposer
medusa
metadata
metrics
ngram_proposer
ngram_proposer_gpu
step3p5
suffix_decoding
utils
structured_output
structured_output
backend_guidance
backend_lm_format_enforcer
backend_outlines
backend_types
backend_xgrammar
request
utils
worker
worker
block_table
cp_utils
cpu_model_runner
cpu_worker
dp_utils
ec_connector_model_runner_mixin
encoder_cudagraph
encoder_cudagraph_defs
gpu_input_batch
gpu_model_runner
gpu_ubatch_wrapper
gpu_worker
kv_connector_model_runner_mixin
lora_model_runner_mixin
mamba_utils
tpu_input_batch
ubatch_utils
ubatching
utils
worker_base
workspace
xpu_model_runner
xpu_worker
cpu
cpu
buffer_utils
model_runner
shm
gpu
gpu
async_utils
attn_utils
block_table
buffer_utils
cp_utils
cudagraph_utils
dp_utils
eplb_utils
input_batch
kv_connector
lora_utils
model_runner
pp_utils
shutdown
states
structured_outputs
warmup
metrics
metrics
logits
mm
mm
encoder_cache
encoder_runner
lora
rope
model_states
model_states
default
interface
mamba_hybrid
whisper
pool
pool
late_interaction_runner
pooling_runner
sample
sample
bad_words
gumbel
logit_bias
logprob
min_p
output
penalties
prompt_logprob
sampler
states
spec_decode
spec_decode
rejection_sampler
rejection_sampler_utils
speculator
utils
autoregressive
autoregressive
cudagraph_utils
speculator
eagle
eagle
eagle3_utils
speculator
utils
gemma4
gemma4
speculator
mtp
mtp
speculator
CLI Reference
CLI Reference
vllm serve
vllm chat
vllm complete
vllm run-batch
vllm bench
vllm bench
vllm bench latency
vllm bench mm-processor
vllm bench serve
vllm bench sweep plot
vllm bench sweep plot_pareto
vllm bench sweep serve
vllm bench sweep serve_workload
vllm bench throughput
vllm launch
vllm launch
vllm launch render
Community
Community
Contact Us
Meetups
Sponsors
Governance
Governance
Collaboration Policy
Committers
Governance Process
Blog
Forum
Slack
Home
API Reference
vllm
v1
kv_offload
cpu
policies
vllm.v1.kv_offload.cpu.policies
¶
Modules:
arc
–
base
–
lru
–
Back to top