Install
Easy, fast, and cheap LLM serving for everyone vLLM is a fast and easy-to-use library for LLM inference and serving. Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry. vLLM is fast with: Efficient management
- 37articles · 30d
- 3+ day agolatest article
- Aug 15, 2026earliest in window
- 0%with images
- 177avg words
- Science & Technology 37
- Software Dev. 37
- Computers & Electronics 36
- Jobs & Education 1
Please confirm you are human
This browser or connection looks automated. Press and continuously hold the control for 3 seconds to enable Google-hosted web results and, when separately allowed, AI-assisted answers.
A successful check enables 100 search requests. Interactive access does not authorize scraping, systematic collection, or reuse of search output.
News
dspark
3+ day, 48+ min ago (47+ words) vLLM CPU DeepSeek-V4 DSpark (speculative decoding) — deferred, not implemented. DSparkDeepseekV4ForCausalLM is only instantiated when the speculative- decoding registry resolves it by name for a DSpark speculative config; a plain (non-spec) DeepSeek-V4 CPU model never constructs this class. This stub exists solely…...
low_latency_gemm
1+ week, 5+ day ago (20+ words) vLLM Qwen4Exp low-latency GEMM hook for AMD ROCm. Keep the standard vLLM linear methods on AMD ROCm....
indexer_qsa
1+ week, 5+ day ago (49+ words) Replicated Q/K projection plus paged, weight-free QSA selection. Normalize pooled K and apply the first token's exact group position. Project replicated Q/K, normalize+rotate Q, and preserve raw K. Use vLLM's portable RMSNorm implementation on ROCm. Apply the main…...
hy_v4_reasoning_parser
1+ week, 6+ day ago (286+ words) Reasoning extraction for HYV4, on plain data. Count reasoning tokens (0 in no_think). The start token is injected at the END of the prompt, so it is normally absent from token_ids. Only skip a leading start token when it actually appears first (legacy / no…...
flashmla_sparse
1+ week, 6+ day ago (254+ words) Sink-capable FlashMLA sparse backend for HY V4 (NVIDIA). HY V4 adds a per-head learnable attention sink on top of sparse MLA. The vendored FlashMLA kernels already accept an attn_sink argument, but vLLM's shared FLASHMLA_SPARSE backend neither advertises sink support nor forwards the tensor, so…...
hy_v4_tool_parser
1+ week, 6+ day ago (620+ words) Pure tool-call parsing logic for HYV4. Returns plain dicts; no vLLM types. Holds only parser state: the structural token strings/ids, the compiled regexes, and the streaming incremental state. Stateless argument parsing is delegated to the module-level utilities above. Streaming has…...
sse_keep_alive
2+ week, 6+ day ago (48+ words) vLLM SSE keep-alive comments for idle streaming responses. Emit an SSE keep-alive comment when generator is idle. A non-positive or non-finite interval returns generator unchanged, so the default path has no overhead. Otherwise a keep-alive comment is yielded whenever no…...
sharded_rdt_engine
2+ week, 6+ day ago (1746+ words) Sharded Ray Direct Transport (RDT) weight transfer engine (consumer side). Pulls only the slice each vLLM worker consumes under tensor/expert parallelism, not the full HF-format tensor. One thing at four resolutions, over three lifetimes. FetchKey -- (name, op_chain), "which slice of…...
Sharded RDT Engine
2+ week, 6+ day ago (259+ words) Discovery is expensive, so it happens once, at init_transfer_engine, as a dry run over model.load_weights with every parameter on meta. Nothing is transferred; the engine just records, per leaf module, which slice feeds which destination region. Every later sync is pure…...
server_utils
3+ week, 3+ day ago (47+ words) vLLM Get the uvicorn log config based on the provided arguments. Priority: 1. If log_config_file is specified, use it 2. If disable_access_log_for_endpoints is specified, create a config with the access log filter 3. Otherwise, return None (use uvicorn defaults)...