Please confirm you are human
This browser or connection looks automated. Press and continuously hold the control for 3 seconds to enable Google-hosted web results and, when separately allowed, AI-assisted answers.
A successful check enables 100 search requests. Interactive access does not authorize scraping, systematic collection, or reuse of search output.
News
FP8 W8A8 - vLLM-Omni
2+ mon, 3+ week ago (218+ words) FP8 quantization converts BF16/FP16 weights to FP8 at model load time. Online activation scaling is the default and does not require calibration. Static activation scaling is supported when calibrated scale information is available. For ModelOpt-produced pre-quantized checkpoints, see ModelOpt Quantization. Some architectures can…...
Attention Backend Feature Support
5+ mon, 3+ week ago (268+ words) Do not edit this file manually. Run the following command to regenerate it: There are two ways to specify the backend from the command line: Note: --attention-backend and --attention-config.backend are mutually exclusive. Use one or the other, not both....
Optimization and Tuning
4+ day, 22+ hour ago (504+ words) This guide aims to help users improve vLLM Ascend performance at the system level. It includes OS configuration, library optimization, deployment guide, and so on. Any feedback is welcome. Install vLLM and vLLM Ascend: Please follow the Installation Guide to…...
inkling
6+ day, 3+ hour ago (92+ words) vLLM Inkling model configs for the text backbone and audio/vision towers. NVFP4 (ModelOpt) support for the Inkling mixture-of-experts. Text-only entry point ( inkling_model checkpoints). Top-level (multimodal) entry point. Bases: _TmlForCausalLMBase, SupportsMultiModal Builds the vision + audio towers on top of the shared text backbone....
bf16x3_router_gemm_cutedsl
6+ day, 1+ hour ago (64+ words) vLLM CuteDSL BF16x3 router GEMM. Computes X @ W.T for BF16 X with shape [N, K] and FP32 router weights W with shape [M, K] by decomposing each FP32 weight value into three BF16 residual terms inside the kernel, then accumulating the three BF16 MMA results into…...
attention
6+ day, 3+ hour ago (13+ words) vLLM Project the per-head relative branch r to per-distance logits....
fa4_rel_attention
6+ day, 3+ hour ago (82+ words) vLLM Round the FA4 scheduling bound up to a power of two. Return the split-KV cap for FA4 with sheared relative bias. Paged varlen FA4 over the bound K/V cache with the Inkling relative bias. q is (num_tokens, num_heads, head_dim); key_cache / value_cache are the paged caches (num_blocks, block_size, num_kv_heads, head_dim); block_table is…...
config
5+ day, 23+ hour ago (30+ words) vLLM Translate vLLM KV cache metadata for native offloading backends. Translate vLLM configuration into the native offloading boundary. Return whether a KV cache tensor uses a packed block stride....
mm_towers
6+ day, 3+ hour ago (77+ words) vLLM Fused CUDA kernels for the Inkling vision/audio towers. Both kernels keep the reference paths' fp32 accumulation and per-op bf16 rounding points (native rms_norm / F.gelu ); outputs are frequently bit-identical and otherwise differ by 1-2 bf16 ulps from reduction-order (real-checkpoint-weight cosine vs reference > 0.9999998). Fused gelu…...
norm
6+ day, 2+ hour ago (42+ words) vLLM Fused res = residual + delta; y = rmsnorm(res). Fused rmsnorm(embed_table[input_ids], weight) row gather + norm. add_rmsnorm(residual, delta, weight, eps) ¶ Returns (y, res); both are fresh tensors (cudagraph-friendly, no in-place update of the inputs)....