Activation Compression at LLM Inference

Mechanism

hivemind.compression.BlockwiseQuantization (compression_type = runtime_pb2.BLOCKWISE_8BIT) calls bitsandbytes quantize_blockwise during tensor serialization on the RPC path between consecutive Petals servers, and dequantizes on receipt. Selected per-server through the --compression flag, plumbed as getattr(CompressionType, compression_type).

Key numbers

_BLOCKWISE_QUANTIZATION_BLOCKSIZE = 4096. Paper: "we apply dynamic blockwise quantization to the hidden states before pipeline-parallel communication, which halves the bandwidth requirements without any noticeable effect on generation quality". Petals --compression defaults to 'NONE', so it is opt-in.

Ecosystem

The closest prior art to PP wire quantization, worth citing honestly rather than claiming the idea is untouched. But the design targets slow WAN links rather than NVLink/IB, stops at 8 bits, and its 4096-element block is 128-256x coarser than the microscaled blocks (32 for MXFP4, 16 for NVFP4) our results indicate are needed to hold 4-bit at depth. No depth study, no 4-bit wire path.

arXiv:2312.08361 · Code · Source adversarially re-checked: confirmed

Mechanism

A torch.compile pattern matcher replaces all_reduce+RMSNorm(+quant) with the sequence-parallel form. In FirstAllReduceRMSNormStaticFP8Pattern.replacement the quant matcher runs on the RMSNorm output and the result is passed to _all_gather(quant). The NVFP4 variant calls scaled_fp4_quant then all-gathers the packed E2M1 data and its block-scale tensor as two separate collectives.

Key numbers

enable_sp: off by default (pinned via IS_DENSE=False, vllm 25689); requires FULL-graph compilation (inductor graph partition or empty splitting_ops; piecewise is NOT supported). Gated to compute capability 90 (H100) and 100/103 (datacenter Blackwell) at hidden_size >= 8192, plus XPU at 4096, above an autoconfigured token threshold. fuse_allreduce_rms: on at -O2 for Hopper/Blackwell with TP>1 plus FlashInfer.

Ecosystem

Important nuance for our framing: upstream vLLM does compress activations on the TP/SP wire, but only as a by-product of fusing the next layer's quantization epilogue, and only on the all-gather (pure data-movement) leg. The plain fuse_allreduce_rms pattern (AllReduce → RMSNorm → quant) reduces in high precision and quantizes afterwards, so it is not wire compression. Neither path applies to PP.

Code · Source adversarially re-checked: corrected here

Mechanism

The Ascend path compresses TP collective payloads for Qwen3 on NPU. DeepEP dispatches expert-parallel tokens as FP8 (data, scale_factors) tuples while combining expert outputs in BF16. The ROCm kernel fuses the all-reduce epilogue with RMSNorm and per-group FP8 quantization so the next GEMM's input is produced in place, rather than compressing the reduction itself.

Key numbers

Merged: PR 20520 2026-05 (Ascend TP, Qwen3); PR 22316 2026-04 (DeepEP low-latency dispatch FP8, DeepSeek-R1 W4A8; the dispatcher now also supports NVFP4 and INT8 dispatch dtypes); PR 24651 2026-07 (AMD fused all-reduce + RMSNorm + per-group FP8, Qwen3.5). Open: PR 29723 (per-token FP8/MXFP4), PR 32479 (SM120 PCIe FP8 DMA ring). PR 17262 (ROCm AITER) closed unmerged.

Modern architectures

The EP dispatch/combine path is MoE-specific and is where most shipped comm quantization lives, because expert routing moves tokens without reducing them.

Ecosystem

None of these touch the PP stage boundary; all are TP or EP collectives. DeepEP's asymmetry (FP8 dispatch, BF16 combine) matches the general pattern that reductions resist low precision while pure data movement tolerates it. SGLang's own flashinfer_comm_fusion.py uses only kAllReduce and kARResidualRMSNorm patterns with no quant epilogue, so the CUDA fusion path is unquantized.

Source adversarially re-checked: corrected here

Mechanism

ANS entropy coding over floating-point exponent fields, extending DietGPU into the communication layer: Uzip-P2P for point-to-point transfers with a split-send pipeline overlapping compression with transmission, and Uzip-NCCL integrating compression into NCCL's persistent kernels (NCCL 2.23.4). Numerical correctness is preserved exactly; no application changes.

Key numbers

Accelerates RL weight synchronization by up to 47.5%; cuts vLLM end-to-end inference latency by up to 10%. Measured lossless ratios: 0.679 for BF16 activations (~1.47x), 0.675 BF16 weights, 0.848 FP32 gradients. That 1.47x is the hard zero-risk control number any lossy wire codec must beat.

Modern architectures

The essential control for our contribution: it shows what the wire buys you with zero quality risk (~10% end-to-end), which is the number lossy wire quantization must beat to justify any accuracy exposure. Also note the ceiling - lossless ratios on activation tensors are far below the 2-4x that int8/fp8/mxfp8 achieve.

Ecosystem

Part of the UCCL stack and integrated into NCCL and vLLM, with Berkeley/AWS-affiliated authors, so closer to deployable infrastructure than most items here.

arXiv:2604.17172 · Code adversarially re-checked: corrected here

Mechanism

send_tensor_dict splits the dict into a metadata list plus a tensor list and isends each tensor as-is. TensorMetadata carries the source dtype, and the receiver allocates torch.empty(value.size, dtype=value.dtype, device=value.device) to match exactly. No scale is computed, transmitted, or applied anywhere on the stage boundary.

Key numbers

Zero quantization call sites in the PP transfer of vllm/distributed/parallel_state.py; identical in sglang srt/distributed/parallel_state.py (send_tensor_dict at line 1685, torch.distributed.isend/send on native-dtype tensors). PP-quantization issue searches return zero hits in vllm, sglang and TensorRT-LLM.

Ecosystem

The gap our work targets. Corroborating: SGLang roadmap #31783 Section 4 lists TP, EP, CP, all-gather, reduce-scatter, all-reduce and disaggregated KV transfer as the axes needing a wire format, omitting PP P2P; TensorRT-LLM's quantization page covers FP4/FP8/NVFP4 weights, activations and KV cache but never mentions communication precision. The only low-precision wire paths upstream are vLLM's SP all-gather and SGLang's TP/EP collectives.

Code · Source adversarially re-checked: confirmed

Mechanism

The section tracks per-path comm quantization as merged PRs plus two unowned generalization items: "Generalize communication quantization beyond individual Qwen3, DeepEP, AITER, and hardware-specific paths" and "Define scale ownership and wire format for TP, EP, CP, all-gather, reduce-scatter, all-reduce, and disaggregated KV transfer."

Key numbers

Issue #31783, opened 2026-07-20, updated 2026-07-29, open. Section 4 holds 3 merged PRs, 3 open PRs and 2 unowned generalization items. Section 3 separately notes: "Add quantized KV transfer for disaggregated serving. No dedicated implementation PR is currently tracked."

Ecosystem

Primary documentary evidence for the PP gap: SGLang treats communication quantization as a first-class 2026 objective, yet never lists the pipeline-stage P2P boundary among the axes needing a wire format or scale-ownership rule. The predecessor roadmap (#8180, opened 2025-07-20, now closed) carried "Communication Quantization" as a single unchecked box.

Source adversarially re-checked: confirmed

Mechanism

Replaces collective-heavy patterns with point-to-point stage transfers overlapped against compute, and dynamically sizes chunks to keep the pipeline full. Communication volume falls because fewer and better-scheduled tensors move; the payload dtype stays the model's activation dtype.

Key numbers

Flags: --pp-size, --chunked-prefill-size, --enable-dynamic-chunking, SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR (default 0.75), SGLANG_PP_LAYER_PARTITION. "the activation precision is FP8 (1 byte)" appears only as an assumption inside the analytical comm-volume formula, not as an implemented wire dtype.

Ecosystem

Useful negative evidence: the most recent and most heavily engineered upstream PP work in either major framework reduces bytes on the stage boundary without touching precision. That leaves wire quantization orthogonal to, and composable with, the state of the art in PP scheduling.

Source extracted from local verified reports

Mechanism

Per-layer, quantize weight columns in sequence and update the not-yet-quantized columns to compensate the induced error, using a Cholesky-factorized approximation of the layer Hessian from a small calibration set. No gradient training; one pass over the calibration data. Activations untouched.

Key numbers

175B params quantized in ~4 GPU hours to 3-4 bits with negligible degradation; first 175B model in a single GPU for generative inference; ~3.25x end-to-end speedup vs FP16 on A100, 4.5x on A6000.

Modern architectures

Still the algorithmic core of 2026 FP4 work: MR-GPTQ (arXiv:2509.23202) is GPTQ specialized to MXFP4/NVFP4 with block-wise Hadamard transforms, and the 2026 NVFP4 scale-search papers use GPTQ-style reconstruction objectives.

Ecosystem

Official repo github.com/IST-DASLab/gptq; GPTQModel is listed among vLLM's supported quantization methods (verified in vLLM docs), and GPTQ checkpoints are a de facto Hugging Face standard.

arXiv:2210.17323 · Code adversarially re-checked: confirmed

Mechanism

Salient weight channels are identified by activation magnitude, not weight magnitude; protecting ~1% of channels captures most of the error. Rather than mixed precision, AWQ derives that scaling up those channels reduces quantization error, and applies an equivalent per-channel scale determined from offline activation statistics. No backprop or reconstruction, so it generalizes off the calibration set.

Key numbers

TinyChat gives >3x speedup over the HF FP16 implementation on desktop and mobile GPUs; runs 70B Llama-2 on mobile GPUs; strong on instruction-tuned and multimodal LMs.

Modern architectures

Because activations are untouched, AWQ is the baseline that isolates how much of the accuracy loss in W4A4/W4A8 is attributable to activations rather than weights, useful framing for a wire-compression result where only the activation side moves.

Ecosystem

Official repo github.com/mit-han-lab/llm-awq plus TinyChat; AWQ is a supported method in vLLM and SGLang, and AWQ checkpoints are ubiquitous on Hugging Face. Note the upstream AutoAWQ package is archived (May 2025); llm-compressor and GPTQModel are the maintained producers.

arXiv:2306.00978 · Code adversarially re-checked: confirmed

Mechanism

Per-channel scale s_j = max|X_j|^a / max|W_j|^(1-a) (a=0.5 default) divides activations and multiplies the corresponding weight rows, a mathematically equivalent transform absorbed into the previous LayerNorm/linear, so no runtime cost. Three efficiency levels: O1 per-token dynamic activations, O2 per-tensor dynamic, O3 per-tensor static. All GEMMs (QKV, O, FFN) run INT8.

Key numbers

Up to 1.56x speedup and 2x memory reduction with negligible accuracy loss; enables serving a 530B model (MT-NLG) within a single node; validated on OPT, BLOOM, GLM, Llama-1/2, Falcon, Mistral, Mixtral.

Modern architectures

Later versions add Llama-1/2, Falcon, Mistral, Mixtral. Known weakness on GLU-based FFNs, where activation spikes are token-local rather than channel-persistent, so channel smoothing does not catch them (arXiv:2405.14428 shows SmoothQuant failing to control these); also weak against massive activations on start/delimiter tokens.

Ecosystem

Official repo github.com/mit-han-lab/smoothquant. vLLM's quantization docs (verified) list LLM Compressor INT8 W8A8 and FP8 W8A8 paths; smoothing-style preprocessing is standard for those W8A8 recipes and is also present in TensorRT-LLM and Atom's pipeline.

arXiv:2211.10438 · Code adversarially re-checked: confirmed

Mechanism

Two equivalent per-channel operations before the GEMM: a shift that removes the channel-wise asymmetry (recentering each channel) and a scale that removes concentration. Both are absorbed into subsequent weights/bias so inference cost is unchanged. Scale/shift chosen to minimize a quantization-error objective rather than the heuristic max-ratio of SmoothQuant.

Key numbers

Near-floating-point accuracy at 8-bit and 6-bit on BERT, OPT, BLOOM, BLOOMZ and LLaMA; +15.5% for 4-bit BERT over prior work.

Modern architectures

Evaluated on pre-GLU-era and early Llama models; the shift term is what later work generalizes into per-block scale/offset asymmetry, which is exactly what AMXFP4 (arXiv:2411.09909) re-derives inside microscaling blocks.

Ecosystem

Official repo github.com/ModelTC/Outlier_Suppression_Plus, from the ModelTC group that also maintains LightLLM; the shift+scale pair is a common building block in later PTQ toolkits.

arXiv:2304.09145 · Code adversarially re-checked: confirmed

Mechanism

Three parts: (1) fine-grained hardware-friendly scheme, group-wise quantization for weights, token-wise dynamic quantization for activations, so no calibration of activation ranges is needed; (2) layer-by-layer knowledge distillation (LKD) that recovers INT4 FFN weights without original training data; (3) an optimized backend that fuses quant/dequant to remove the conversion overhead.

Key numbers

INT8 W8A8 essentially cost-free for BERT and GPT-3-style models: up to 5.19x/4.16x speedup vs FP16. W4(FFN)/W8(attn)/A8 gives 3x memory reduction but needs LKD layer-by-layer distillation (so that path is training, not training-free). Applied to GPT-J-6B and GPT-NeoX-20B with up to 5.2x better efficiency.

Modern architectures

Per-token dynamic activation scaling introduced here is still the default in W8A8 serving; note that per-token granularity is exactly what breaks down at 4 bits (see PrefixQuant on token-wise outliers and the microscaling papers).

Ecosystem

Landed in DeepSpeed as DeepSpeed-Inference/Compression; the follow-ups ZeroQuant-V2 and ZeroQuant-FP (arXiv:2307.09782, W4A8 with FP8 activations) extend the same line.

arXiv:2206.01861 adversarially re-checked: corrected here

Mechanism

Channels are permuted and grouped into clusters of similar range; each cluster gets its own activation scale, so a wide channel no longer sets the scale for narrow ones. The permutation is folded into the LayerNorm parameters and the following linear weights, so no runtime reorder kernel is needed.

Key numbers

First reported 3-bit activations in LLMs; up to 80% memory reduction when quantizing OPT-175B.

Modern architectures

The clustering assumption, outliers sit in fixed channels across tokens, holds for channel-persistent outliers but not for the token-local activation spikes of GLU FFNs. OSC (arXiv:2604.12782) re-validates the token-persistent channel clustering on Qwen3 in 2026, so the assumption survives into modern dense/MoE models for most tensors.

arXiv:2304.01089 · Code adversarially re-checked: confirmed

Mechanism

Systematic PTQ study of E5M2, E4M3 and E3M4 across tasks, trading dynamic range against precision, with a standardized workflow (format choice per operator type, scale calibration, expanded operator coverage including non-GEMM ops). Weights and activations both go FP8; the wider exponent removes the need for aggressive outlier handling that INT8 requires.

Key numbers

FP8 beats INT8 on workload coverage 92.64% vs 65.87% across 75 architectures, and on accuracy and operator suitability; E4M3 best for NLP, E3M4 marginally better for vision.

Modern architectures

FP8 W8A8 is now the standard lossless-quality serving point on Hopper/Blackwell for dense and MoE models, which matches wire-quantization results where int8/fp8 are quality-neutral. The study's own recommendation is E4M3 for NLP; E3M4 was marginally better on vision tasks.

Ecosystem

Code in Intel Neural Compressor (github.com/intel/neural-compressor). vLLM docs (verified) list FP8 W8A8 via LLM Compressor plus NVIDIA Model Optimizer, AMD Quark and INC as supported paths.

arXiv:2309.14592 · Code adversarially re-checked: corrected here

Mechanism

Combines four things: outlier channels kept at higher precision while the rest go INT4; fine-grained group-wise (group-128) quantization for both weights and activations; activation scales computed dynamically per group at runtime; and an INT4 asymmetric KV cache. Kernels fuse quantization/dequantization into the GEMM so the INT4 tensor cores are actually reachable in a batched serving loop.

Key numbers

Up to 7.7x end-to-end throughput (token/s) over FP16 and 2.5x over INT8 at matched latency (repo reports 7.73x / 2.53x), with negligible accuracy loss.

Modern architectures

Targets Ada-generation INT4 tensor cores (kernels optimized for RTX 4090, sm_89) on dense Llama-family models; predates GQA-heavy and MoE architectures and predates hardware microscaling, so its group-128 scheme is coarser than MXFP4 (32) or NVFP4 (16). Direct ancestor of the group-granularity choice in QoQ and COMET.

Ecosystem

Official repo github.com/efeslab/Atom (verified), which reuses code from SmoothQuant, GPTQ and FlashInfer; includes accuracy eval, kernel benchmarks and end-to-end serving evaluation.

arXiv:2310.19102 · Code adversarially re-checked: corrected here

Mechanism

QoQ uses progressive two-level weight quantization (INT4 group scales nested inside INT8 per-channel scales) so W4A8 GEMM dequantizes with cheap INT8 arithmetic; activations are per-token dynamic INT8; SmoothAttention migrates KV outliers so the KV cache can go INT4. The QServe runtime adds compute-aware weight reordering, register-level parallelism, and a fused attention kernel made memory-bound by KV4.

Key numbers

Existing INT4 methods carry 20-90% dequantization runtime overhead. QServe raises max throughput 1.2x (A100) / 1.4x (L40S) on Llama-3-8B and 2.4x / 3.5x on Qwen1.5-72B vs TensorRT-LLM; L40S beats TensorRT-LLM on A100; ~3x lower dollar cost.

Modern architectures

The dequantization-overhead argument is architecture-independent and still frames the W4A4-vs-W4A8 debate on Blackwell; LiquidGEMM (arXiv:2509.01229) pushes W4A8 kernels a further 2.9x. Supports Llama, Qwen, Mistral, Yi in the released engine.

Ecosystem

Code at github.com/mit-han-lab/omniserve (verified): a unified engine merging QServe and LServe, supporting W4A8KV4/KV8, W8A8KV4/KV8, per-channel or g128, paged attention and in-flight batching.

arXiv:2405.04532 · Code adversarially re-checked: confirmed

Mechanism

Identifies the small set of tokens that carry extreme activations and prefixes them into the KV cache so they never appear in the quantized activation tensor; this is training-free and takes about a minute even for Llama-3-70B. Optional block-wise trainable parameters then compensate residual quantization error.

Key numbers

+3.08 and +2.85 average points over SpinQuant on five zero-shot reasoning tasks for W4A4KV4 Llama-3-8B (dynamic and static settings); outlier-token prefixing itself is training-free and takes about 1 minute for Llama-3-70B (optional block-wise trained compensation improves it further); up to 2.74x prefill and 2.16x decode speedup at W4A4.

Modern architectures

Most relevant item for anyone whose per-token 4-bit activation quantization degrades: it shows the failure is driven by a few token positions (start/delimiter tokens, the massive-activation phenomenon), and that removing those tokens from the quantized path recovers accuracy without rotation. Compares against rotation baselines but is not itself a rotation method.

Ecosystem

Code at github.com/ChenMnZ/PrefixQuant, from the OmniQuant authors. Closely related to CushionCache (arXiv:2406.12016), which also prefixes attention-sink tokens to mitigate activation outliers.

arXiv:2410.05265 · Code adversarially re-checked: confirmed

Mechanism

Establishes that outlier channels are largely predictable and token-invariant, then builds a dual-path design: a 4-bit path for the bulk of channels and a 16-bit path for the offline-identified outlier channels, extracted during inference. For down-projection inputs, where channel clustering is less pronounced, it falls back to FP8, making the design effectively three-tier. Because the channel set is static, routing stays hardware-friendly.

Key numbers

Accuracy drops of 2.19 and 1.12 points on Qwen3 models; up to 1.78x speedup over the W8A8 GEMM baseline on a modern AI accelerator.

Modern architectures

The offline channel-invariance claim is the practical alternative to grouping: rather than a per-group scale absorbing the outlier, you remove the outlier channels entirely. Directly comparable to our nvfp4 result and a useful ablation axis - channel separation vs microscaling as two answers to one problem.

arXiv:2604.12782 adversarially re-checked: corrected here

Mechanism

Starts from the observation that massive activations with extreme values dominate the quantization range and that static smoothing methods (SmoothQuant-style) cannot fully resolve them. Instead of a global transform, it analyses residual-stream dynamics to identify quantization-sensitive layers, then assigns those layers 8-bit while keeping 4-bit elsewhere - mixed precision allocated by measured sensitivity rather than a fixed heuristic.

Key numbers

Improved perplexity and downstream performance under W4A4KV4 on LLaMA models, with 1.05-1.07x throughput improvement at modest memory (not compute) overhead. The 8-bit escapes are activation-only; weights and KV stay 4-bit. Sensitivity is detected via two metrics, Jump Ratio and Historical Feature SNR, and it composes with rotation methods (QuaRot/SpinQuant/FlatQuant).

Modern architectures

The layer-heterogeneity result matters for pipeline work: if sensitivity is concentrated in identifiable layers, a stage boundary landing on a sensitive layer will behave much worse than one that does not. Suggests boundary placement, not just boundary datatype, is a tunable.

arXiv:2606.12487 adversarially re-checked: corrected here

Mechanism

Builds a strong FP-PTQ baseline by jointly searching the exponent/mantissa split and the clipping range per tensor. Observing high inter-channel and low intra-channel variance in activations, it applies per-channel activation quantization and then absorbs those extra scales into the weights as exponential biases, so the GEMM stays a plain FP4 matmul.

Key numbers

LLaMA-13B W4A4 averages 63.1 on common-sense zero-shot reasoning, 5.8 below full precision and 12.7 points above the previous SOTA.

Modern architectures

Pre-microscaling: its per-channel FP4 scales are the coarse-grained alternative that hardware MXFP4/NVFP4 replaced with per-block shared scales. The inter-channel/intra-channel variance observation it reports across LLMs, BERT and ViTs is the empirical justification for block scaling.

Ecosystem

Official repo github.com/nbasyl/LLM-FP4; simulation-only (no FP4 tensor cores existed in 2023), so numbers are accuracy rather than measured speedup.

arXiv:2310.16836 · Code adversarially re-checked: confirmed

Mechanism

Replaces the single shared scale of an MX block with an asymmetric pair, so positive and negative tails of a 4-bit group are scaled independently. Pure direct-cast: no calibration set, no rotation matrices, no multi-hour preprocessing. A custom MAC engine handles the asymmetric scales at negligible hardware cost.

Key numbers

Outperforms MXFP4 by 3% on VQA and exceeds rotation-based INT4 methods by 1.6% on CSQA; surpasses deployed commercial MXFP4 variants. Notes rotation-based INT4 adds multi-hour calibration and leaves key ops in full precision.

Modern architectures

The asymmetry-vs-outlier-suppression tradeoff is the mechanism-level reason microscaled FP4 rescues 4-bit where per-token INT4 fails, and why the scale representation (FP8 vs power-of-two) matters, the same axis MR-GPTQ later quantifies for MXFP4 vs NVFP4.

Ecosystem

Code at github.com/aiha-lab/MX-QLLM (verified): an MX-format LLM inference framework built on Microsoft's microxcaling, with PoT and FP8 scale modes and an AMXFP4-FP8 config (fp4_e2m1_asym elements, E5M2 shared scale).

arXiv:2411.09909 · Code adversarially re-checked: confirmed

Mechanism

Scores each weight/activation block by its quantization perturbation weighted by Fisher information, keeping high-impact blocks in FP8 and the rest in NVFP4, plus sensitivity-weighted clipping for blocks that do go low. Hardware adds a block-granularity mixed-precision datapath and a runtime unit that assigns activation blocks to high or low precision on the fly with minimal energy overhead.

Key numbers

<1% Wikitext-103 perplexity degradation for Llama-2-7B relative to an all-FP8 baseline, with 14% lower inference energy and 30% less weight memory.

Modern architectures

Prototyped on NVFP4/FP8, i.e. the actual Blackwell datatype pair; the per-block activation decision made at runtime is the design point for architectures whose outlier location is input-dependent (GLU FFNs, MoE experts) rather than fixed by channel. Requires datapath support, so it is not deployable on stock GPUs today.

Ecosystem

No public code released. Complements FGMP-free approaches like ARCQuant that deliberately keep a strictly unified format to stay on standard GEMM kernels.

arXiv:2504.14152 adversarially re-checked: confirmed

Mechanism

Explicitly rejects the three existing families for NVFP4, rotation breaks fine-grained block isolation, smoothing cannot absorb 4-bit error, mixed precision violates unified-precision hardware constraints. Instead it augments the activation matrix with extra quantized residual channels so error compensation happens inside the matmul's reduction dimension (dual-stage quantization), leaving the GEMM shape and format untouched.

Key numbers

Worst-case error bound of the dual-stage NVFP4 scheme is comparable to 8-bit formats such as MXFP8; accuracy comparable to full precision in perplexity and downstream tasks on LLaMA and Qwen; up to 3x speedup over FP16 on RTX 5090 and RTX PRO 6000.

Modern architectures

Evaluated on current LLaMA and Qwen models on Blackwell-generation consumer/workstation GPUs. The 'error bound comparable to MXFP8' result is the cleanest theoretical statement that a properly compensated 4-bit microscaled format can reach 8-bit-class fidelity.

Ecosystem

Code at github.com/actypedef/ARCQuant. Same group's follow-up SharQ (arXiv:2606.26587) combines N:M activation sparsity with FP4 and reports NVFP4/HiF4/MXFP4 coverage on Qwen3-30B-A3B and Qwen3-VL-8B.

arXiv:2601.07475 · Code adversarially re-checked: confirmed

Mechanism

Sweeps 7 PTQ algorithms x 15 benchmarks x 3 model families under MXFP8 (E4M3) and MXFP4 (E2M1) with block size 32, in weight-only, weight-activation and KV-cache configurations. Identifies scale-factor quantization as a critical error source and proposes a pre-scale (multiply inputs by 3/4 before quantizing) to reduce MXFP4 clipping bias.

Key numbers

MXFP8 W8A8 ~99-100% accuracy recovery. MXFP4: W4A8 95.44-98.83%, W4A4 86.37-96.79%. Pre-scale lifts one worst case from 52.39% to 56.76% recovery.

Modern architectures

Independent confirmation of the mxfp8-lossless / mxfp4-fragile split, and it localizes the 4-bit failure to the block scale rather than the mantissa - exactly the axis NVFP4 changes (E4M3 scale per 16 values instead of E8M0 per 32). Caveat: only 7-8B dense and VL models, no MoE, no hybrid linear attention, and no NVFP4 arm, so frontier scale and depth are untouched.

Ecosystem

Models evaluated: Llama-3.1-8B-Instruct, openPangu-Embedded-7B-V1.1, openPangu-VL-7B, Qwen2.5-VL-7B. Authors explicitly flag missing 30B+ scale and missing NVFP variants as limitations.

arXiv:2601.09555 adversarially re-checked: confirmed

Mechanism

Different treatment per tensor position: inputs to attention and FFN go INT4/FP4, while the intermediate states (post-GLU FFN activations and attention output projection input, where outliers concentrate) are sparsified and then quantized to 8 bits. Trained by continuing from a BitNet b1.58 recipe, so it needs training, not calibration.

Key numbers

Matches BitNet b1.58 quality at equivalent training cost while enabling 4-bit kernels; activates only 55% of parameters and supports a 3-bit KV cache.

Modern architectures

The positional split is the key transferable lesson: in GLU-based FFNs the down-projection input is the tensor that resists 4 bits, consistent with the GLU activation-spike literature and with per-tensor-position findings in FP4 PTQ work. Applies to a from-scratch 1-bit model family, not off-the-shelf checkpoints.

Ecosystem

No official code released with the paper; the surrounding BitNet family ships inference kernels via microsoft/BitNet (bitnet.cpp), but that is not verified here as containing a4.8 support.

arXiv:2411.04965 adversarially re-checked: confirmed

Mechanism

Quantization-aware distillation: the NVFP4-quantized student is trained against the FP teacher's output distribution under KL divergence, rather than re-running task-loss QAT. Reported to be markedly more stable than QAT for checkpoints that went through SFT, RL and model merging, and robust to calibration/distillation data quality and coverage, so full training data is not required.

Key numbers

Positioned as accuracy recovery for NVFP4 LLMs/VLMs; the headline claims are stability on multi-stage post-trained models where QAT is unstable, and robustness to data quality rather than a single speedup number.

Modern architectures

Directly addresses today's checkpoints, instruction-tuned, RL-trained, merged LLMs and VLMs, where naive PTQ or QAT breaks. This is the training-side counterpart to the PTQ items and the practical answer when NVFP4 PTQ alone leaves a quality gap.

Ecosystem

No code URL in the paper; the surrounding NVFP4 tooling is NVIDIA TensorRT Model Optimizer, which vLLM's docs list as a supported quantization path (verified). Large NVIDIA author list including Song Han and Tijmen Blankevoort.

arXiv:2601.20088 adversarially re-checked: confirmed

Mechanism

QuIP (arXiv:2307.13304, verified) pre/post-multiplies weights by random orthogonal matrices to bound incoherence, then adaptively rounds against a Hessian proxy. QuIP# replaces those with the randomized Hadamard transform (faster, better theoretical coherence mu=1/sqrt(n)), vector-quantizes the resulting ball-shaped weights with E8-lattice codebooks, and optionally fine-tunes.

Key numbers

First method where 3-bit models scale better than theoretically lossless 4-bit. CUDA kernels on RTX6000 ADA at 2-bit: 176 tok/s for 7B, 21.9 tok/s for 70B. Fine-tuning is optional (--ft_epochs 0 reproduces the no-FT result).

Modern architectures

Weight-only: activations stay 16-bit, so it does nothing for a stage-boundary wire or for compute-bound prefill. Included because every later activation-rotation method inherits its incoherence argument, and the randomized Hadamard transform it introduced is exactly the transform QuaRot and SpinQuant later fuse into weights.

Ecosystem

llm-compressor ships a QuIPModifier (transform_type 'hadamard' via Sylvester construction) alongside SpinQuantModifier; its docs note input rotations (v) are on by default while output rotations (u) are off because they are not performant.

arXiv:2402.04396 · Code adversarially re-checked: confirmed

Mechanism

Computational invariance y=Wx=(WQ^T)(Qx). A global randomized Hadamard Q is fused offline into embedding and residual-adjacent weights; head-wise Hadamards fold into v_proj/o_proj; online (unfused) Hadamards run before FFN down_proj, on post-RoPE K/Q, and across heads before o_proj. Weight rounding via GPTQ (needs calibration) or RTN (calibration-free at 6/8 bit).

Key numbers

LLaMA2-70B W4A4KV4: <=0.47 WikiText-2 PPL loss, ~99% of zero-shot retained. 6/8-bit lossless via RTN with zero calibration data. RTX 3090 kernels: 1.97-2.16x prefill at 7B, up to 3.33x at 70B, 3.89x decode-stage memory saving.

Modern architectures

Evaluated on Llama-2 7/13/70B, Llama-3 8/70B, Phi-3-mini; no MoE model in the paper, and the official repo covers Llama-family only. The rotation graph is derived for RMSNorm+GQA decoders, so MoE (per-expert down_proj) and MLA (compressed latent KV, decoupled RoPE) need the invariance argument re-derived per projection.

Ecosystem

Reference CUDA kernels; depends on Dao-AILab/fast-hadamard-transform. QuaRot/SpinQuant-style transforms are reimplemented in vLLM's llm-compressor as SpinQuantModifier and QuIPModifier, with hadacore Hadamard kernels providing accelerated inference in vLLM. 532 stars, Apache-2.0.

arXiv:2404.00456 · Code adversarially re-checked: confirmed

Mechanism

Two learnable components optimized by differentiable block-wise reconstruction-error minimization. LWC (Learnable Weight Clipping) tunes weight clipping thresholds; LET (Learnable Equivalent Transformation) learns per-channel scales and shifts that are folded into adjacent weights and biases, so activations are flattened with zero runtime cost. Non-orthogonal and diagonal-only; calibration data and a few GPU-hours, no full finetuning.

Key numbers

Llama-2 7B-70B quantized on a single A100-40G in 1-16 hours with 128 calibration samples; SOTA at the time across W4A4/W6A6/W4A16/W3A16/W2A16.

Modern architectures

Its W4A4 gap is precisely what motivated the rotation line: diagonal scaling cannot remove the massive down_proj/residual outliers that a dense rotation can. LET has zero online cost because it folds entirely into weights, making it the cheapest possible transform for a wire, but the weakest at 4-bit activations. The learnable-scale idea is the direct ancestor of the 2026 NVFP4 scale-optimization papers.

Ecosystem

Official repo github.com/OpenGVLab/OmniQuant (907 stars, MIT). The authors ship a demo Android/iOS app built by compiling OmniQuant W3A16/W2A16 Llama-2 checkpoints through MLC-LLM; no commercial deployment is claimed.

arXiv:2308.13137 · Code adversarially re-checked: corrected here

Mechanism

Identifies four rotation slots in a Llama-style block: R1 (residual/embedding), R2 (head-wise value), R3 (post-RoPE K/Q), R4 (FFN down_proj input). R1/R2 merge into weights offline; R3/R4 stay as online Hadamards. R1/R2 are optimized on the Stiefel manifold with Cayley SGD on a small calibration set to minimize the quantized network's loss.

Key numbers

LLaMA-2 7B W4A4KV4: 2.9-point zero-shot gap to FP16, beating LLM-QAT by 19.1 and SmoothQuant by 25.0 points. LLaMA-3 8B: closes up to 45.1% of the gap QuaRot leaves. Spread between random rotation seeds: up to 13 points.

Modern architectures

Rotation learning needs the whole model resident for backprop (the complaint BASE-Q and DartQuant target); KurTail reports >=4x H100 80GB for Llama3-70B. Slots are defined for Llama-style blocks, so an MoE needs an R4 per expert and MLA needs R3 redefined around the decoupled-RoPE key path.

Ecosystem

Meta's official quantized Llama 3.2 1B/3B release (Oct 2024) ships a SpinQuant PTQ variant alongside a QAT+LoRA variant, reporting up to 2-4x inference speedup, ~56% smaller models and ~41% lower memory on mobile CPUs (Arm/MediaTek/Qualcomm). llm-compressor's SpinQuantModifier reproduces R1-R4, with R4 (down_proj) added in a 2025 release.

arXiv:2405.16406 · Code adversarially re-checked: confirmed

Mechanism

Block-diagonal rotation matrices constructed greedily from identified outlier dimensions redistribute 'massive' outliers (concentrated at the FFN down_proj input) into adjacent channels; a zigzag channel permutation equalizes outlier mass across blocks, reducing block-wise variance; a second rotation smooths the result. Transforms fold into weights where the graph allows.

Key numbers

NeurIPS 2024 Oral; SOTA W4A4 perplexity and zero-shot accuracy at publication across LLaMA-1/2/3, Vicuna-v1.5 and Mistral at 7B-70B. Pipeline needs a one-off rotation build (get_rot.py) plus per-model activation scale/shift calibration.

Modern architectures

Official repo lists LLaMA-1/2/3, Vicuna-v1.5 and Mistral only, no Mixtral/MoE. Its central diagnosis (massive outliers localized to the down_proj input, distinct from ordinary per-channel outliers) is the mechanism that recurs inside MoE expert FFNs, and DuQuant++ later re-targets the same construction at MXFP4 blocks.

arXiv:2406.01721 · Code adversarially re-checked: confirmed

Mechanism

Each linear layer gets a learnable affine transform (not restricted to Hadamard/orthogonal), fitted by a lightweight gradient objective in hours. Observing that Hadamard-rotated tensors are still steep and dispersed, it optimizes flatness directly. The transform is factored as a Kronecker product of two small matrices, and transform plus quantization are fused into one Triton kernel (the INT4 GEMM itself stays a separate CUTLASS kernel).

Key numbers

LLaMA-3-70B W4A4: under 1% accuracy drop, +7.5% over SpinQuant. 2.3x prefill and 1.7x decode speedup vs FP16. End-to-end transform overhead is 0.07x of runtime for FlatQuant's five learned transforms, vs 0.26x for QuaRot's three online Hadamards.

Modern architectures

Per-layer transforms imply per-expert transforms on an MoE: parameter count, calibration cost and online matmuls all scale with expert count. Learned affine transforms give the strongest 4-bit accuracy in this list but are the least hardware-neutral, they only pay off with the fused kernel (kron_matmul/block_matmul), and the transform cannot be folded away entirely.

arXiv:2410.09426 · Code adversarially re-checked: corrected here

Mechanism

Every fully-connected layer gets a learnable (orthogonal, scaling) transformation pair: the orthogonal part rotates away outliers, the scaling part stretches the distribution to fill the quantization grid, both guided by the QSUR metric derived analytically. Trained with a KL-Top loss over the FP model's top logits to extract more signal from the small PTQ calibration set.

Key numbers

W4-only: retains 99.5% of floating-point accuracy. W4A4KV4 LLaMA-3-8B: closes 32% of the gap left by prior state-of-the-art. BASE-Q later reports closing a further 29.2% of the remaining FP gap over OSTQuant.

Modern architectures

Per-FC transform pairs mean the parameter and calibration cost multiplies by expert count on an MoE, and the learned rotation shares SpinQuant's full-model-backprop memory problem. QSUR is useful independently of the method as a cheap diagnostic for how much of the grid an activation tensor actually occupies before you quantize it.

arXiv:2501.13987 · Code adversarially re-checked: confirmed

Mechanism

Distribution-aware rotational calibration: constrain the shape of activations after rotation rather than fine-tuning end-to-end, which cuts both compute and overfitting to the calibration task. QR-Orth keeps the matrix orthogonal via a QR parameterization instead of expensive alternating/Cayley optimization, so the rotation can be fitted layer-locally.

Key numbers

47x faster and 10x less memory for rotation optimization on a 70B model; first rotational calibration of a 70B model on a single RTX 3090. Llama-3-70B W4A4KV16: 7.99 PPL / 69.39 zero-shot vs SpinQuant 9.61 / 66.06 and OSTQuant 7.67 / 67.94.

Modern architectures

The practical answer to the memory wall that blocks learned rotations on very large models. One of the few rotation papers with MoE datapoints: Mixtral-8x7B and DeepSeek-MoE are evaluated alongside Llama-2 7/13/70B and Llama-3 8/70B, so it is the best available evidence that learned rotations transfer to expert FFNs.

arXiv:2511.04063 · Code adversarially re-checked: confirmed

Mechanism

PCA over calibration activations finds the low-rank subspace where activation variance concentrates. Coefficients in that subspace stay at 8-bit while the complementary subspace goes to 4-bit; an invariant random rotation is applied within each subspace to suppress residual outliers. Most projections fold into adjacent weights, but the down_proj input transform runs at runtime (like QuaRot's R4), and further transforms act inside attention (v_proj/o_proj and query/key).

Key numbers

Up to 33% lower WikiText perplexity than SpinQuant on Llama and Qwen2.5 families; up to 3x speedup over the 16-bit baseline. Claimed provably optimal mixed-precision split under its error model.

Modern architectures

The 'keep a rank-r slice in high precision, quantize the rest hard' recipe is the closest published analogue to a mixed-precision activation wire format, and it acts on the residual stream rather than attention internals, which makes it among the easiest here to port to MLA or MoE. Cost is a per-model PCA over calibration data.

arXiv:2412.14363 · Code adversarially re-checked: corrected here

Mechanism

Step 1 shifts outliers from activations into weights (smoothing). Step 2 splits W into a high-precision low-rank branch obtained by SVD plus a low-bit residual branch, so both sides see easier distributions. The Nunchaku engine fuses the low-rank branch's kernels into the low-bit kernels to remove the extra activation memory traffic that would otherwise cancel the speedup; off-the-shelf LoRAs plug in without re-quantization.

Key numbers

SDXL, PixArt-Sigma, FLUX.1. 12B FLUX.1: 3.5x memory reduction and 3.0x speedup over the W4A16 baseline on a 16GB laptop 4090 (INT4); 3.1x over W4A16 on RTX 5090 using NVFP4.

Modern architectures

Diffusion transformers rather than LLMs, but the cleanest demonstration that a rank-r high-precision side channel plus a 4-bit main path beats pure transform/smoothing methods at 4-bit activations, and one of the first published NVFP4 activation results. Fusing the side branch into the low-bit kernel is what makes the rank-r channel free.

Ecosystem

Quantizer: nunchaku-ai/deepcompressor; runtime: nunchaku-ai/nunchaku (renamed from mit-han-lab, 3.9k stars, Apache-2.0). Heavily deployed in ComfyUI/Diffusers workflows, making it arguably the most battle-tested 4-bit weight+activation stack outside LLM serving, and a working reference for NVFP4 activation kernels.

arXiv:2411.05007 · Code adversarially re-checked: corrected here

Mechanism

Adds (i) bias correction so nonzero post-rotation channel means no longer inflate the symmetric quantization range, and (ii) asymmetric per-channel scaling to trade rounding against clipping error. Both are optimizable blockwise, removing the need for SpinQuant/OSTQuant-style full-model backprop caused by rotation parameter sharing across the residual stream.

Key numbers

Narrows the accuracy gap to full precision by 50.5%, 42.9% and 29.2% relative to QuaRot, SpinQuant and OSTQuant respectively. Blockwise optimization only, no full-model backprop. Official code released (MIT); its example runs W4A4KV4.

Modern architectures

The Gaussianization point matters most at very low bit width: a rotation lightens the tail but widens the bulk, so clipping choice becomes the binding constraint rather than outliers. This is the published mechanism closest to explaining why a single per-token int4 scale degrades while a microscaled format with per-16/32 scales does not.

arXiv:2506.15689 · Code adversarially re-checked: corrected here

Mechanism

Analysis shows block-rotation outlier suppression is bounded by input-vector geometry, minimized when blockwise l1 norms are equal. PeRQ = Permute, Rotate, then Quantize: a greedy mass-diffusion algorithm finds a channel permutation equalizing blockwise l1 norms, and the permutation is merged into weights before deployment so inference cost is unchanged.

Key numbers

Llama3-1B INT4 with rotation block size 16: PeRQ recovers up to 90% of the perplexity achievable with full-vector rotation, versus 46% without permutations. Author group affiliated with AMD/Brevitas.

Modern architectures

Explains both DuQuant's zigzag permutation and why small-group formats constrain what rotation can achieve, the studied block size 16 is exactly NVFP4's group size. Permutation is the cheapest preconditioner available for a wire because it folds entirely into weights and costs nothing at runtime, unlike an online Hadamard.

arXiv:2601.22347 adversarially re-checked: confirmed

Mechanism

A GPTQ variant specialized to microscaling FP4: block-wise Hadamard transforms sized to the format's scaling group plus format-specific scale handling, instead of a global rotation. Rotations are fused into weights where possible and computed online for activations by fast QuTLASS kernels, so the transform adds negligible overhead.

Key numbers

Up to 3.6x layer-wise and 2.2x end-to-end vs FP16 on NVIDIA B200; 6x layer-wise and 4x end-to-end on RTX 5090. Lifts MXFP4 accuracy to within 1-2% of NVFP4. First comprehensive MXFP4-vs-NVFP4 PTQ study.

Modern architectures

Paper title: 'Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization'. Its headline conclusion is that FP4 is not an automatic upgrade over INT4 and that SOTA PTQ methods struggle with it. Two findings dominate: NVFP4's 16-element group size provably neutralizes traditional outlier mitigation (so rotation buys less there), and MXFP4's power-of-two scale quantization is the dominant error term. Read it as the strongest evidence that FP4 needs format-specialized PTQ, and that NVFP4 is the more robust of the two formats. Repo supports Llama and Qwen3; kernels need Blackwell.

Ecosystem

Kernels released as QuTLASS (github.com/IST-DASLab/qutlass): CUTLASS-based MXFP4/NVFP4/MXFP8 microscaling GEMMs for sm_100a (B200) and sm_120a (RTX 5090), CUDA 12.8+, with FlashInfer integration. Runtime via the fp_quant PyTorch package; HF Transformers exposes an FP-Quant backend; vLLM support merged 2025-10 (PR 24440).

arXiv:2509.23202 · Code adversarially re-checked: confirmed

Mechanism

Root cause analysis: MXFP4's power-of-two shared block scale reconstructs large values poorly (error grows sharply with magnitude), while a global rotation spreads outlier energy into otherwise-benign blocks and inflates their scales. Fix: apply independent rotations inside each 32-channel block so the transform's support matches the scale granularity.

Key numbers

MXFP4 W4A4, block vs global rotation: LLaMA-3.2-1B PPL 12.78→11.95 and 48.83→49.87% avg accuracy; Qwen2.5-1.5B 12.80→12.15 and 53.50→54.83%; LLaMA-3-8B 7.68→7.14 and 61.57→63.54%. GPTQ stays strong under MXFP4; global-rotation methods do not.

Modern architectures

The single most directly relevant result for microscaled wire formats: the transform's support must match the scaling group, and a globally-rotated tensor can be strictly worse than an unrotated one once scales are power-of-two. Models span LLaMA-2/3, Mistral, Qwen from 1B to 70B; MXFP4 only, NVFP4 not covered.

arXiv:2511.04214 adversarially re-checked: confirmed

Mechanism

A single outlier-aware fine-grained rotation with block size B=32 matched to the microscaling group. Because each MX group carries its own scale, the cross-block variance problem that forced DuQuant's second rotation plus zigzag permutation disappears, so one rotation replaces the whole pipeline, halving online rotation cost while also smoothing the weight distribution.

Key numbers

LLaMA3-8B MXFP4 W4A4: 6.88 WikiText2 PPL / 67.1% zero-shot vs 7.29 / 66.1% for MR-GPTQ (FP16 6.14). LLaMA3.2-3B: 8.63 vs 8.79; >50% relative gain over QuaRot (17.95→8.87). LLaMA3.1-8B-Instruct 7.89 vs FP16 7.21. Beats QuaRot, FlatQuant and MR-GPTQ across settings.

Modern architectures

Argues explicitly that data-agnostic rotations (randomized Hadamard, learned global rotations) are insufficient for microscaled FP4 because they cannot target the channels where outliers actually concentrate, outlier-aware, statistics-derived rotations are needed. LLaMA-3 family only; the paper does not address MoE models or NVFP4.

arXiv:2604.17789 · Code adversarially re-checked: confirmed

Mechanism

Exploits the observation that activation distributions in LLaMA-architecture models are zero-mean and unimodal, so low-magnitude entries are non-salient and can be zeroed. Per-layer thresholds are fit on a small calibration set (uniform, or block-wise greedy allocation). Zeros let the kernel skip corresponding weight columns/rows, so the saving is in GEMM work and weight fetch, not in a narrower datatype.

Key numbers

40-50% model-wide sparsity with minimal degradation; 1.53x decode speedup at 40%, 1.8x at 50%; Llama-2, Llama-3, Mistral at 7B-70B; composes with weight quantization.

Modern architectures

Explicitly targets non-ReLU (SwiGLU/GELU) models, which is why it supersedes ReLU-era sparsity assumptions. Sparsity is input-dependent per token, so it is orthogonal to (and composable with) a fixed wire datatype at a stage boundary.

Ecosystem

Repo (FasterDecoding/TEAL): FP16-only Triton kernels described by the authors as a proof of concept; quantized sparse kernels are unsupported, so composition with weight quantization is accuracy-level, not a measured speedup.

arXiv:2408.14690 · Code adversarially re-checked: confirmed

Mechanism

Introduces a new non-linear activation function (a cutoff applied to the SwiGLU gate output) whose threshold is set from the empirical activation-magnitude CDF to hit a target sparsity. Downstream up/down projections then skip the masked channels in a custom kernel. Optional parameter-efficient finetuning of ~1% of parameters improves convergence and accuracy over competing sparsification methods.

Key numbers

Within 1-2% of the base model at 50% activation sparsity with no finetuning; ~15% wall-clock generation improvement from the custom kernel (7B models); Mistral-7B, Llama2-7B/13B. Optional LoRA finetuning of ~1% of parameters improves accuracy further.

Modern architectures

Confined to the MLP/gate path rather than model-wide, which is the main axis on which TEAL improves on it; both are gating/thresholding methods rather than datatype methods.

Ecosystem

Repo fetched and confirmed (ScalingIntelligence/CATS): custom GPU kernel, reproduction script, and a README note about an in-progress HuggingFace integration framework.

arXiv:2404.08763 · Code adversarially re-checked: confirmed

Mechanism

Two stages: substitute ReLU for the original activation function, then apply progressive sparsity regularization with a penalty factor increasing smoothly along multi-stage sine curves, so the activation distribution shifts gradually instead of abruptly. A trained predictor anticipates which neurons will be active, letting the runtime skip them.

Key numbers

89.32% sparsity on LLaMA2-7B, 88.80% on LLaMA2-13B, 87.89% on MiniCPM-1B, at quality comparable to the original Swish versions. Speedup depends on the runtime: up to 4.52x via PowerInfer's predictor-based approximate path (vs llama.cpp), 2.44x with the predictor-free accurate custom kernels.

Modern architectures

Requires retraining the activation function itself, so it is a model-production method, not a deployment-time knob. Useful as the boundary marker for how much sparsity is reachable if you are willing to pay training cost.

Ecosystem

Sparsity predictors and converted checkpoints are distributed on HuggingFace under the SparseLLM org (e.g. prosparse-llama-2-7b-predictor), seen in search results rather than fetched.

arXiv:2402.13516 adversarially re-checked: corrected here

Mechanism

Diagnoses NVFP4 activation loss as input-dependent outliers dominating each block's scale factor. For every activation tensor it generates an input-adaptive N:M mask (4:8 in-pairs for NVFP4; 2:4 evaluated with HiF4) to extract an outlier-heavy sparse backbone, quantizes that to FP4, then defines a dense residual relative to the quantized backbone and quantizes it too, so two FP4 passes compensate each other's error. Generalizes across NVFP4, HiF4 and MXFP4. No calibration data, retraining, or per-model tuning.

Key numbers

Recovers 43-63% of the NVFP4-vs-FP16 accuracy gap; 2.2-2.4x latency reduction vs FP16 on RTX 5090; 1.2-1.4x throughput over FP8; up to 1.58x on video generation.

Modern architectures

Directly relevant to our nvfp4 result: it independently identifies per-block scale contamination by outliers as the binding constraint on 4-bit activations, and treats sparsity as the tool for removing that contamination rather than as a separate compression family.

Ecosystem

Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B and Qwen3-VL-8B, so it covers dense, MoE and VL variants. Speedups measured on Blackwell consumer silicon (RTX 5090) with native FP4.

arXiv:2606.26587 · Code adversarially re-checked: corrected here

Mechanism

Builds a calibration-derived projection matrix per activation site and projects the activation into that subspace; because the projection can be fused into the adjacent weight matrix, the GEMM itself shrinks. Framed as model compression achieved through activation dimensionality reduction, which the authors argue is better-posed than weight matrix factorization because the projection is fit to the data distribution actually seen.

Key numbers

50% model compression with 0.18 perplexity increase on GPT3-22B, but reaching that requires pre-training-scale retraining to heal projection error (330B tokens for GPT3, 200B for Llama2, 275B for Nemotron4-15B); only ~20% compression is iso-accurate untrained. The latency win is prefill/TTFT-only; end-to-end throughput is explicitly out of scope.

Modern architectures

The rank axis is orthogonal to the bit axis, so ESPACE-style projection could in principle compose with wire quantization: reduce the dimension first, then quantize the smaller tensor. But the verifier reclassifies ESPACE as a model-production method (retraining at pre-training scale), so the composition is far costlier than it first appears, and nobody has published it at a pipeline boundary.

arXiv:2410.05437 adversarially re-checked: corrected here

Mechanism

Combines progressive and Lanczos decomposition algorithms in a dedicated co-accelerator. Two key techniques: a compute replication methodology that parallelizes the inherently iterative decomposition for a 6.2x speedup, and an output shape-preserving scheme that removes reconstruction cost across consecutive layers so decompositions can be chained without re-materializing full-rank tensors.

Key numbers

6.2x speedup from compute replication; 22% end-to-end latency improvement vs A100 GPU, at modest model quality loss. 12 pages, 13 figures.

Modern architectures

Establishes that the blocker for low-rank activation compression is decomposition latency, not representational quality. That is the key contrast with quantization, where the transform is essentially free and the cost is representational.

Ecosystem

Hardware-architecture paper (cs.AR/cs.PF), so the result is an accelerator design and simulation rather than a deployable kernel; no drop-in GPU path.

arXiv:2510.13147 adversarially re-checked: confirmed

Mechanism

Each activation group gets a shared exponent and a dynamically allocated mantissa bit budget, so precision follows the group's actual dynamic range. An iterative post-training adaptive precision search assigns bit-widths per LLM module to trade accuracy against energy and speed. Bit-plane memory organization and specialized processing units make variable-length groups actually cheaper, not just smaller.

Key numbers

2.4x speedup, 4.0x area efficiency, 3.1x energy efficiency vs baseline across popular LLMs.

Modern architectures

A contemporaneous generalization of MX-style group-shared exponents (the OCP Microscaling spec v1.0 predates it by 14 months) to variable per-group mantissa budgets. It is the cleanest statement of why grouping matters: a group-shared exponent is what stops a single outlier from consuming the whole tensor's range, which is precisely the mechanism behind mxfp4/nvfp4 rescuing 4-bit where per-token int4 fails.

arXiv:2411.15982 adversarially re-checked: corrected here

Mechanism

Identifies residual accuracy loss in fixed microscaling formats and adds minimal metadata to correct it: element-level for activations (2 extra mantissa bits on the top-1 element per 8-value subgroup, group 32) and subgroup-level for weights, paired with algorithm-hardware co-design so the extra metadata does not erase the bandwidth win. Covers both weight and activation quantization.

Key numbers

70.63% reduction in accuracy loss vs MXFP4 and 37.30% vs NVFP4 on LLM benchmarks (algorithmic result). The 1.91x speedup and 1.75x energy saving are SIMULATED (Verilog synthesis, TSMC 28nm) against a state-of-the-art MX accelerator, not measured silicon.

Modern architectures

The most load-bearing external datapoint for our format claim: it independently confirms the MXFP4 < NVFP4 ordering we observe, and quantifies the gap. It also shows NVFP4 is not the end of the line, so our nvfp4-as-robust-choice conclusion should be stated as the best currently-deployable option rather than an optimum.

arXiv:2601.19213 · Code adversarially re-checked: corrected here

Mechanism

Observes that a significant fraction of activations concentrate around zero, which makes the higher-order bits sparse rather than the values themselves. Hardware-software co-design splits each activation tensor into dense and sparse components using precision bitmaps, so common near-zero values are computed on a lower-bit datapath while rare wide-range values retain precision.

Key numbers

Prefill latency reduced 16-24.3% and decode latency 13.5-23.4% across BitNet 3B, Llama2 7B and Llama3 8B.

Modern architectures

A third axis distinct from both quantization and value sparsity: bit-plane sparsity within the activation. Worth noting as evidence that the FP16 activation tensor is redundant in more than one way at once.

Ecosystem

Industry/academic co-design (Intel and Purdue affiliations among the authors); the win depends on datapath support for the bitmap split, so it is not a drop-in GPU kernel.

arXiv:2606.00365 adversarially re-checked: confirmed

Mechanism

Traces strong outliers to heads learning a 'no-op' or partial residual update; achieving the near-zero attention weights that requires pushes pre-softmax logits ever larger during training. Two independent fixes make the no-op cheap: clipped softmax (allows exact zeros without extreme logits) and gated attention (an explicit gate for skipping the update). Applied during pre-training.

Key numbers

Models pre-trained with either modification learn significantly smaller outliers while matching or improving FP task performance, and then quantize to full INT8 activations with no additional effort. Shown on BERT, OPT and ViTs.

Modern architectures

The 'attention wants a no-op' account is the root-cause story behind attention sinks and massive activations, and predicts where activation magnitude concentrates in any residual transformer, including modern GQA/MLA and MoE stacks. Gated attention has since reappeared in production architectures.

Ecosystem

Qualcomm AI Research; no official code URL on the abstract page. Its successor in spirit is Outlier-Safe Pre-Training (arXiv:2506.19697), which prevents outliers via the Muon optimizer, Single-Scale RMSNorm and a learnable embedding projection.

arXiv:2306.12929 adversarially re-checked: confirmed

Mechanism

Two empirical mitigations that exclude the spiking computation from quantization rather than rescaling it: Quantization-free Module (QFeM) leaves the specific FFN linear layers where spikes occur in higher precision, and Quantization-free Prefix (QFeP) pins the spike-carrying prefix tokens so the remaining tokens quantize cleanly. Both are calibration-driven and training-free, and stack on top of existing smoothing.

Key numbers

Spikes localize to the FFN of specific early and late layers and to a couple of tokens rather than being spread over the sequence; the methods improve coarse-grained activation quantization on Llama-2/3, Mistral, Mixtral, SOLAR and Gemma, including cases where SmoothQuant fails.

Modern architectures

The most directly relevant outlier science for current architectures, since GLU/SwiGLU FFNs are universal in Llama, Mistral, Mixtral, Qwen and Gemma. Layer-position dependence (early and late layers worst) is a concrete prediction for any depth-dependent activation compression scheme, including per-stage pipeline boundaries.

Ecosystem

Code at github.com/onnoo/activation-spikes. Complements the channel-persistent outlier picture of SmoothQuant/RPTQ: spikes are token-local, so channel-wise scaling cannot see them.

arXiv:2405.14428 · Code adversarially re-checked: confirmed

Mechanism

Separates the two distinct outlier phenomena that break activation quantization - massive activations (a few tokens/dimensions with enormous magnitude) and channel-wise outliers (persistent high-magnitude channels) - then investigates the underlying mechanisms producing each. Proposes mitigation strategies to eliminate them while preserving accuracy, so downstream quantization sees a better-conditioned tensor.

Key numbers

Study paper; contribution is the mechanistic account and mitigation rather than a headline speedup or perplexity number.

Modern architectures

The channel-wise vs per-token distinction it draws is the crux of our own finding: per-token int4 scaling cannot see a persistent channel outlier, which is why grouping (mxfp4/nvfp4) or channel separation is required. Pair it with OSC, which operationalizes the channel-persistence observation.

arXiv:2505.21670 adversarially re-checked: confirmed

Mechanism

Mode I, Signal Degradation (~4-bit): computational patterns stay intact and quantization error acts as cumulative noise impairing precision, which targeted repair fixes without retraining. Mode II, Computation Collapse (~2-bit): key components stop functioning - attention loses focus (entropy >0.8), FFN gates flip sign >30% of the time, and representational structure collapses in early layers, which compensation cannot repair.

Key numbers

GPTQ at 2/3/4/8-bit primary, AWQ for validation, calibrated on 128 C4 sequences; Llama-3.1-8B primary, validated on Qwen3-8B, Mistral-7B-Instruct-v0.3, Gemma-2-9B-it. Collapse markers: attention entropy >0.8, >30% FFN gate sign flips.

Modern architectures

SCOPE CORRECTION: weight-side, not activation-side. Still the most useful conceptual import for our work - the degradation/collapse dichotomy is exactly the shape of our per-token int4 failure at depth, where error stops being cumulative noise and becomes structural. Use as mechanism analogy, not evidence.

Ecosystem

Verified against the full HTML: the experimental setup states weight-only PTQ explicitly. The abstract alone is ambiguous on this point, which is how it can be mistaken for an activation-quantization paper.

arXiv:2604.19884 adversarially re-checked: confirmed

Mechanism

Builds LLMFI, a configurable and deterministic fault-injection framework, and injects soft errors across multiple models and thirteen representative tasks spanning reasoning, multilingual, mathematical and coding domains. Distills 17 findings about which perturbations are absorbed and which propagate to output corruption, then proposes four software-only reliability mitigations.

Key numbers

17 key findings on error propagation; 4 software-only reliability improvements; thirteen tasks across reasoning, multilingual, math and code.

Modern architectures

SCOPE CORRECTION for the review: this is reliability/soft-error science and contains no quantization. It is legitimate support for the general claim that activation perturbations are unequally consequential, but must not be cited as a quantization result.

Ecosystem

Fault targets are stated: bit flips injected into intermediate ACTIVATIONS (forward hooks), stored WEIGHTS (pre-hooks) and the KV cache, at submodule/layer/transformer-block granularity, on Phi-3.5-mini, DeepSeek-llm-7b-chat and Gemma-7b. Block-granularity activation fault injection is a close analogue to perturbing a pipeline-stage boundary.

arXiv:2606.02430 adversarially re-checked: corrected here

Mechanism

Because weight-tied blocks are reused every recursion step, quantization error is incurred at each step and accumulates with unrolling depth. The authors isolate the cause by holding bit-width and number format fixed while varying scaling granularity: per-tensor 4-bit activation scaling collapses, and MXInt4 (blockwise integer activations with per-block scales) completely restores the accuracy transition.

Key numbers

Per-tensor 4-bit collapses Sudoku exact-solution accuracy from 84.1% to 0.0%; MXInt4 completely restores it. Robust across recursive architectures including equilibrium models, and on open-ended ARC-AGI.

Modern architectures

The closest external corroboration of our central result, in a different geometry: repeated application of one weight-tied block rather than depth across pipeline stages. Their conclusion that collapse is caused by activation-scaling GRANULARITY rather than bit-width or number format restates our per-token-int4-fails-at-depth finding. Caveat that must ship alongside it: the subjects are ~7M-parameter recursive puzzle solvers (TRM, equilibrium models), not LLMs. They also find activation-side sensitivity dominates while weight-only 4-bit stays benign.

arXiv:2607.16237 adversarially re-checked: corrected here

Mechanism

Shows that quantizing the whole inference pipeline degrades agentic performance, but the degradation is concentrated in decode; prefill absorbs FP4 with little loss. Mix-Quant is therefore phase-aware: low-precision FP4 during prefilling, BF16 retained for decoding. This decouples the acceleration target from the accuracy target in multi-turn agent workflows involving planning, tool use and memory retrieval.

Key numbers

Up to 3x prefill speedup while preserving task performance on long-context and agentic benchmarks.

Modern architectures

Strongly relevant to depth findings: decode is the phase that cannot absorb coarse 4-bit. Note the paper's own explanation is that prefilling exhibits substantial quantization redundancy; attributing the asymmetry to autoregressive error compounding in decode is our reading, not the paper's claim.

arXiv:2605.20315 adversarially re-checked: corrected here

Mechanism

MLA compresses K/V into a 512-dim per-token latent (kv_lora_rank=512, q_lora_rank=1536), so the cached tensor is already a learned low-rank code. config.json ships quantization_config = {quant_method fp8, fmt e4m3, weight_block_size [128,128], activation_scheme 'dynamic'}; V3.1/V3.2 add scale_fmt 'ue8m0' (power-of-two E8M0 scales, the microscaling convention). V3.2 adds a 64-head/128-dim lightning indexer selecting top-2048 tokens.

Key numbers

V3 (arXiv 2412.19437): 671B total / 37B active, 61 layers, 256 routed experts top-8 + 1 shared, kv_lora_rank 512, 163,840 ctx. V3.2 (arXiv 2512.02556) indexer: 64 heads, head_dim 128, index_topk 2048.

Modern architectures

MLA's latent is a pre-compressed state, so generic per-token KV quantization has little redundancy left to exploit and the RoPE-carrying dims need protection. The base checkpoint is already FP8 with power-of-two (UE8M0) scales, so any further activation or wire quantizer is a second quantization of an FP8 tensor; E8M0-scaled mxfp8 composes cleanly with it, per-tensor int8 does not. Top-8 routing makes activation statistics expert-conditional.

Ecosystem

Weights on HF, served by SGLang/vLLM. The DSA lightning indexer is the one component the V3.2 paper explicitly designates FP8. UE8M0 was publicly framed (Aug 2025) as targeting next-generation domestic Chinese accelerators.

arXiv:2412.19437 · Source adversarially re-checked: corrected here

Mechanism

Model card states: 'FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.' config.json sets expert_dtype='fp4' plus quantization_config {fp8, e4m3, weight_block_size [128,128], scale_fmt 'ue8m0'}. V4-Pro safetensors census: 1.573T params packed as I8 (FP4 containers), 23.2B F8_E4M3, 2.8B BF16, 87.8M F32 scales. Attention uses head_dim 512 with 1 KV head; per-layer compress_ratios alternate 128 and 4.

Key numbers

V4-Pro 1.6T total / 49B active, 61 layers, 384 experts top-6 + 1 shared; V4-Flash 284B / 13B, 43 layers, 256 experts. Both 1M ctx, 32T pretrain tokens. At 1M context V4-Pro needs 27% of V3.2's single-token FLOPs and 10% of its KV cache.

Modern architectures

First open frontier model whose default weight format is 4-bit, so activation-side compression becomes the marginal cost rather than the headline. CSA/HCA already cut the KV read to ~10% of V3.2, leaving KV quantization less to win; the surviving traffic is expert dispatch/combine plus the FP8 non-expert path, i.e. exactly the wire tensors. Any further activation quantizer stacks on a 4-bit weight path.

Ecosystem

Preview release, MIT license, weights on HF and ModelScope. Report: arXiv 2606.19348, 'DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence' (Apr 2026), which carries the 1.6T/49B, 284B/13B, 32T-token and 27%-FLOPs/10%-KV figures. Base variants (V4-Pro-Base, V4-Flash-Base) ship FP8-only for comparison.

arXiv:2606.19348 · Source adversarially re-checked: corrected here

Mechanism

Model card: 'Quantization-Aware Training (QAT) during the post-training phase, applying INT4 weight-only quantization to the MoE components.' All published benchmark numbers are measured at INT4. The quantized tensors are the expert GEMM weights; the MLA attention path and the rest of the model are untouched.

Key numbers

1T total / 32B activated, 384 experts with 8 selected per token, MLA attention. ~2x generation speedup at INT4; ~594 GB footprint reported for the INT4 checkpoint.

Modern architectures

A useful control: a trillion-scale MoE absorbs 4-bit weights via QAT with activations left alone, and it is Moonshot's own next model (K3) that moves activations to MXFP8. The card names long thinking traces as the failure mode for low-bit MoE - the same length/depth error-accumulation regime in which per-token int4 activations break down.

Ecosystem

Weights on HF. The base K2 report (arXiv 2507.20534) also specifies activation compression in storage: inputs of MoE up-projections and SwiGLU are compressed to FP8-E4M3 in 1x128 tiles with FP32 scales, without applying FP8 in computation. The INT4 QAT detail appears on the K2-Thinking model card; LMSYS's Jan 2026 INT4 RL rollout work builds on the same W4A16 scheme.

arXiv:2507.20534 · Source adversarially re-checked: corrected here

Mechanism

Official README: 'Kimi K3 applies quantization-aware training from the SFT stage onward, using MXFP4 weights with MXFP8 activations for broad hardware compatibility.' config.json quantization_config: compressed-tensors 'mxfp4-pack-quantized', num_bits 4, group_size 32, scale_dtype torch.uint8 (E8M0). Ignore list: self_attn, shared_experts, dense mlp proj, lm_head, vision_tower, mm_projector - only routed-expert linears are 4-bit.

Key numbers

2.8T total / 104B active, 93 layers = 69 KDA + 24 Gated MLA, 896 experts top-16 + 2 shared, kv_lora_rank 512, 160K vocab, 1,048,576 ctx. MXFP4 group size 32; ~1.56 TB on disk vs ~5.6 TB BF16.

Modern architectures

Direct corroboration of the mxfp8-activation result: a frontier lab chose MXFP8 (block-32, E8M0 scales) for activations rather than per-token int8, and reached 4 bits only on weights and only via QAT. With 69 of 93 layers KDA, most intermediates feed a persistent recurrent state; those modules plus attention and shared experts are excluded from MXFP4. PTQ tuned for BF16 checkpoints does not transfer here.

Ecosystem

Kimi K3 does exist (verified via official GitHub README, HF model card, and config.json). Recommended engines: vLLM, SGLang, TokenSpeed. Weights under the Kimi K3 License; native multimodal (MoonViT-V2, 401M) and 1M context.

Code · Source adversarially re-checked: confirmed

Mechanism

Official FP8 configs: quant_method 'fp8', weight_block_size [128,128], activation_scheme 'dynamic'. The 122B FP8 repo excludes, per GDN layer, linear_attn.conv1d / in_proj_a / in_proj_b, plus every mlp.gate router, shared_expert_gate, lm_head, embed_tokens, the vision tower and the MTP head. The Qwen3.6-35B FP8 config (fmt 'e4m3') goes further, additionally excluding in_proj_ba, A_log, dt_bias, linear_attn.norm, q_norm/k_norm and layernorms. GDN out_proj and full-attention projections ARE quantized in both.

Key numbers

122B/10B: 48 layers = 36 linear_attention + 12 full_attention (interval 4), 256 experts 8 routed + 1 shared, 262,144 ctx extensible to ~1.01M. 35B/3B: 40 layers = 30 GDN + 10 gated attention, same MoE, expert dim 512, GDN 32 V heads / 16 QK heads, head dim 128, conv kernel 4.

Modern architectures

These exclusion lists are concrete evidence that a linear-attention state is not an ordinary activation: the GDN input projections, short conv, decay parameters (A_log, dt_bias) and state norm stay BF16 while neighbouring projections go FP8. Routers are always excluded because a small perturbation flips top-k and changes the computation path. Only the parts of the activation-quantization toolkit acting on transient projection outputs transfer.

Ecosystem

Base Qwen/Qwen3.5-122B-A10B and Qwen/Qwen3.6-35B-A3B repos are BF16; FP8 ships as separate official repos described as 'fine-grained fp8 quantization with block size of 128'. Community INT4/NVFP4/MXFP4 and int4+fp8 hybrid variants exist. Runs on transformers, vLLM, SGLang, KTransformers.

Source adversarially re-checked: corrected here

Mechanism

hf_quant_config.json for DeepSeek-V4-Pro-NVFP4 is quant_algo MIXED_PRECISION with per-layer entries 'layers.N.ffn.experts' → NVFP4, group_size 16; everything else remains FP8. Card: 'Only the weights and activations of the linear operators within transformer blocks in MoE are quantized.' Calibrated with cnn_dailymail plus Nemotron-Post-Training-Dataset-v2 using nvidia-modelopt v0.44.

Key numbers

DeepSeek-V4-Pro FP8→NVFP4: GPQA-D 89.49→89.33, AA-LCR 66.89→66.33, tau2-Bench Telecom 94.25→94.83, SciCode 51.08→53.45, IFBench 77.82→77.21. Qwen3.6-35B-A3B BF16→NVFP4: MMLU-Pro 85.6→85.0, GPQA-D 84.9→84.8, AIME25 89.2→88.8.

Modern architectures

Shows what vendors currently believe transfers to modern MoE: NVFP4's finer 16-element block scale is applied to expert activations and weights, but nobody quantizes the router, the MLA/attention projections, or GDN state modules. Group size 16 with an E4M3 scale (NVFP4) versus 32 with an E8M0 scale (MXFP4) is precisely the axis leaned on for 4-bit activation robustness.

Ecosystem

Verified serving on B200/GB300 with vLLM (--kv-cache-dtype fp8, TP4) and SGLang (TP8); SGLang integration merged 2026-06 (PR 25820), auto-detected from the checkpoint config.json quantization_config ('moe_quant_algo': 'NVFP4').

Code · Source adversarially re-checked: corrected here

Mechanism

Three expert-aware components: smoothing aggregation to suppress activation outliers across experts sharing an input, routing consistency alignment so top-k expert selection survives quantization, and calibration data balance so sparsely activated experts receive enough samples to estimate scales.

Key numbers

1.15-13.81% accuracy improvement over prior PTQ across three MoE architectures, with the largest gains on reasoning tasks.

Modern architectures

This is the per-expert calibration problem stated precisely: in an MoE the activation statistics a quantizer observes are conditioned on the router, so a single global scale is wrong and a calibration set that never activates expert i gives it no scale at all. The routing-consistency term has no dense-model analogue - it is the only part of the activation-quantization toolkit that is MoE-only.

Ecosystem

Code released. Applies to the 256-896-expert regime of DeepSeek-V3/V4, Kimi K2/K3 and Qwen3.5/3.6, though the published evaluation uses smaller MoE architectures than those.

arXiv:2506.13329 · Code adversarially re-checked: corrected here

Mechanism

Formulates mixed-precision assignment over experts and linear blocks from two signals: differing per-block quantization sensitivity, and heterogeneous expert activation frequency. Emits fused GroupGEMM kernels that execute GEMMs of different precisions in parallel, so mixed precision is not paid back in kernel-launch overhead.

Key numbers

2.25-bit config beats GPTQ by 2.4 Wikitext-2 perplexity; 5-bit weight-activation setting reaches 3.4x speedup over full precision and 29.4% over uniform quantization at equal accuracy.

Modern architectures

Expert activation frequency is the MoE-specific lever: hot experts see many tokens per batch and dominate throughput while cold experts see few and dominate calibration error, so uniform precision is wrong in both directions. Ports cheaply to Qwen3.5/K3-scale expert counts, but the payoff is concentrated on weights rather than on the tensor crossing a stage or all-to-all boundary.

Ecosystem

Kernel-level co-design; the GroupGEMM approach is the same primitive production MoE engines use, so the mixed-precision assignment is realizable rather than simulation-only.

arXiv:2505.05799 · Code adversarially re-checked: confirmed

Mechanism

KDA is a Diagonal-Plus-Low-Rank linear attention with two dynamic gates - a per-channel forget gate and an update rate - giving a fixed-size recurrent memory; every fourth layer is full MLA to preserve global information flow. Decode carries the state forward instead of growing a KV cache.

Key numbers

48B total / 3B activated; 3:1 KDA:MLA layer ratio; up to 75% less KV cache and up to 6x decoding throughput at 1M context.

Modern architectures

The recurrent state is written once per step and read on every subsequent step, so a quantization error in it compounds across the whole generation rather than being consumed immediately like a pipeline-boundary activation. That is the mechanistic reason hybrid checkpoints keep state modules in BF16, and why per-token quantizers designed for transient tensors do not transfer to the state.

Ecosystem

Open kernels and 48B-A3B checkpoints released; the same KDA block scales up to Kimi K3 (69 of 93 layers). Qwen3-Next/Qwen3.5/3.6 use Gated DeltaNet at the same 3:1 ratio, so the state-persistence argument covers both families.

arXiv:2510.26692 · Code adversarially re-checked: confirmed

Mechanism

A lightweight lightning indexer scores past tokens and a fine-grained selector keeps top-k, reducing attention cost from O(L^2) to O(Lk). The paper: 'Given that the lightning indexer has a small number of heads and can be implemented in FP8, its computational efficiency is remarkable.'

Key numbers

Released config: index_n_heads 64, index_head_dim 128, index_topk 2048 (V3.2); the same indexer appears in V4-Pro with topk 1024 and V4-Flash with topk 512.

Modern architectures

Sparse attention is a different axis from numeric compression but interacts with it twice. The indexer is the one component explicitly designated FP8, and it is a scoring head whose errors change which tokens get read - a routing-like sensitivity closer to an MoE gate than a value projection. Token selection also removes the very redundancy a KV quantizer exploits, so the two do not stack multiplicatively.

Ecosystem

Reference implementation and kernels in deepseek-ai/DeepSeek-V3.2-Exp; DSA carries forward into the V4 series alongside CSA/HCA. Note: the paper's only precision statement is the FP8 indexer sentence - training precision is not discussed.

arXiv:2512.02556 · Code adversarially re-checked: confirmed

Mechanism

RoPE-Aware Per-Token KV Quantization splits the MLA latent so the RoPE component stays high precision while the compressed content dimensions go FP8, combined with FP8 P*V computation and a hardware-aware pipelined schedule. Training-free, aimed at long-output decode.

Key numbers

Up to 1.91x throughput on long-output decoding workloads versus BF16, with near-parity quality on reasoning and code-generation benchmarks.

Modern architectures

The concrete answer to how you quantize an MLA latent: you cannot treat it as one homogeneous vector, because the RoPE-carrying dimensions have different dynamics from the compressed content dimensions. That heterogeneity-inside-one-tensor problem is MLA's analogue of the outlier-channel problem, and it explains why a single per-token scale underperforms a split scheme.

Ecosystem

Code released (meituan-longcat/SGLang-FluentLLM; authors are Meituan LongCat). Directly relevant to DeepSeek-V3/V3.2/V4 and Kimi K2/K3, all of which carry MLA (K3 as Gated MLA on 24 of 93 layers).

arXiv:2602.10718 · Code adversarially re-checked: confirmed

Mechanism

A MatMul-only algorithm for strictly lower-triangular inversion using truncated Neumann expansion with structural masking and parallel residual correction, extended to low-bit INT by mitigating dynamic range expansion. Evaluated on Qwen3.5-family models and targeted at NPUs without a fast inverse.

Key numbers

Up to 5x kernel-level speedup and roughly 20% reduction in decode-layer overhead while preserving accuracy, on Qwen3.5-family models.

Modern architectures

The clearest published account of why hybrid models resist activation quantization: the delta-rule chunk update needs a triangular matrix inverse, the repeated matrix-power terms of its Neumann approximation expand dynamic range, and the resulting intermediates are far harder to represent in low-bit INT than an ordinary projection output. The fix is to change the operator, not the scale, demonstrated on the exact Qwen3.5 GDN blocks that official FP8 checkpoints refuse to quantize.

Ecosystem

The only paper found that quantizes a Gated DeltaNet path specifically rather than Mamba/SSM generally. Complements the empirical evidence from Qwen's own FP8 exclusion lists.

arXiv:2606.06034 adversarially re-checked: corrected here

Mechanism

Head-to-toe quantization of Mamba1/Mamba2: inputs of the linear recurrence are quantized to 8-bit offline by sorting and clustering channels, weights are rearranged to match the clustering so SSM output stays compute-invariant, and B and C get per-state-group quantization.

Key numbers

1.3x prefill and 3x generation speedup, 4x memory reduction, 1.6% average accuracy drop. Predecessor Quamba reports naive 8-bit causing model collapse at all sizes, while self-attention layers tolerate the same error.

Modern architectures

The baseline result the hybrid era inherited: naive 8-bit on a recurrent state collapses the model while attention layers survive it - the same asymmetry the Qwen FP8 exclusion lists encode in production. Transfer to GDN/KDA is only partial, since those add data-dependent per-channel decay and delta-rule updates that Mamba's channel-order-preserving trick does not cover.

Ecosystem

Code under enyac-group/Quamba. Follow-ons include Quamba-SE (multi-scale activation quantizer for SSMs), MambaQuant (rotation methods, arXiv 2501.13484) and Ternary Mamba (W1.58A16 QAT).

arXiv:2503.22879 · Code adversarially re-checked: corrected here

Mechanism

Quantized linear/MoE kernels are selected by a Config → Method → Scheme → Kernel dispatch. Online paths quantize weights at load and activations per-forward (per-token or per-block); offline paths load pre-quantized checkpoints. GEMM backend is chosen with --fp8-gemm-backend (deep_gemm, flashinfer_trtllm/cutlass/deepgemm, cutlass, triton, aiter) and --fp4-gemm-backend (flashinfer_cutlass/cudnn/cutedsl/trtllm, marlin).

Key numbers

Docs table lists 23 methods; server_args.py accepts 32 values (extensible), incl. fp8, w8a8_fp8, w8a8_int8, mxfp8, mxfp4, mxfp_w4a8, nvfp4_online, modelopt/_fp8/_fp4/_mixed, awq(_marlin), gptq(_marlin), compressed-tensors, quark variants, auto-round, petit_nvfp4, w4afp8, moe_wna16, gguf, bitsandbytes, modelslim, mlx_q4/q8. Gotcha: blockwise_int8 is documented and registered but not in the argparse choices list, so the flag is rejected.

Modern architectures

Strong MoE emphasis: separate MoE schemes for compressed-tensors, ModelSlim and Quark, plus a MoeRunner migration for W4A4 NVFP4 and the SM100 TRT-LLM MXFP4 MoE path.

Ecosystem

Mature for fp8/int8 W8A8 and awq/gptq. nvfp4_online is MoE-only and needs SM100+; modelopt_fp4 falls back to Marlin W4A16 on Ampere/Hopper; Marlin kernels are CUDA-only. Roadmap #31783 is mid-refactor toward Config→Method→Scheme→Kernel and intends to split --quantization (checkpoints) from --online-quantization (runtime). Nothing here touches parallelism-wire precision.

Source adversarially re-checked: corrected here

Mechanism

The KV pool is allocated in the low-precision dtype; K/V are quantized on write and either dequantized or consumed natively by the attention backend. mxfp8 is gated to the FA4 backend; nvfp4 is block-16 E2M1 with FP8 scales, fp4_mx_block16 is the MX-style block-16 E2M1 recipe. ServerArgs post-processing validates each recipe against the selected attention backend and SM level.

Key numbers

Nine accepted values in server_args.py: auto, fp8_e5m2, fp8_e4m3, mxfp8, bf16, bfloat16, nvfp4, fp4_mx_block16, fp4_e2m1. A separate --speculative-draft-kv-cache-dtype exists for draft models.

Ecosystem

The rendered docs table lists only 7 of the 9 values (mxfp8 and fp4_e2m1 appear in server_args.py but not the published table), so source is the better reference. FP4 KV is still tracked as an open roadmap thread (#29913) and backend coverage is uneven per recipe. Note the contrast: SGLang will store KV at 4 bits but sends PP activations at full width.

Source adversarially re-checked: confirmed

Mechanism

A quantization-config registry picks a per-layer scheme and kernel at load time. Online quantization quantizes weights at load and activations per-forward at the declared granularity; pre-quantized checkpoints are parsed from config.json. Mixed-input W4A16/W8A16 GEMMs dispatch to Marlin or, on Hopper, to the CUTLASS-based Machete kernel.

Key numbers

Online flags: fp8_per_tensor, fp8_per_block (fp32 per-128x128 weight, per-1x128 activation scale), mxfp8, mxfp4; --quantization-config for per-layer control. FP8 needs CC>=8.9 (Ada/Hopper); INT8 needs CC>7.5 and is unsupported on Blackwell (CC>=10.0). mxfp8 W8A8 needs SM100+, else W8A16 fallback.

Ecosystem

The de-facto reference implementation with the broadest hardware matrix (Volta through Blackwell, AMD, Intel, CPU). Marlin (marlin_utils.py, marlin_utils_fp4.py, marlin_utils_fp8.py) and Machete (machete_utils.py) both live under vllm/model_executor/layers/quantization/utils; Machete is the Hopper/TMA successor to Marlin, and Marlin doubles as the W4A16 fallback when no native FP4 GEMM exists.

Source adversarially re-checked: confirmed

Mechanism

K and V are written to the paged cache in FP8 and consumed by the attention kernel. Two granularities: one scale per Q/K/V tensor, or per-head scales when the checkpoint carries them from llm-compressor and the Flash Attention backend is active. --kv-cache-dtype-skip-layers leaves sensitive layers (e.g. sliding_window) in full precision.

Key numbers

Docs values: auto, fp8, fp8_e4m3, fp8_e5m2 (fp8_e4m3 on CUDA 11.8+ and ROCm). Source (vllm/config/cache.py CacheDType) additionally ships nvfp4, nvfp4_4over6, int4/int8/fp8_per_token_head, fp8_ds_mla, fp8_inc and four turboquant 3/4-bit recipes, backed by dedicated CUDA kernels and the FlashInfer backend. kv_cache_dtype='fp8' alone pins all scales to 1.0 (the common footgun); --kv-cache-dtype-skip-layers exempts sensitive layers.

Ecosystem

Broader than the docs suggest, and at least as broad as SGLang on KV dtypes once the source is consulted. Calibrated scales come from llm-compressor checkpoints.

Source adversarially re-checked: corrected here

Mechanism

The loader reads the ModelOpt quant config, selects an FP4/FP8 scheme, then picks a GEMM backend at load time from CUTLASS, FlashInfer or Marlin based on the detected SM level. FP8 variants cover per-tensor ("FP8"), per-channel weight with dynamic per-token activation ("FP8_PER_CHANNEL_PER_TOKEN"), and block-scaled weight-only ("FP8_PB_WO").

Key numbers

Flags: --quantization modelopt, modelopt_fp4, modelopt_mxfp8. MXFP8 on SM100 with BF16 activations wants --linear-backend flashinfer_trtllm. On GPUs without a supported native FP4 GEMM, vLLM warns and falls back to W4A16 via Marlin.

Ecosystem

The main production route to true 4-bit activation compute in vLLM, and Blackwell-gated in practice: the W4A16 Marlin fallback keeps activations at 16 bits on Ampere/Hopper, so the FP4 activation path really only exists on SM100+. Useful confirmation that microscaled FP4 (NVFP4 block-16 with FP8 scales) is the format the vendor stack converged on.

Source adversarially re-checked: confirmed

Mechanism

llm-compressor applies modifiers (GPTQModifier, SmoothQuantModifier with smoothing_strength, AWQ, AutoRound, rotation-based SpinQuant/QuIP, REAP expert pruning) to a transformers model, then serializes to compressed-tensors. compressed-tensors encodes scheme, strategy (per-tensor, per-channel, per-token, per-group/block), format and ignore list in config.json so vLLM rebuilds the right kernel at load.

Key numbers

Recipe scheme strings include "FP8_DYNAMIC" (static per-channel weights, dynamic per-token activations) and "W8A8" (int8 W8A8-dynamic-per-token, typically SmoothQuant at smoothing_strength=0.8 plus GPTQ), both with targets="Linear" and ignore=["lm_head"]. compressed-tensors was at 0.18.1.dev0, Apache-2.0.

Modern architectures

Dedicated MoE schemes exist in vLLM for compressed-tensors W4A4 NVFP4/MXFP4, W4A8 fp8/int8, W8A8 fp8/int8/mxfp8 and wna16, including RDNA3 variants.

Ecosystem

Mature and the most widely adopted checkpoint format for quantized LLMs; it also carries rotation methods (SpinQuant, QuIP) into a serving-ready artifact. Canonical repo is now vllm-project/compressed-tensors (neuralmagic/compressed-tensors redirects). Everything it expresses is per-layer weight/activation/KV precision; the format has no concept of a parallelism-wire dtype.

Source adversarially re-checked: confirmed

Mechanism

Recipes are attached to the engine/LLM config and compiled into architecture-specific kernels. Per-arch support: Blackwell SM100/103 gets NVFP4 (incl. NVFP4 KV cache) and FP8 block scaling; SM120 gets NVFP4, MXFP4, FP8 per-tensor and FP8 KV; Hopper gets all FP8 variants (rowwise is Hopper-only) plus W4Ax; Ada gets FP8 per-tensor, FP8 KV cache and all four W4A8/W4A16 AWQ/GPTQ recipes; Ampere is limited to W4A16 AWQ/GPTQ plus FP8 KV cache.

Key numbers

Eleven listed recipes: FP4, MXFP4, FP8 Per Tensor, FP8 Block Scaling, FP8 Rowwise, FP8 KV Cache, NVFP4 KV Cache, W4A16 GPTQ, W4A8 GPTQ, W4A16 AWQ, W4A8 AWQ. KV cache set via KvCacheConfig(dtype='fp8') or 'nvfp4'.

Ecosystem

The most hardware-tuned of the three but the least flexible: recipes are largely pre-baked per architecture and the fastest paths assume Blackwell. Along with NVFP4 KV cache it is the clearest vendor endorsement of microscaled FP4 for inference. Verified absence of any communication/wire precision discussion on the quantization page, and an issue search for PP communication quantization returns nothing.

Source adversarially re-checked: corrected here

Mechanism

5th-gen tensor cores add FP4/FP6 to Hopper's FP8 Transformer Engine. Block-scaled MMA is the warp-level mma.sync path (.kind::mxf4nvf4 / .mxf8f6f4), not datacenter Blackwell's tcgen05 + Tensor Memory, so SM100 kernels are both binary- and algorithm-incompatible. GeForce also halves FP32-accumulate tensor rates: FP8 is 838 TFLOPS with FP16 accumulate but only 419 with FP32.

Key numbers

FP4 1676/3352 TFLOPS (dense/sparse); FP8 838/1676 (FP16 acc), 419/838 (FP32 acc); INT8 838/1676 TOPS; FP16 419/838 (FP16 acc). 21,760 CUDA cores, 680 TCs, 2.41 GHz. 32 GB GDDR7, 512-bit, 28 Gbps, 1,792 GB/s. PCIe Gen 5. NVLink: No. 575 W. NVIDIA headline: 3,352 AI TOPS.

Ecosystem

Cheapest silicon where NVFP4/MXFP4 GEMM and an fp8/mxfp8 wire codec are hardware-native rather than software-emulated. But no NVLink, and NVIDIA states P2P is unsupported on GeForce, so PP/TP activation traffic rides PCIe and can stage through host memory. This is exactly the regime where compressing stage-boundary activations buys wall-clock instead of nothing.

Source adversarially re-checked: confirmed

Mechanism

Identical narrow-precision set to GeForce Blackwell (FP4/FP6/FP8/INT8, block-scaled NVFP4/MXFP4 via mma.sync, no tcgen05 or Tensor Memory). The 600W Workstation Edition reaches FP4 2015.2/4030.4 TFLOPS (dense/sparse) and FP8/INT8 1007.6/2015.2; the 300W Max-Q Edition runs FP4 1755.7/3511.4 and FP8/INT8 877.9/1755.7. The 96 GB frame buffer is the practical reason to prefer it over a 5090 when sizing PP stages.

Key numbers

96 GB GDDR7 with ECC, 1,792 GB/s (Server Edition SKU: 1,597 GB/s), PCIe Gen 5, 4,000 AI TOPS headline (Workstation Edition), 5th-gen tensor cores.

Ecosystem

Unlike GeForce, the pro line does expose PCIe P2P; dual-card NCCL P2P hangs were reported (NVIDIA/nccl 1999, since closed) with NCCL_P2P_DISABLE=1 as the workaround. Either way the ceiling is PCIe Gen 5 at ~64 GB/s per direction, so activation wire compression still pays on these boxes.

Source adversarially re-checked: corrected here

Mechanism

sm_89 4th-gen tensor cores accelerate FP8 e4m3/e5m2 through warp-level mma.sync (mapping to the QMMA instruction), with no wgmma, no TMA, and no block-scaled MMA. INT8/FP16/BF16/TF32 are supported; FP4, FP6 and any microscaled format are absent and must be emulated in software. FP32-accumulate tensor rates are halved versus FP16 accumulate, as on Blackwell GeForce.

Key numbers

FP8 660.6/1321.2 TFLOPS (FP16 acc), 330.3/660.6 (FP32 acc); INT8 660.6/1321.2 TOPS; FP16 330.3/660.6 (FP16 acc), 165.2/330.4 (FP32 acc). FP4: N/A. 24 GB GDDR6X, 384-bit, 21 Gbps, 1,008 GB/s. PCIe Gen 4. NVLink: No. 450 W. NVIDIA headline: 1,321 AI TOPS.

Ecosystem

So int8 and fp8 wire codecs are free on the compute side, but per-token int4 and microscaled mxfp4/nvfp4 are pure software on Ada: eager pack/unpack with zero tensor-core payoff. Worst interconnect in this set - PCIe Gen 4 at ~32 GB/s per direction, no NVLink, and an NVIDIA moderator relayed that Engineering says P2P is not supported on the 4090.

Source adversarially re-checked: corrected here

Mechanism

4th-gen tensor cores reach FP8/INT8 via wgmma (warpgroup async MMA) plus TMA, at 2x the FP16 rate, with full-rate FP32 accumulate unlike GeForce. There is no block-scaling hardware, so MXFP8/MXFP4/NVFP4 buffers must be dequantized in software before the GEMM. Sub-byte wmma integer paths (experimental u4/s4, and b1 with XOR) are deprecated at sm_90, so int4 activations get no supported tensor-core path either.

Key numbers

SXM: FP8 and INT8 3,958 TFLOPS/TOPS with sparsity (~1,979 dense), FP16/BF16 1,979 with sparsity (~990 dense), TF32 989 with sparsity (~495 dense), FP64 TC 67. 80 GB HBM3, 3.35 TB/s, NVLink 900 GB/s, PCIe Gen 5 128 GB/s. H100 NVL: 94 GB HBM3, 3.9 TB/s, NVLink 600 GB/s, FP8 3,341.

Ecosystem

The FP8-native, FP4-absent generation: fp8 weight+activation serving is free, everything microscaled is emulated exactly the way our eager wire codecs do it. NVLink 4 at 900 GB/s against PCIe Gen 5's 128 GB/s is a ~7x gap, so inside an NVLink island a bf16 PP hop is already cheap; across PCIe-attached NVL pairs or across nodes, wire compression matters again.

Source adversarially re-checked: corrected here

Mechanism

Same sm_90 4th-gen tensor cores and Transformer Engine as H100: FP8 and INT8 at 3,958 TFLOPS/TOPS with sparsity, FP16/BF16 1,979 and TF32 989 with sparsity (dense is half). No block-scaling hardware and no FP4/FP6. The delta is 141 GB HBM3e and 4.8 TB/s, which lifts KV-cache headroom and decode bandwidth without adding any new quantization format.

Key numbers

141 GB HBM3e, 4.8 TB/s, NVLink 900 GB/s, SXM up to 700 W; H200 NVL air-cooled PCIe dual-slot up to 600 W with FP8/INT8 3,341, FP16 1,671 (sparse), TF32 835 (sparse), FP64 TC 60. NVIDIA marks the table preliminary.

Ecosystem

For activation compression the important fact is negative: no native microscaling anywhere in Hopper, so mxfp8/nvfp4 wire codecs on H200 are software pack/unpack just as on H100. The 141 GB per GPU also reduces pressure to shard deep, which cuts the number of PP stage boundaries there are to compress in the first place.

Source adversarially re-checked: corrected here

Mechanism

5th-gen tensor cores are driven by tcgen05.mma out of dedicated Tensor Memory. CUTLASS lists inputs e4m3/e5m2 (FP8), e2m3/e3m2 (FP6), e2m1 (FP4), plus block-scaled aggregates mx_float8_t/mx_float6_t/mx_float4_t/nv_float4_t with ue8m0 or ue4m3 scales. NVFP4 = 16-element blocks with an e4m3 scale and a per-tensor FP32 scale; MXFP4 = 32-element blocks with a power-of-two ue8m0 scale.

Key numbers

Per GPU: NVFP4 10|20 PFLOPS dense|sparse, FP8 5|10 PFLOPS, max 192 GB HBM3E, 8 TB/s, NVLink 1.8 TB/s (18x100 GB/s). DGX B200: 1,440 GB total (180 GB/GPU), 64 TB/s, 14.4 TB/s aggregate NVLink. HGX B200: FP4 144|72 PFLOPS, 1.4 TB total.

Modern architectures

Second-gen Transformer Engine applies micro-tensor scaling automatically, which is what lets MoE and long-context serving stacks run NVFP4 end to end without hand-written scale plumbing.

Ecosystem

Where a microscaled wire codec becomes genuinely free: the same nvfp4 or mxfp8 buffer you put on the wire is also a legal tensor-core operand, so quantize-once-use-twice works instead of dequantizing at every stage. Datatype list confirmed at docs.nvidia.com/cutlass/latest/media/docs/cpp/blackwell_functionality.html. Caveat: NVLink 5 makes intra-node PP hops so cheap that compression saves little inside the box.

Source adversarially re-checked: confirmed

Mechanism

Same 5th-gen tensor core datatypes as B200 (FP8/FP6/NVFP4, block-scaled MX) but a distinct compile target - compute capability 10.3 / sm_103 per CUDA arch tables - so cubins differ from sm_100. Dense NVFP4 rises 10 to 15 PFLOPS while FP8 is unchanged at 5|10 PFLOPS, making the uplift specifically 4-bit. SFU EX2 throughput goes 5 to 10.7 TeraExponentials/s, roughly doubling softmax/attention headroom.

Key numbers

288 GB HBM3E via 12-high stacks (vs B200's max 192 GB), 8 TB/s, NVLink 1.8 TB/s, ~1,400 W TDP. NVFP4 15|20 PFLOPS dense|sparse vs B200's 10|20; FP8 5|10 unchanged. SFU 10.7 vs 5 TeraExponentials/s. HGX B300: FP4 144|108 PFLOPS, 2.1 TB total memory.

Modern architectures

The 2x SFU exponential rate is aimed at attention and long reasoning traces; it shifts the bottleneck for long-context serving away from softmax and back onto memory and interconnect.

Ecosystem

The FP8-flat / FP4-up shape is a hardware bet that 4-bit is the serving format - the same regime as our per-token int4 failures and mxfp4/nvfp4 rescue. On this chip the microscaled 4-bit path is the fast path and plain per-token int4 is not a tensor-core citizen at all. 288 GB also means fewer PP stages per model, hence fewer boundaries to compress.

Source adversarially re-checked: confirmed

Mechanism

NVLink 5 switches make the whole rack one 72-GPU fabric at 1.8 TB/s per GPU, so tensor, expert and pipeline parallelism no longer have to respect an 8-GPU node boundary. NVFP4 is the headline inference datatype: GB200 NVL72 reports FP4 1,440|720 PFLOPS sparse|dense, GB300 NVL72 1,440|1,080 - matching 72x10 and 72x15 PFLOPS dense per GPU.

Key numbers

72 GPUs + 36 Grace CPUs. NVLink 5: 130 TB/s aggregate, 1.8 TB/s per GPU. GB200 NVL72: 13.4 TB HBM3E, FP4 1,440|720 PFLOPS, FP8 720 PFLOPS. GB300 NVL72: 20 TB HBM3E, up to 576 TB/s memory bandwidth, FP4 1,440|1,080 PFLOPS, FP8/FP6 720 PFLOPS.

Ecosystem

The rack is the counterexample to the wire-compression thesis: at 1.8 TB/s per GPU, bf16 activations on a stage boundary cost microseconds and squeezing them is noise. That asymmetry is the point - the same PP topology that is bandwidth-bound on a PCIe consumer box is compute-bound here, so wire-quantization results are only interpretable with the interconnect named.

Source adversarially re-checked: confirmed

Mechanism

NVIDIA's own figures: NVLink 4 (Hopper) 900 GB/s per GPU over 18 links, NVLink 5 (Blackwell) 1,800 GB/s over 18x100 GB/s, NVLink 6 (Rubin) 3,600 GB/s over 36 links; NVSwitch NVL72 aggregate 130 TB/s. PCIe Gen 5 x16 is 128 GB/s bidirectional per NVIDIA's H100 page. NVIDIA states NVLink 6 delivers over 14x the bandwidth of PCIe Gen6.

Key numbers

NVLink 4: 900 GB/s. NVLink 5: 1,800 GB/s. NVLink 6: 3,600 GB/s. NVSwitch NVL72: 130 TB/s. PCIe Gen 5 x16: 128 GB/s bidir. PCIe Gen 4 x16: ~64 GB/s bidir. Ratios: NVLink5:PCIe5 ~14x, NVLink5:PCIe4 ~28x, NVLink4:PCIe5 ~7x.

Ecosystem

Implied rule: on NVLink islands the PP boundary is not the bottleneck and int8/fp8 wire codecs buy little; on PCIe-only boxes (5090, 4090, RTX PRO 6000) the same codec removes a first-order cost and 4-bit vs 8-bit on the wire is a real 2x. Consumer boards also often run slots at x8/x4, and GeForce P2P is disabled, so measured PCIe throughput usually sits well below the x16 spec.

Source adversarially re-checked: corrected here

Mechanism

Recipe-driven sequential pipeline over a HF model: forward hooks capture per-layer activation statistics on a calibration set, fit per-channel weight scales and per-token or per-block activation scales (static or dynamic), optionally rotate to kill outliers, then serialize quantization config plus packed payloads into safetensors.

Key numbers

3,714 stars; v0.13.0 released 2026-08-11; last push 2026-08-23; Apache-2.0; 3,190 commits

Modern architectures

Actively tracks MoE and hybrid models (Qwen3 MoE, DeepSeek-family, Nemotron hybrids); expert-wise and attention/KV-cache quantization schemes are first-class rather than bolt-ons.

Ecosystem

Output format is compressed-tensors, loaded natively by vLLM and SGLang and by HF Transformers. Calibration-free paths exist (RTN, dynamic-activation FP8), so not every scheme needs data. Docs: https://docs.vllm.ai/projects/llm-compressor. Red Hat AI publishes pre-quantized checkpoints from it claiming >99% baseline accuracy recovery.

Code · Source adversarially re-checked: confirmed

Mechanism

Per-tensor QuantizationArgs record num_bits, dtype, granularity/strategy (per-tensor, per-channel, per-group, per-token, per-block), symmetry, and static-vs-dynamic. Weights ship packed (int4/int8/fp4) alongside scale and zero-point tensors; separate arg blocks describe input-activation, output-activation, KV-cache quantization, and Hadamard transform metadata.

Key numbers

315 stars; v0.18.0 released 2026-08-08; last push 2026-08-22; 575 commits; 42 open PRs

Ecosystem

Because it carries activation-quant args and rotation metadata in the checkpoint itself, a wire/activation scheme described here is portable across engines. Recent commits show a QuantizedKVCache wrapper and an fp4 rounding Triton kernel, i.e. activation/KV compression is where the format is currently growing.

Code · Source adversarially re-checked: confirmed

Mechanism

Wraps torch modules in quantizer modules, runs a forward calibration pass to collect amax/histogram statistics, then fits per-tensor, per-channel, or per-block scales. Supports SmoothQuant migration, AWQ weight scaling, double-level NVFP4 scaling, and quantization-aware training/distillation when PTQ loses too much.

Key numbers

3,470 stars; v0.46.0 released 2026-08-18; last push 2026-08-24; repo dates to 2024-04, first full Apache-2.0 OSS release (0.23.0) published 2025-01 with NVFP4 support; 1,160+ commits. README: compress model size by 2x-4x via PTQ.

Modern architectures

Ships recipes for MoE (per-expert scaling) and for MLA-style attention; also covers speculative decoding and pruning in the same toolkit, so a single config can stack quantization with other inference-time compression.

Ecosystem

Practitioner gotchas: the repo was renamed to NVIDIA/Model-Optimizer (old TensorRT-Model-Optimizer URL redirects) while the pip package stays nvidia-modelopt; the PTQ example lives at examples/hf_ptq, and --kv_cache_qformat defaults to fp8_cast. Checkpoints are consumed by TensorRT-LLM, TensorRT, vLLM, SGLang (modelopt_fp8 / modelopt_fp4) and Diffusers. This is the reference path to NVFP4 activations plus NVFP4 KV cache on Blackwell. The toolkit has no notion of quantizing tensors in flight between GPUs, so PP-wire precision is out of scope by construction.

Code · Source adversarially re-checked: confirmed

Mechanism

quantize_(model, Config()) swaps linear weights for AffineQuantizedTensor subclasses whose torch-function dispatch calls a quantized kernel. Dynamic-activation configs quantize the incoming activation per token (or per microscale block) inside the op, so activation scales never need to be calibrated or stored. Static variants are the exception and do need calibration.

Key numbers

2,955 stars; v0.18.0 released 2026-08; last push 2026-08-24. Inference: int4 weight-only 1.73x speedup / 65% less memory (Gemma3-12b-it on H100). Training-side (not inference): float8 up to 1.5x at 405B/512-GPU pre-training scale, MXFP8 MoE prototype 1.45x on Llama4 Scout.

Modern architectures

MX/NVFP4 work is explicitly aimed at MoE layers and at edge (int8 activations + int4 weights running Qwen3-4B on an iPhone), which brackets the two regimes where activation traffic dominates.

Ecosystem

Config names verified in source: Int8DynamicActivationInt8WeightConfig, Float8DynamicActivationFloat8WeightConfig, Float8DynamicActivationInt4WeightConfig, and under prototype/mx_formats MXDynamicActivationMXWeightConfig and NVFP4DynamicActivationNVFP4WeightConfig (so treat 4-bit microscaled activations as pre-stable). Consumed by HF Transformers (TorchAoConfig), vLLM, SGLang, Diffusers, ExecuTorch, TorchTune/TorchTitan.

Code · Source adversarially re-checked: corrected here

Mechanism

Layer-wise second-order error compensation (GPTQ) or activation-aware channel rescaling (AWQ) fitted on a calibration set, then repacks weights into a kernel-specific layout (Marlin, Machete, ExLlama V2/V3, BitBLAS, Triton, TorchFused) chosen per target device.

Key numbers

1,240 stars; v7.3.4 released 2026-08-19 (README cites 7.4.0, Aug 2026); last push 2026-08-23

Ecosystem

Checkpoints load in HF Transformers, Optimum, PEFT, vLLM and SGLang. Backends span NVIDIA Turing+, AMD ROCm 6.2+, Intel XPU/CPU, Huawei Ascend NPU, and Apple Silicon via MLX conversion. README states it has 'fully supplanted AutoGPTQ and AutoAWQ' for the HF integration; GGUF and FP8 paths are weight-only and skip calibration.

Code · Source adversarially re-checked: confirmed

Mechanism

Activation-aware weight quantization: a calibration pass measures per-channel activation magnitude, salient weight channels are scaled up before INT4 group-wise rounding so their relative error shrinks, and the inverse scale is folded into the preceding op. Activations stay FP16 throughout.

Key numbers

2,349 stars; archived 2025-05-11; final release v0.2.9 on 2025-05-11; no commits since

Ecosystem

README states AutoAWQ 'is officially deprecated and will no longer be maintained' and points users to vllm-project/llm-compressor, plus MLX-LM for Mac. vLLM and SGLang still read its checkpoint format via awq / awq_marlin kernels, so old AutoAWQ artifacts remain servable even though the tool is dead.

Code · Source adversarially re-checked: confirmed

Mechanism

LLM.int8() does vector-wise (per-row/per-token) int8 quantization of both activations and weights, detects outlier feature dimensions at runtime, and routes just those columns through a 16-bit matmul, summing the two partial products. NF4/FP4 4-bit is block-wise weight-only quantization with a normal-distribution-optimal codebook plus double quantization of the scales.

Key numbers

8,433 stars; v0.50.1 released 2026-08-13; last push 2026-08-19; ~913 forks

Ecosystem

Consumed via HF Transformers BitsAndBytesConfig, PEFT/QLoRA, and vLLM's bitsandbytes loader. Multi-backend as of the recent releases: CUDA SM60+ (SM75+ recommended), AMD ROCm/CDNA, Intel Arc and Data Center GPU Max, CPU with AVX2/AVX512, and partial Apple Metal. optimum-quanto's own README recommends it as an alternative.

Code · Source adversarially re-checked: confirmed

Mechanism

quantize() replaces modules with QTensor-backed equivalents holding float weights and dynamic activation quantizers. An optional Calibration context manager runs representative samples and records activation ranges with momentum-tracked min/max, turning activation quantization from dynamic into static. freeze() then bakes integer weights, and requantize() restores from safetensors.

Key numbers

1,052 stars; last functional release v0.2.7 on 2025-03-06; commits since are dependabot, CI-security, and docs only (latest 2026-08-03, 'docs: update maintenance references')

Ecosystem

README states: 'This project is currently in maintenance mode. We accept pull requests only for minor bug fixes, documentation improvements, and other maintenance tasks,' and recommends bitsandbytes or torchao instead. Its quantize/calibrate/tune/freeze/serialize API is still the clearest short illustration of a static activation-quant workflow.

Code · Source adversarially re-checked: confirmed

Mechanism

Wraps PyTorch or ONNX graphs with quantizer nodes, runs calibration to fit scales, and supports AWQ, GPTQ, SmoothQuant, and rotation (Hadamard) pre-processing to move activation outliers before quantizing. MX datatypes share one power-of-two scale per block of 32 values, static for weights and dynamic for activations.

Key numbers

156 stars on github.com/amd/Quark; v0.12 released 2026-07-03, last push 2026-07-15; docs site shows 0.12.post1; MIT; pip package amd-quark

Ecosystem

Exports JSON+safetensors, ONNX, and GGUF; vLLM documents a dedicated 'quark' quantization method that loads OCP-compliant MXFP4/MXFP6, and SGLang lists Quark among its offline formats. AMD publishes MXFP4 checkpoints (e.g. MiniMax-M2.1-MXFP4, GLM-5.2-MXFP4) on the Hub. Repo is source-complete but not accepting community contributions.

Code · Source adversarially re-checked: confirmed

Mechanism

Instead of round-to-nearest, AutoRound learns a per-weight rounding perturbation and per-group clipping range by sign gradient descent against the layer's own output on a small calibration set, minimizing block-wise reconstruction error. A model-free mode trades accuracy for speed.

Key numbers

1,582 stars; v0.14.2 released 2026-07-13; last push 2026-08-24; defaults to 128 samples of NeelNanda/pile-10k, ~10 min for a 7B on one GPU

Ecosystem

Consumed by vLLM (CUDA, Intel GPU, CPU), SGLang (which lists an auto-round-INT8 online path plus offline auto-round loading), and HF Transformers. Also selectable as an algorithm inside llm-compressor, so the same recipe is reachable from either entry point.

Code · Source adversarially re-checked: confirmed

Mechanism

AWQ search measures per-input-channel activation magnitude on calibration text and solves for a per-channel scaling factor that protects salient weight channels before INT4 group-wise (g=128) rounding; the reciprocal scale is folded into the previous layer, so activations stay FP16. TinyChat adds fused GEMM/GEMV kernels for prefill and decode.

Key numbers

3,619 stars; last push 2025-07-17; ~319 forks; reports 38 tok/s on NVIDIA Jetson Orin with the newer GEMM/GEMV kernels

Modern architectures

Covers instruction-tuned and multimodal LMs (VILA) and got DeepSeek-R1 and BF16 updates in 2025, so it is not strictly a 2023-era artifact.

Ecosystem

README itself points forward to QServe for large-batch serving. The group's quantization tooling has since consolidated into deepcompressor (now nunchaku-ai/deepcompressor, 800 stars, last push 2025-08-14). Useful for the review as the source of the activation-aware idea that later tools reimplement.

Code · Source adversarially re-checked: confirmed

Mechanism

QoQ ('quattuor-octo-quattuor') uses progressive group quantization so INT4 weights dequantize into INT8 registers, keeping all GEMM math on INT8 tensor cores and eliminating the fp16 dequant path that makes naive W4A8 slow. SmoothAttention migrates KV outliers before 4-bit KV quantization; compute-aware weight reordering hides remaining dequant in the memory pipeline.

Key numbers

854 stars; last push 2025-03-06 (last substantive commits 2025-02-21); 1.2-1.4x throughput over TensorRT-LLM on Llama-3-8B, 2.4-3.5x on Qwen1.5-72B; QServe on L40S beats TensorRT-LLM on A100, ~3x cost reduction; MLSys'25

Ecosystem

Checkpoints are produced by deepcompressor (github.com/nunchaku-ai/deepcompressor, 800 stars, last push 2025-08-14) and pre-quantized weights are published. Bundles LServe (unified sparse attention for long context) in the same repo. vLLM and SGLang have since grown their own W4A8 kernels, which is the maintained route today.

Code · Source adversarially re-checked: confirmed

Mechanism

Inserts pairs of orthogonal (randomized Hadamard) matrices Q and Q-transpose around linear layers so the network computes the same function while hidden states are rotated into a basis with no large-magnitude channels. Rotations at residual boundaries fold into adjacent weights for free; online Hadamard transforms handle the down-projection and attention paths.

Key numbers

532 stars; Apache-2.0; last substantive commit 2024-07. LLaMA2-70B W4A4KV4 loses at most 0.47 WikiText-2 perplexity (NeurIPS camera-ready figure; the earlier 0.29 was arXiv v1) and retains ~99% of zero-shot accuracy.

Ecosystem

The rotation itself is data-free; the optional GPTQ weight step uses calibration. Hadamard-transform metadata is now a first-class field in compressed-tensors and a selectable pre-processing step in llm-compressor and AMD Quark, so the deployable version of QuaRot is 'transforms' in those tools rather than this repo.

arXiv:2404.00456 · Code · Source adversarially re-checked: corrected here

Mechanism

Parameterizes the outlier-removing rotations as learnable orthogonal matrices and optimizes them with Cayley SGD on the Stiefel manifold against a small calibration set, keeping orthogonality exact so the network function is preserved. Requires fast-hadamard-transform; FSDP is used for larger models.

Key numbers

428 stars; last push 2025-02-14; CC-BY-NC-4.0; narrows the W4A4KV4 zero-shot gap on LLaMA-2 7B to 2.9 points vs full precision; released quantized Llama-3.2 variants

Ecosystem

Requires Python 3.9+, PyTorch 2.0+, CUDA, fast-hadamard-transform; FSDP for larger models. Because of the CC-BY-NC-4.0 license, deployable stacks generally reimplement the idea: llm-compressor's transforms and AMD Quark's rotation option are the licence-clean substitutes.

arXiv:2405.16406 · Code · Source adversarially re-checked: corrected here

Mechanism

Mixed-input GEMM: weights stay packed at 4 or 8 bits through HBM and shared memory and are dequantized in-register immediately before the tensor-core MMA, so memory traffic is 4-bit while the math stays 16-bit. Machete rebuilds this on CUTLASS 3.5.1 wgmma instructions with a pre-shuffled weight layout and deeper pipelining to stay compute-competitive on H100.

Key numbers

Machete: geomean +29% input-token and +32% output-token throughput on Llama-3.1 70B on one H100, +42% both on 405B across 4xH100; shipped in vLLM >= 0.6.2. Marlin repo IST-DASLab/marlin: 1,133 stars, Apache-2.0, last push 2024-09-04, claims ~4x speedup up to batch 16-32.

Ecosystem

vLLM exposes these as gptq_marlin / awq_marlin / machete backends and auto-selects them for compressed-tensors, GPTQ, and AWQ checkpoints; GPTQModel can emit FORMAT.MARLIN directly. The original Marlin repo is stale - the maintained descendants live in vLLM's csrc. Blog by Lucas Wilkinson, 2024-10-14.

Code · Source adversarially re-checked: confirmed

Mechanism

Transformers exposes each backend through an HfQuantizer subclass selected by a quantization_config, so the same from_pretrained call can swap in bitsandbytes, compressed-tensors, torchao, GPTQModel, Quark, or a hub kernel. The 'on the fly quantization' column is effectively a calibration-required flag, which is the fastest way to see which schemes need data.

Key numbers

NVFP4 and FINEGRAINED_FP8 are marked on-the-fly (no calibration); AQLM, AutoRound, AWQ, GPTQModel, compressed-tensors, Quark, SpQR and VPTQ are marked not-on-the-fly

Ecosystem

Evidence that 4-bit microscaled activation formats reached the mainstream loader layer: NVFP4 now ships as a kernels-community hub kernel rather than a research fork. Two newer table entries worth following for this review are IST-DASLab/FP-Quant (119 stars, last push 2026-02-26) and mit-han-lab/fouroversix (202 stars, MIT, 2026-04-21, adaptive block scaling for NVFP4).

Source adversarially re-checked: confirmed

Mechanism

Runs llm-compressor's INT8 W8A8 recipe (SmoothQuant plus GPTQ) on Granite 3.3 8B with a calibration set, serves the compressed-tensors checkpoint in vLLM on an L4, then reports paired accuracy and performance numbers against the bf16 baseline rather than perplexity alone.

Key numbers

99.43% GSM8K accuracy recovery (66.72% vs 67.1% baseline); ~40% faster time-to-first-token on synchronous requests; 4x context window on identical hardware; llm-compressor v0.6.0 + vLLM v0.9.2; by James Harmison, 2025-08-18

Ecosystem

Red Hat's broader claim across its Hugging Face checkpoint collection is over 99% baseline accuracy recovery. Companion posts cover the FP8 W8A8 rollout in vLLM and llm-compressor release notes. The int8 W8A8 path matters because A100 (SM 8.0) lacks FP8 tensor cores; the L4 used in the article is Ada (SM 8.9) and does have FP8, so int8-vs-fp8 there is a choice, not a constraint.

Source adversarially re-checked: corrected here

Mechanism

Two-level scaling. Each 4-bit element is E2M1 (1 sign, 2 exponent, 1 mantissa). Every 16 consecutive values share an FP8 E4M3 scale, so the scale can land between powers of two and track awkward local distributions; a single FP32 per-tensor factor keeps the FP8 scales in range. Hardware support is Blackwell tensor cores.

Key numbers

DeepSeek-R1-0528: <=1% accuracy degradation moving FP8 → NVFP4 on key language-modeling tasks; ~3.5x memory reduction vs FP16 and ~1.8x vs FP8; published 2025-06-24 by Alvarez, Almog, Chung, Layton, Stosic, Krashinsky, Aubrey

Modern architectures

The headline evidence is a large MoE (DeepSeek-R1-0528), the regime where activation and KV traffic rather than weight storage is the binding constraint - directly relevant to per-boundary activation compression at depth.

Ecosystem

Checkpoints are produced by NVIDIA Model Optimizer or llm-compressor and served by TensorRT-LLM and vLLM, with SGLang listed as upcoming at publication and now shipping NVFP4 offline and online paths. Pre-quantized NVFP4 frontier-model checkpoints are on the Hub. NVIDIA has a follow-up post on NVFP4 KV cache for long context and large batch (not fetched here).

Source adversarially re-checked: confirmed

Mechanism

Analyses the KV cache element distribution and finds an asymmetry: the key cache has persistent channel-wise outliers so it must be quantized per-channel, while the value cache has no such structure so per-token is correct. A streaming implementation quantizes newly appended tokens as they arrive with a small full-precision residual window, requiring no hyperparameter tuning.

Key numbers

2.6x less peak memory including model weights, enabling up to 4x larger batches; 2.35x-3.47x throughput on real workloads; works across Llama, Falcon and Mistral without tuning.

Modern architectures

BOUNDARY MARKER: KV cache is a persistent memory-capacity problem measured in GB and bytes-per-token, whereas our wire problem is a transient bandwidth problem measured in bytes-per-boundary-crossing. The per-channel-keys finding is nonetheless the same outlier geometry we hit, so the review should state the boundary rather than ignore the family.

Ecosystem

Widely reimplemented and the standard KV-quantization reference point; repo jy-yuan/KIVI referenced from the paper (search-confirmed, not separately fetched).

arXiv:2402.02750 · Code adversarially re-checked: confirmed

Mechanism

Combines per-channel key quantization; pre-RoPE key quantization, since applying RoPE before quantizing mixes channels and spreads outliers; non-uniform datatypes fit offline and weighted by per-layer sensitivity; and per-vector dense-and-sparse quantization that isolates a small fraction of outlier values at full precision.

Key numbers

<0.1 perplexity degradation at 3-bit across LLaMA variants and Mistral; 7B model served at 1M context on a single GPU and 10M multi-GPU; ~1.7x speedup over baseline.

Modern architectures

BOUNDARY MARKER, and the pre-RoPE insight is the transferable one: where in the computation you place the quantization step changes the outlier structure you have to survive. That is the direct analogue of choosing where in the residual stream a pipeline boundary quantizes.

Ecosystem

SqueezeAILab repo referenced from the paper (search-confirmed, not separately fetched). Latest arXiv revision May 2025.

arXiv:2401.18079 · Code adversarially re-checked: confirmed