Pipeline-parallel communication

Compressing activations on the pipeline wire

We measured how BF16, 8-bit, and 4-bit wire formats reduce transferred bytes while preserving model quality as pipeline depth, model size, and decode horizon increase.

Qwen3.6-35B-A3B and Qwen3.5-122B-A10B-FP8 on 4 × and 8 × RTX 5090 systems.

TLDR

Only the pipeline boundary traffic is quantized. The hidden_states and residual tensors are compressed for each stage to stage send and reconstructed on arrival. Weights, KV cache and all compute stay untouched. Measured at 1, 3 and 7 boundaries on two mixture of experts models, a 35B with BF16 weights and a 122B with FP8 weights.
8-bit is free, but which 8-bit matters once the model is large. On the 35B every 8-bit format costs at most +0.29% NLL and leaves GSM8K accuracy unchanged. On the 122B they separate: FP8 and MXFP8 sit at the control while INT8 costs +1.17% NLL for the same 4,104 bytes per token under the same per-token scale. A floating point element absorbs activation outliers that a uniform integer grid cannot, so element format decides this, not bit width.
At 4-bit the scale granularity decides. Per-token INT4 collapses to +495% NLL and 0% GSM8K accuracy at seven boundaries. Microscaled NVFP4 holds at +1.86% NLL and 98% GSM8K while compressing 3.54 times. MXFP4 lands between them at +6.65% and 89%, and the distance to NVFP4 comes mostly from its power of two E8M0 scale rather than from its larger block.
Error accumulates with boundary count, never with decode position. The NLL penalty grows from 1 to 3 to 7 boundaries for every lossy format, while top-1 agreement against a frozen trajectory stays flat across 1,024 decode steps for every usable format, including INT4 at its own degraded level.
NLL and GSM8K accuracy answer different questions. NLL is teacher forced over 24,576 tokens and resolves shifts far below what a 100 question benchmark can see. Across all 30 measured cells GSM8K moved significantly only where NLL exceeded about 11%, so NLL is the early warning instrument and GSM8K accuracy is the survival check. Differences of a point or two in accuracy are inside the sampling interval and should not be read as a ranking.

2.00×

INT8 halves boundary traffic with no material quality loss in the tested range.

3.54×

NVFP4 retains strong quality through seven boundaries and is the best tested 4-bit trade-off.

1,024 steps

No tested format shows progressive quality decay across the decode horizon.

01 · Mechanism

Only the bytes between pipeline stages change

Stock SGLang sends hidden_states and residual as model-dtype tensors at each pipeline boundary. The experimental codec quantizes those two tensors before the send and reconstructs BF16 immediately after the receive.

One boundary crossing

Pipeline stage n hidden_states [tokens, H] residual [tokens, H] BF16 compute

quantize NCCL payload dequantize

Pipeline stage n + 1 hidden_states [tokens, H] residual [tokens, H] BF16 compute resumes

02 · Format map

Bit width is only half of the design

The key distinction is how many values share a scale. A single scale across 2,048 channels is cheap, but one outlier can reduce resolution for the whole token. Microscaling confines that loss to blocks of 16 or 32 values.

4-bit methods tested

All three store 4-bit values. Their scale granularity determines whether outlier channels damage an entire token or only a small block.

INT4One scale for all 2,048 values in a token row. This is the failing 4-bit method.

MXFP4E2M1 values with one power-of-two E8M0 scale per 32 values.

NVFP4E2M1 values with one E4M3 scale per 16 values and a global token scale.

Result. Microscaling rescues 4-bit quality. NVFP4 is the most robust tested 4-bit method. MXFP4 remains strong on 35B, but loses more quality on 122B at seven boundaries.

8-bit methods tested

All three are close to BF16 quality on the 35B model. On the 122B model the two floating point formats stay at the control while the integer format does not.

INT8Integer values with one FP32 scale per token. It has the most mature fused implementation, but it is the only 8-bit format that loses measurable quality on the 122B model.

FP8E4M3 floating-point values with one FP32 scale per token. It was tested at one, three, and seven boundaries.

MXFP8E4M3 values with one E8M0 scale per 32 values. It has slightly more metadata.

Result. MXFP8 has the smallest seven-boundary NLL penalty among the tested 8-bit formats, but its metadata makes it slightly larger than INT8. INT8 is the practical twofold wire baseline.

Wire cost for one token crossing one boundary. Each crossing includes two tensors and uses H = 2,048.
Family	Wire format	Scale granularity	Bytes	Compression	Observed role
Normal floating point
Baseline	BF16	No quantization	8,192	1.00×	Quality control and wire baseline
4-bit compression
Per-token integer	INT4	One FP32 scale per token	2,056	3.98×	Fails on quality
Microscaled float	MXFP4	One E8M0 scale per 32 values	2,176	3.76×	Sensitive on the larger model
Microscaled float	NVFP4	One E4M3 scale per 16 values plus a token scale	2,312	3.54×	Strongest tested 4-bit option
8-bit compression
Per-token integer	INT8	One FP32 scale per token	4,104	2.00×	Costs +1.17% NLL on the 122B
Per-token float	FP8 E4M3	One FP32 scale per token	4,104	2.00×	Quality-safe through seven boundaries
Microscaled float	MXFP8	One E8M0 scale per 32 values	4,224	1.94×	Quality-safe through seven boundaries

Payload per token and boundary

Smaller bars mean fewer bytes on the wire. Scale metadata is included.

↓ fewer bytes is better

Why microscaling wins at 4 bits. INT4 and NVFP4 have the same value width, but they do not share scale information in the same way. NVFP4 isolates outlier channels inside 16-value blocks. That difference is more important than the small metadata overhead.

03 · Experiment design

What we varied and why

The primary question is how far boundary traffic can be reduced before repeated quantization changes model behavior. Every quality result is compared with BF16 wire tensors at the same model size and pipeline topology.

Target
Only pipeline traffic changes. The experiment compresses hidden_states and residual for each stage-to-stage send. Computation resumes in BF16 after each receive.
Four GPUs
The 35B grid used 4 × RTX 5090. PP2/TP2 creates one boundary. PP4/TP1 creates three boundaries.
Eight GPUs
The depth extension used 8 × RTX 5090. PP8/TP1 creates seven boundaries and provides the 1,024-step stress test.
Larger model
The 122B checkpoint uses FP8 weights because that is what fit. It missed the four-GPU memory budget by 18 MB. Native BF16 weights did not fit on eight 32 GB GPUs. FP8 describes stored model weights, while the control still sends BF16 activations between stages.
Quality checks
WikiText NLL and GSM8K answer accuracy test different failure modes. NLL is the mean negative log likelihood per token, measured teacher forced: the model scores the true next token at every position given the true prefix, so no text is generated and no sampling is involved. Each cell covers 24,576 tokens, being 24 non-overlapping 1,024 token windows from the WikiText-2 raw test split, and the tables report the change relative to the matching BF16 control, where lower is better. The measurement is deterministic to the last floating point digit on this stack, which is what makes changes well below a tenth of a percent readable: the same configuration re-run three weeks later on a different machine returned a value identical in all 17 significant digits. NLL detects small distribution shifts that leave greedy answers intact. GSM8K tests whether greedy generated answers remain correct. Each GSM8K cell is one greedy pass, at temperature 0, over the same first 100 questions of the test split, graded by numeric match.
System support
The uncompressed SGLang path sends model-dtype PP tensors. This research branch adds INT8, FP8, INT4, MXFP8, MXFP4, and NVFP4 wire modes. RTX 5090 is a 32 GB Blackwell GPU with FP8 and FP4 Tensor Core arithmetic, but the microscaled wire encoders in this study are still eager prototypes rather than fused hardware paths. NVIDIA architecture reference.

04 · Quality and scale

Repeated boundaries expose scaling failures

Every additional boundary inserts another quantize, send, and reconstruct step before later layers consume the activation. Fine-grained formats keep each error small. A coarse per-token scale re-injects a larger error at every boundary, which later stages can amplify.

INT4 failure is about its scale

The failing INT4 method uses one scale across all 2,048 values. On 35B, its NLL penalty grows from +11% to +59% and then +190% as boundaries increase from one to three and seven. This does not apply to microscaled MXFP4 or NVFP4.

NVFP4 is more robust than MXFP4

NLL favors NVFP4 at every shared topology. The clearest separation is the 122B model at seven boundaries, where NVFP4 records +1.86% NLL and 98% GSM8K while MXFP4 records +6.65% and 89%. That MXFP4 result was re-run on separate hardware and reproduced exactly, to every digit of NLL and to all one hundred GSM8K predictions, so it reflects the format rather than one unlucky run.

MXFP4 is still viable on 35B

MXFP4 stays within +1.66% NLL through seven boundaries on the 35B model. The separation from NVFP4 comes mostly from the scale format rather than the block count. Replacing MXFP4's power-of-two E8M0 scale with NVFP4's E4M3 scale accounts for 76 to 90 percent of the measured gap, while halving the block from 32 to 16 values accounts for 1 to 23 percent.

The 8-bit split appears only at scale

On the 35B model the three 8-bit formats are equivalent, spanning +0.08% to +0.29% NLL at seven boundaries. On the 122B model they separate: FP8 and MXFP8 hold at the control, -0.02% and -0.00%, while INT8 costs +1.17%. FP8 and INT8 send the same 4,104 bytes per token under the same per-token scale, so element format is the only variable between them. A floating point exponent absorbs the activation outliers that a uniform integer grid cannot, and that matters more as the hidden size grows. Prefer FP8 or MXFP8 over INT8 for an 8-bit wire, and note that INT8 at 2.00x compression is barely ahead of NVFP4 at 3.54x.

35B BF16-weight quality cost by pipeline depth

The deployable range and the INT4 failure range use separate scales so the near-BF16 formats remain readable.

↓ lower is better · 0% = BF16

Deployable range

Per-token INT4 is excluded here. Lower values are better.

●INT8 ■FP8 ▲MXFP8 ◆MXFP4 ⬟NVFP4

INT4 failure range

The same lower-is-better metric on a separate scale.

The larger model narrows the margin

At seven boundaries, the NLL penalty is roughly four times larger on 122B-FP8 than on 35B for every shared usable format. INT8 grows from +0.29% to +1.17%, NVFP4 from +0.43% to +1.86%, and MXFP4 from +1.66% to +6.65%. The larger hidden size creates a 1.5 times larger activation payload, but model size and FP8 weight precision are confounded. This is a sensitivity result, not a pure scaling law. Width alone does not explain the MXFP4 gap: running the wire codecs directly on synthetic activations, the MXFP4 to NVFP4 error ratio is flat between hidden 2048 and 3072, and instead rises steeply with activation outlier severity, which points at the larger checkpoint's residual stream rather than its width.

122B model with FP8 weights: quality cost

NVFP4 remains close to INT8. MXFP4 separates as depth increases.

↓ lower is better · 0% = BF16

●INT8 ◆MXFP4 ⬟NVFP4 ■FP8 ▲MXFP8

INT4 is outside the plotted range. Its NLL penalty is 421.20% at three boundaries and 495.31% at seven. GSM8K is 0% in both cases.

GSM8K accuracy confirms the family split, not a ranking

GSM8K checks generated answers rather than teacher-forced likelihood. Higher values are better. Each bar is a single greedy decoding pass, at temperature 0, over the same first 100 questions of the GSM8K test split, graded by numeric match. That makes these bars a screen for gross damage at an early stage of the experiment, not a ranking. One hundred questions is one hundred questions: a gap of two or three points is two or three answers, so read the large gaps and ignore the small ones.

35B BF16-weight GSM8K accuracy by bit-width family

Both panels use the full 0 to 100% scale, so bar heights are directly comparable. Each bar is labelled with its score.

↑ higher is better

Microscaled 4-bit methods

BF16, MXFP4, and NVFP4 on the shared scale.

BF16 MXFP4 NVFP4

8-bit methods

BF16, INT8, FP8, and MXFP8 on the shared scale.

BF16 INT8 FP8 MXFP8

Per-token INT4 is not plotted in either panel. It is the non-microscaled control rather than a candidate format, and it fails outright: 64%, then 3%, then 0% as boundaries increase from one to three and seven.

122B model with FP8 weights: GSM8K accuracy

MXFP4 at seven boundaries is the only gap in this set wider than a few questions. INT4 is omitted from the plot because it records 0% at both depths.

↑ higher is better

BF16 INT8 MXFP4 NVFP4 FP8 MXFP8

05 · Decode horizon

Error does not grow with decode position

The boundary experiment asks what happens within one forward pass. The decode-horizon probe asks whether the model carries compression damage forward through its recurrent state as the context grows.

How the probe works

1 · Freeze
BF16 first generates twelve greedy 1,024-token continuations from six WikiText and six GSM8K prompts. These token sequences become the shared references.
2 · Replay
Each wire format replays the same prompt and continuation with teacher forcing at seven pipeline boundaries. Input tokens are identical, while activations and recurrent state evolve under that format.
3 · Compare
At every position, the probe records NLL and whether the format's top prediction matches the frozen BF16 token. The 1,024 positions are averaged into eight 128-token segments.
4 · Interpret
A downward trend would indicate progressive drift as context grows. The usable formats remain flat, and their NLL improves with added context in the same direction as BF16.

Top-1 agreement across 1,024 decode steps

Each point is one 128-step segment on the 35B model with seven boundary crossings per forward pass.

↑ higher is better · flat is stable

BF16 and usable formats

The vertical scale runs from 94% to 100%.

○BF16 ●INT8 ◆MXFP4 ⬟NVFP4 ■FP8 ▲MXFP8

INT4 failure level

The vertical scale runs from 20% to 35%.

What this supports. Additional pipeline boundaries can amplify coarse quantization error inside one forward pass. The tested frozen trajectories show no progressive representational drift across decode position. Because the probe is teacher-forced, it does not prove that every free-running generation will remain identical. GSM8K provides the separate free-running task check.