Model wrappers: the role-aware decoder layer
Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857
In one sentence
An AFD model wrapper replaces each decoder layer with a role-aware layer. It builds only the modules its attention or FFN role owns.
The idea#
AFD runs one model as two server processes. The two processes load the same checkpoint, but each one needs only part of it. The attention process needs attention; the FFN process needs the feed-forward and MoE experts. A model wrapper makes this split at the decoder-layer level.
For DeepSeek, the wrapper is a small set of classes in
afd_plugin/model_executor/models/deepseek_v2.py. The key class is
AFDDeepseekV2DecoderLayer, a subclass of the native vLLM decoder layer
(afd_plugin/model_executor/models/deepseek_v2.py:375). Its constructor reads the AFD role, then
builds only that role's modules. The stage it does not own becomes a placeholder or a
remote FFN proxy.
An everyday comparison. Think of two cooks who share one recipe card. One cook keeps only the sauce steps; the other keeps only the bake steps. Neither buys the ingredients for the other half. The comparison stops here: the two cooks still pass the half-made dish between them on every step, which the connector does for hidden states.
The picture#
Read panel 1 top to bottom. The attention role always builds the native attention module, then picks one of
four branches for mlp. Panel 2 shows the FFN role, where the attention module is dropped and
mlp is the real compute. Panel 3 lists the four model families and their main differences.
In the code#
Step 1: One decoder-layer class, two roles#
AFDDeepseekV2DecoderLayer.__init__ reads the role from the AFD config, then calls
torch.nn.Module.__init__ directly instead of the native constructor. This way, it never allocates
the modules of the other role (afd_plugin/model_executor/models/deepseek_v2.py:391-396).
# ### PATCH START: require an explicit AFD role before allocation.
afd_config = parse_afd_config(vllm_config, validate=False)
afd_role = afd_config.role
torch.nn.Module.__init__(self)
# ### PATCH ENDThe layer also records whether it is a MoE layer and whether the gate runs on the attention side
(afd_plugin/model_executor/models/deepseek_v2.py:423-444). The flag
uses_remote_experts is true only on CUDA for a MoE layer
(afd_plugin/model_executor/models/deepseek_v2.py:444).
Step 2: The attention role builds native attention and one of four mlp branches#
On the attention role, self_attn is always a native attention module; the class picks the
right attention variant for the model (afd_plugin/model_executor/models/deepseek_v2.py:459-491).
The mlp stage then takes one of four branches. The table names the class and the exact lines.
| Branch (condition) | mlp becomes | Lines |
|---|---|---|
CUDA MoE layer (uses_remote_experts) | AFDDeepseekV2RemoteExpertsMoE (a remote-experts shell) | afd_plugin/model_executor/models/deepseek_v2.py:493-500 |
| NPU MoE layer + compute_gate_on_attention | GateOnlyRemoteMoE | afd_plugin/model_executor/models/deepseek_v2.py:501-507 |
| dense layer + gate on attention | native DeepseekV2MLP (computed locally) | afd_plugin/model_executor/models/deepseek_v2.py:508-515 |
| otherwise (dense default, or MoE with the gate on the FFN side) | RemoteFFNProxy | afd_plugin/model_executor/models/deepseek_v2.py:516-517 |
if self.uses_remote_experts:
self.mlp = AFDDeepseekV2RemoteExpertsMoE(
config=config,
parallel_config=parallel_config,
layer_idx=layer_idx,
prefix=f"{prefix}.mlp",
compute_gate_on_attention=self.compute_gate_on_attention,
)
elif self.compute_gate_on_attention and self.is_moe_layer:
self.mlp = GateOnlyRemoteMoE(
config=config,
layer_idx=layer_idx,
prefix=f"{prefix}.mlp",
vllm_config=vllm_config,
)
elif self.compute_gate_on_attention:
self.mlp = native.DeepseekV2MLP(
hidden_size=config.hidden_size,
intermediate_size=config.intermediate_size,
hidden_act=config.hidden_act,
quant_config=quant_config,
prefix=f"{prefix}.mlp",
)
else:
self.mlp = RemoteFFNProxy(layer_idx=layer_idx)Step 3: The FFN role drops attention and keeps the real FFN#
On the FFN role, self_attn is a PPMissingLayer: a placeholder that holds no
weights (afd_plugin/model_executor/models/deepseek_v2.py:520). The mlp stage takes one
of three branches.
elif afd_role == "ffn":
self.self_attn = native.PPMissingLayer()
if self.compute_gate_on_attention and not self.is_moe_layer:
self.mlp = native.PPMissingLayer()
elif self.is_moe_layer:
# vLLM models bind FusedMoEFactory at module import time. AFD
# can import native DeepSeek before vLLM-Ascend patches the
# package factory, so refresh that binding after platform
# initialization and before constructing the NPU FFN MoE.
if device_type == "npu":
native.FusedMoEFactory = fused_moe.FusedMoEFactory
moe_config = config
if afd_config.connector == AFD_ASYNC_CONNECTOR:
moe_config = copy(config)
# HF validates this public field as int, but pinned native
# MoE uses only None to omit shared construction (0 builds
# a zero-width MLP). Apply its sentinel to this private
# construction copy; preserve the real model config.
object.__setattr__(moe_config, "n_shared_experts", None)
self.mlp = native.DeepseekV2MoE(
config=moe_config,
parallel_config=parallel_config,
quant_config=quant_config,
prefix=f"{prefix}.mlp",
# aiter applies routed_scaling_factor internally
apply_routed_scale_to_output=(
not native.rocm_aiter_ops.is_fused_moe_enabled()
),
)
if self.compute_gate_on_attention and device_type == "cuda":
# Keep the gate weights loadable, but with the runner's
# gate cleared it consumes the Attention-side router
# logits instead of computing its own (gate is None means
# external routing in the v0.30 runner contract).
self.mlp.experts.gate = None
else:
self.mlp = native.DeepseekV2MLP(
hidden_size=config.hidden_size,
intermediate_size=config.intermediate_size,
hidden_act=config.hidden_act,
quant_config=quant_config,
prefix=f"{prefix}.mlp",
)- Dense layer with the gate on the attention side. The dense MLP runs on the attention
side, so the FFN role drops it too (
afd_plugin/model_executor/models/deepseek_v2.py:521-522). - MoE layer. The FFN role builds the native
DeepseekV2MoE(afd_plugin/model_executor/models/deepseek_v2.py:538-547). On CUDA with the gate on the attention side, it clears the runner gate, so the experts consume the transported router logits (afd_plugin/model_executor/models/deepseek_v2.py:548-553). - Dense layer (the gate stays on the FFN side). The FFN role builds the native
DeepseekV2MLP(afd_plugin/model_executor/models/deepseek_v2.py:554-561).
Step 4: RemoteFFNProxy sends, yields, then receives#
The remote FFN proxy is a parameter-free module
(afd_plugin/model_executor/models/deepseek_v2.py:117-125). Its
_send_and_receive is the one place where the attention role crosses to the FFN role inside a
layer.
def _send_and_receive(
self,
hidden_states: torch.Tensor,
**send_kwargs: torch.Tensor,
) -> torch.Tensor:
afd_metadata = get_afd_metadata_from_forward_context()
if afd_metadata is None:
raise RuntimeError("RemoteFFNProxy requires AFD forward metadata")
forward_context = get_forward_context()
stage_idx = int(
getattr(forward_context, "ubatch_idx", afd_metadata.stage_idx),
)
afd_metadata.stage_idx = stage_idx
metadata = AFDTransferMetadata.create_attention_metadata(
layer_idx=self.layer_idx,
stage_idx=stage_idx,
seq_len=int(hidden_states.shape[0]),
)
context = AFDTransferContext(metadata=metadata)
afd_metadata.connector.send_attn_output(
hidden_states,
context,
**send_kwargs,
)
hidden_states = maybe_apply_dbo_yield(
hidden_states,
role="attention",
)
return afd_metadata.connector.recv_ffn_output(
ref_tensor=hidden_states,
ubatch_idx=stage_idx,
)- Lines 132-139. Read the AFD forward metadata and the stage index from the vLLM forward context.
- Lines 140-145. Build the transfer metadata for this layer and stage, with the row count equal to the number of tokens.
- Line 146.
send_attn_outputsends the hidden states to the FFN side. - Line 151.
maybe_apply_dbo_yieldyields to the other microbatch under DBO. - Line 155.
recv_ffn_outputreceives the FFN result into the same tensor and returns it.
Step 5: Weight loading keeps only this role's checkpoint paths#
Both roles read the same checkpoint, but each keeps only its own weights. The causal-LM wrapper filters the
weight stream before the native loader sees it
(afd_plugin/model_executor/models/deepseek_v2.py:964-974).
def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
return super().load_weights(
_iter_role_weights(
weights,
role=self.afd_role,
config=self.config,
compute_gate_on_attention=self.afd_config.compute_gate_on_attention,
attention_shared_experts=self.afd_config.connector
== AFD_ASYNC_CONNECTOR,
)
)The filter is _iter_role_weights, which calls _checkpoint_weight_roles for each
weight path and keeps a weight only when this role owns it
(afd_plugin/model_executor/models/deepseek_v2.py:98-114).
def _checkpoint_weight_roles(
name: str,
config: _DeepseekAdapterConfig,
*,
compute_gate_on_attention: bool,
attention_shared_experts: bool = False,
) -> frozenset[str]:
"""Classify one native checkpoint path by its AFD execution owner."""
layer_path = _weight_layer_path(name)
if layer_path is None:
return _BOTH_ROLES
layer_idx, stage, remainder = layer_path
if stage == "self_attn":
return _ATTENTION_ROLE
if stage != "mlp":
return _BOTH_ROLES
if not _is_moe_layer(config, layer_idx):
return _ATTENTION_ROLE if compute_gate_on_attention else _FFN_ROLE
if attention_shared_experts and remainder and remainder[0] == "shared_experts":
return _ATTENTION_ROLE
is_moe_gate = bool(remainder) and remainder[0] == "gate"
if is_moe_gate and compute_gate_on_attention:
return _BOTH_ROLES
return _FFN_ROLE- Lines 82-83. A
self_attnweight belongs to the attention role. - Lines 84-85. A weight that is not under
mlpbelongs to both roles (for example embeddings). - Lines 87-88. A dense-layer
mlpweight follows the gate flag: attention side when the gate runs there, otherwise the FFN side. - Lines 92-95. A MoE
gateweight goes to both roles when the gate runs on the attention side; the other expert weights go to the FFN role.
Deep dive#
Deep diveThe other three model families, one at a time
Four wrappers follow the same pattern. The differences are the native base model, the device support, and the extra payloads.
Qwen3-MoE (afd_plugin/model_executor/models/qwen3_moe.py:102). The wrapper reuses
the native decoder_layer_type hook instead of rebuilding make_layers
(afd_plugin/model_executor/models/qwen3_moe.py:198-202). It supports CUDA only and requires
compute_gate_on_attention=false, so the gate always runs on the FFN side
(afd_plugin/model_executor/models/qwen3_moe.py:82-87). On the attention role, every
layer (dense or MoE) uses a plain RemoteFFNProxy
(afd_plugin/model_executor/models/qwen3_moe.py:157); there is no remote-experts shell. The FFN role
builds the native Qwen3MoeSparseMoeBlock or Qwen3MoeMLP
(afd_plugin/model_executor/models/qwen3_moe.py:167-180).
Qwen3.5 / 3.6 MoE. The code registers the architecture
Qwen3_5MoeForConditionalGeneration as AFDQwen3_5MoeForConditionalGeneration
(afd_plugin/model_executor/models/qwen3_5.py:444). It supports CUDA only
(afd_plugin/model_executor/models/model_utils.py:34-38) and text-only execution; a multimodal config
raises (afd_plugin/model_executor/models/qwen3_5.py:22-29). It uses a remote-experts shell built on the
native Qwen3NextSparseMoeBlock (afd_plugin/model_executor/models/qwen3_5.py:88-90), and
reuses AFDAttentionFusedMoE from the DeepSeek wrapper
(afd_plugin/model_executor/models/qwen3_5.py:15).
DeepSeek-V4 (CUDA wrapper,
afd_plugin/model_executor/models/deepseek_v4.py:574-578). It supports CUDA only and the synchronous
P2P connector (afd_plugin/model_executor/models/deepseek_v4.py:392-397). Its hash-router MoE needs
the token ids, so the wrapper sets afd_requires_input_ids = True
(afd_plugin/model_executor/models/deepseek_v4.py:578) and its proxy sends the
input_ids alongside the hidden states
(afd_plugin/model_executor/models/deepseek_v4.py:108-112). The mHC residual state stays on the
attention role (afd_plugin/model_executor/models/deepseek_v4.py:175-178).
Deep diveThe NPU wrappers (named here, covered on the overview page)
The model table maps DeepseekV4ForCausalLM to the NPU wrapper only when
torch_npu is importable; otherwise it maps to the CUDA wrapper
(afd_plugin/__init__.py:109-113). The NPU DeepSeek-V4 wrapper is
afd_plugin/model_executor/models/npu/deepseek_v4.py
(afd_plugin/model_executor/models/npu/deepseek_v4.py:649). The DeepSeek V2/V3 NPU gate helpers live
next to it (afd_plugin/model_executor/models/npu/deepseek_v2_attention_gate.py:31-80). The
Ascend NPU overview: the same contract, different transports page reads the NPU code; this page covers the CUDA path.
Deep diveWorked example: which modules load for DeepSeek-V2-Lite
Take DeepSeek-V2-Lite with the GPU connector and compute_gate_on_attention=false. This model
has 27 decoder layers, with the first layer dense and the rest MoE. The 27 and the dense/MoE split are
inferred from the DeepSeek-V2-Lite checkpoint config; they are not stored in either repository.
- Attention role. Each layer builds a native attention module. Layer 0 (dense) and every
MoE layer use the branches from Step 2: layer 0 gets a
RemoteFFNProxy, each MoE layer gets the remote-experts shell on CUDA. No expert weights load, because_checkpoint_weight_rolessends them to the FFN role. - FFN role. Each
self_attnis aPPMissingLayer, so no attention weights load. Layer 0 builds the native dense MLP; each MoE layer builds the native MoE block with its experts. The attention and embedding weights do not load. - Shared weights. The embeddings, the final norm and the LM head are not under a decoder
mlporself_attn, so_checkpoint_weight_rolesreturns both roles (afd_plugin/model_executor/models/deepseek_v2.py:78-79).
What can go wrong#
- EPLB on CUDA remote experts. The attention role with CUDA remote experts and EPLB
raises at construction (
afd_plugin/model_executor/models/deepseek_v2.py:445-452). - Gate on attention off CUDA and NPU.
compute_gate_on_attentionneeds CUDA or NPU; another device raises (afd_plugin/model_executor/models/deepseek_v2.py:453-456). - No AFD metadata in the context. The proxy raises when the forward context has no AFD
metadata (
afd_plugin/model_executor/models/deepseek_v2.py:133-134). - DeepSeek-V4 without the P2P connector. The wrapper raises if the connector is not the
synchronous P2P connector (
afd_plugin/model_executor/models/deepseek_v4.py:394-397).
Why it matters. The role filter and the branch set are what let two processes load one checkpoint without duplicating the large weights. A wrong branch would load the wrong half, or run the FFN in the wrong process.
Check yourself#
- 1On the FFN role, what iseasy
self_attn?Show answer
A
PPMissingLayer: a placeholder with no weights (afd_plugin/model_executor/models/deepseek_v2.py:520). The FFN role does no attention. - 2Which model family must run the gate on the FFN side?easy
Show answer
Qwen3-MoE. The wrapper requires
compute_gate_on_attention=false(afd_plugin/model_executor/models/qwen3_moe.py:84-87). - 3On the attention role, which class does a CUDA MoE layer build formedium
mlp, and why notGateOnlyRemoteMoE?Show answer
AFDDeepseekV2RemoteExpertsMoE(afd_plugin/model_executor/models/deepseek_v2.py:493-500). On CUDA,uses_remote_expertsis true for a MoE layer, so the first branch is taken;GateOnlyRemoteMoEis the NPU path (afd_plugin/model_executor/models/deepseek_v2.py:444). - 4How does the attention role avoid loading the expert weights?medium
Show answer
load_weightswraps the weight stream in_iter_role_weights, which keeps a weight only when the role owns it. Expert weights map to the FFN role (afd_plugin/model_executor/models/deepseek_v2.py:964-974,afd_plugin/model_executor/models/deepseek_v2.py:92-95). - 5Why does DeepSeek-V4 sethard
afd_requires_input_ids, and where does the value change the run?Show answer
Its hash-router MoE needs the token ids to route (
afd_plugin/model_executor/models/deepseek_v4.py:578). The attention proxy sends the ids (afd_plugin/model_executor/models/deepseek_v4.py:108-112), and the FFN runner reads the flag to request them from the connector (afd_plugin/v1/worker/ffn_model_runner.py:221).