AFD Plugin Explained

Model wrappers: the role-aware decoder layer

AdvancedAFD conceptsabout 8 min read + 3 min deep dives

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

In one sentence

An AFD model wrapper replaces each decoder layer with a role-aware layer. It builds only the modules its attention or FFN role owns.

The idea#

AFD runs one model as two server processes. The two processes load the same checkpoint, but each one needs only part of it. The attention process needs attention; the FFN process needs the feed-forward and MoE experts. A model wrapper makes this split at the decoder-layer level.

For DeepSeek, the wrapper is a small set of classes in afd_plugin/model_executor/models/deepseek_v2.py. The key class is AFDDeepseekV2DecoderLayer, a subclass of the native vLLM decoder layer (afd_plugin/model_executor/models/deepseek_v2.py:375). Its constructor reads the AFD role, then builds only that role's modules. The stage it does not own becomes a placeholder or a remote FFN proxy.

An everyday comparison. Think of two cooks who share one recipe card. One cook keeps only the sauce steps; the other keeps only the bake steps. Neither buys the ingredients for the other half. The comparison stops here: the two cooks still pass the half-made dish between them on every step, which the connector does for hidden states.

The picture#

Figure: each role builds only its own modules. The attention role has four mlp branches; the FFN role has three.
Open full size ↗

Read panel 1 top to bottom. The attention role always builds the native attention module, then picks one of four branches for mlp. Panel 2 shows the FFN role, where the attention module is dropped and mlp is the real compute. Panel 3 lists the four model families and their main differences.

In the code#

Step 1: One decoder-layer class, two roles#

AFDDeepseekV2DecoderLayer.__init__ reads the role from the AFD config, then calls torch.nn.Module.__init__ directly instead of the native constructor. This way, it never allocates the modules of the other role (afd_plugin/model_executor/models/deepseek_v2.py:391-396).

        # ### PATCH START: require an explicit AFD role before allocation.
        afd_config = parse_afd_config(vllm_config, validate=False)
        afd_role = afd_config.role

        torch.nn.Module.__init__(self)
        # ### PATCH END
The constructor reads the role before any allocation.

The layer also records whether it is a MoE layer and whether the gate runs on the attention side (afd_plugin/model_executor/models/deepseek_v2.py:423-444). The flag uses_remote_experts is true only on CUDA for a MoE layer (afd_plugin/model_executor/models/deepseek_v2.py:444).

Step 2: The attention role builds native attention and one of four mlp branches#

On the attention role, self_attn is always a native attention module; the class picks the right attention variant for the model (afd_plugin/model_executor/models/deepseek_v2.py:459-491).

The mlp stage then takes one of four branches. The table names the class and the exact lines.

Branch (condition)mlp becomesLines
CUDA MoE layer (uses_remote_experts)AFDDeepseekV2RemoteExpertsMoE (a remote-experts shell)afd_plugin/model_executor/models/deepseek_v2.py:493-500
NPU MoE layer + compute_gate_on_attentionGateOnlyRemoteMoEafd_plugin/model_executor/models/deepseek_v2.py:501-507
dense layer + gate on attentionnative DeepseekV2MLP (computed locally)afd_plugin/model_executor/models/deepseek_v2.py:508-515
otherwise (dense default, or MoE with the gate on the FFN side)RemoteFFNProxyafd_plugin/model_executor/models/deepseek_v2.py:516-517
            if self.uses_remote_experts:
                self.mlp = AFDDeepseekV2RemoteExpertsMoE(
                    config=config,
                    parallel_config=parallel_config,
                    layer_idx=layer_idx,
                    prefix=f"{prefix}.mlp",
                    compute_gate_on_attention=self.compute_gate_on_attention,
                )
            elif self.compute_gate_on_attention and self.is_moe_layer:
                self.mlp = GateOnlyRemoteMoE(
                    config=config,
                    layer_idx=layer_idx,
                    prefix=f"{prefix}.mlp",
                    vllm_config=vllm_config,
                )
            elif self.compute_gate_on_attention:
                self.mlp = native.DeepseekV2MLP(
                    hidden_size=config.hidden_size,
                    intermediate_size=config.intermediate_size,
                    hidden_act=config.hidden_act,
                    quant_config=quant_config,
                    prefix=f"{prefix}.mlp",
                )
            else:
                self.mlp = RemoteFFNProxy(layer_idx=layer_idx)
The four mlp branches on the attention role.

Step 3: The FFN role drops attention and keeps the real FFN#

On the FFN role, self_attn is a PPMissingLayer: a placeholder that holds no weights (afd_plugin/model_executor/models/deepseek_v2.py:520). The mlp stage takes one of three branches.

        elif afd_role == "ffn":
            self.self_attn = native.PPMissingLayer()
            if self.compute_gate_on_attention and not self.is_moe_layer:
                self.mlp = native.PPMissingLayer()
            elif self.is_moe_layer:
                # vLLM models bind FusedMoEFactory at module import time. AFD
                # can import native DeepSeek before vLLM-Ascend patches the
                # package factory, so refresh that binding after platform
                # initialization and before constructing the NPU FFN MoE.
                if device_type == "npu":
                    native.FusedMoEFactory = fused_moe.FusedMoEFactory
                moe_config = config
                if afd_config.connector == AFD_ASYNC_CONNECTOR:
                    moe_config = copy(config)
                    # HF validates this public field as int, but pinned native
                    # MoE uses only None to omit shared construction (0 builds
                    # a zero-width MLP). Apply its sentinel to this private
                    # construction copy; preserve the real model config.
                    object.__setattr__(moe_config, "n_shared_experts", None)
                self.mlp = native.DeepseekV2MoE(
                    config=moe_config,
                    parallel_config=parallel_config,
                    quant_config=quant_config,
                    prefix=f"{prefix}.mlp",
                    # aiter applies routed_scaling_factor internally
                    apply_routed_scale_to_output=(
                        not native.rocm_aiter_ops.is_fused_moe_enabled()
                    ),
                )
                if self.compute_gate_on_attention and device_type == "cuda":
                    # Keep the gate weights loadable, but with the runner's
                    # gate cleared it consumes the Attention-side router
                    # logits instead of computing its own (gate is None means
                    # external routing in the v0.30 runner contract).
                    self.mlp.experts.gate = None
            else:
                self.mlp = native.DeepseekV2MLP(
                    hidden_size=config.hidden_size,
                    intermediate_size=config.intermediate_size,
                    hidden_act=config.hidden_act,
                    quant_config=quant_config,
                    prefix=f"{prefix}.mlp",
                )
The FFN role: a missing attention module and the real FFN/MoE block.
  1. Dense layer with the gate on the attention side. The dense MLP runs on the attention side, so the FFN role drops it too (afd_plugin/model_executor/models/deepseek_v2.py:521-522).
  2. MoE layer. The FFN role builds the native DeepseekV2MoE (afd_plugin/model_executor/models/deepseek_v2.py:538-547). On CUDA with the gate on the attention side, it clears the runner gate, so the experts consume the transported router logits (afd_plugin/model_executor/models/deepseek_v2.py:548-553).
  3. Dense layer (the gate stays on the FFN side). The FFN role builds the native DeepseekV2MLP (afd_plugin/model_executor/models/deepseek_v2.py:554-561).

Step 4: RemoteFFNProxy sends, yields, then receives#

The remote FFN proxy is a parameter-free module (afd_plugin/model_executor/models/deepseek_v2.py:117-125). Its _send_and_receive is the one place where the attention role crosses to the FFN role inside a layer.

    def _send_and_receive(
        self,
        hidden_states: torch.Tensor,
        **send_kwargs: torch.Tensor,
    ) -> torch.Tensor:
        afd_metadata = get_afd_metadata_from_forward_context()
        if afd_metadata is None:
            raise RuntimeError("RemoteFFNProxy requires AFD forward metadata")
        forward_context = get_forward_context()
        stage_idx = int(
            getattr(forward_context, "ubatch_idx", afd_metadata.stage_idx),
        )
        afd_metadata.stage_idx = stage_idx
        metadata = AFDTransferMetadata.create_attention_metadata(
            layer_idx=self.layer_idx,
            stage_idx=stage_idx,
            seq_len=int(hidden_states.shape[0]),
        )
        context = AFDTransferContext(metadata=metadata)
        afd_metadata.connector.send_attn_output(
            hidden_states,
            context,
            **send_kwargs,
        )
        hidden_states = maybe_apply_dbo_yield(
            hidden_states,
            role="attention",
        )
        return afd_metadata.connector.recv_ffn_output(
            ref_tensor=hidden_states,
            ubatch_idx=stage_idx,
        )
_send_and_receive: read the AFD metadata, send, yield for DBO, then receive.
  1. Lines 132-139. Read the AFD forward metadata and the stage index from the vLLM forward context.
  2. Lines 140-145. Build the transfer metadata for this layer and stage, with the row count equal to the number of tokens.
  3. Line 146. send_attn_output sends the hidden states to the FFN side.
  4. Line 151. maybe_apply_dbo_yield yields to the other microbatch under DBO.
  5. Line 155. recv_ffn_output receives the FFN result into the same tensor and returns it.

Step 5: Weight loading keeps only this role's checkpoint paths#

Both roles read the same checkpoint, but each keeps only its own weights. The causal-LM wrapper filters the weight stream before the native loader sees it (afd_plugin/model_executor/models/deepseek_v2.py:964-974).

    def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
        return super().load_weights(
            _iter_role_weights(
                weights,
                role=self.afd_role,
                config=self.config,
                compute_gate_on_attention=self.afd_config.compute_gate_on_attention,
                attention_shared_experts=self.afd_config.connector
                == AFD_ASYNC_CONNECTOR,
            )
        )
load_weights wraps the weight iterator with a role filter.

The filter is _iter_role_weights, which calls _checkpoint_weight_roles for each weight path and keeps a weight only when this role owns it (afd_plugin/model_executor/models/deepseek_v2.py:98-114).

def _checkpoint_weight_roles(
    name: str,
    config: _DeepseekAdapterConfig,
    *,
    compute_gate_on_attention: bool,
    attention_shared_experts: bool = False,
) -> frozenset[str]:
    """Classify one native checkpoint path by its AFD execution owner."""
    layer_path = _weight_layer_path(name)
    if layer_path is None:
        return _BOTH_ROLES

    layer_idx, stage, remainder = layer_path
    if stage == "self_attn":
        return _ATTENTION_ROLE
    if stage != "mlp":
        return _BOTH_ROLES

    if not _is_moe_layer(config, layer_idx):
        return _ATTENTION_ROLE if compute_gate_on_attention else _FFN_ROLE

    if attention_shared_experts and remainder and remainder[0] == "shared_experts":
        return _ATTENTION_ROLE
    is_moe_gate = bool(remainder) and remainder[0] == "gate"
    if is_moe_gate and compute_gate_on_attention:
        return _BOTH_ROLES
    return _FFN_ROLE
_checkpoint_weight_roles: which role owns each decoder weight.
  1. Lines 82-83. A self_attn weight belongs to the attention role.
  2. Lines 84-85. A weight that is not under mlp belongs to both roles (for example embeddings).
  3. Lines 87-88. A dense-layer mlp weight follows the gate flag: attention side when the gate runs there, otherwise the FFN side.
  4. Lines 92-95. A MoE gate weight goes to both roles when the gate runs on the attention side; the other expert weights go to the FFN role.

Deep dive#

Deep dive

The other three model families, one at a time

Four wrappers follow the same pattern. The differences are the native base model, the device support, and the extra payloads.

Qwen3-MoE (afd_plugin/model_executor/models/qwen3_moe.py:102). The wrapper reuses the native decoder_layer_type hook instead of rebuilding make_layers (afd_plugin/model_executor/models/qwen3_moe.py:198-202). It supports CUDA only and requires compute_gate_on_attention=false, so the gate always runs on the FFN side (afd_plugin/model_executor/models/qwen3_moe.py:82-87). On the attention role, every layer (dense or MoE) uses a plain RemoteFFNProxy (afd_plugin/model_executor/models/qwen3_moe.py:157); there is no remote-experts shell. The FFN role builds the native Qwen3MoeSparseMoeBlock or Qwen3MoeMLP (afd_plugin/model_executor/models/qwen3_moe.py:167-180).

Qwen3.5 / 3.6 MoE. The code registers the architecture Qwen3_5MoeForConditionalGeneration as AFDQwen3_5MoeForConditionalGeneration (afd_plugin/model_executor/models/qwen3_5.py:444). It supports CUDA only (afd_plugin/model_executor/models/model_utils.py:34-38) and text-only execution; a multimodal config raises (afd_plugin/model_executor/models/qwen3_5.py:22-29). It uses a remote-experts shell built on the native Qwen3NextSparseMoeBlock (afd_plugin/model_executor/models/qwen3_5.py:88-90), and reuses AFDAttentionFusedMoE from the DeepSeek wrapper (afd_plugin/model_executor/models/qwen3_5.py:15).

DeepSeek-V4 (CUDA wrapper, afd_plugin/model_executor/models/deepseek_v4.py:574-578). It supports CUDA only and the synchronous P2P connector (afd_plugin/model_executor/models/deepseek_v4.py:392-397). Its hash-router MoE needs the token ids, so the wrapper sets afd_requires_input_ids = True (afd_plugin/model_executor/models/deepseek_v4.py:578) and its proxy sends the input_ids alongside the hidden states (afd_plugin/model_executor/models/deepseek_v4.py:108-112). The mHC residual state stays on the attention role (afd_plugin/model_executor/models/deepseek_v4.py:175-178).

Deep dive

The NPU wrappers (named here, covered on the overview page)

The model table maps DeepseekV4ForCausalLM to the NPU wrapper only when torch_npu is importable; otherwise it maps to the CUDA wrapper (afd_plugin/__init__.py:109-113). The NPU DeepSeek-V4 wrapper is afd_plugin/model_executor/models/npu/deepseek_v4.py (afd_plugin/model_executor/models/npu/deepseek_v4.py:649). The DeepSeek V2/V3 NPU gate helpers live next to it (afd_plugin/model_executor/models/npu/deepseek_v2_attention_gate.py:31-80). The Ascend NPU overview: the same contract, different transports page reads the NPU code; this page covers the CUDA path.

Deep dive

Worked example: which modules load for DeepSeek-V2-Lite

Take DeepSeek-V2-Lite with the GPU connector and compute_gate_on_attention=false. This model has 27 decoder layers, with the first layer dense and the rest MoE. The 27 and the dense/MoE split are inferred from the DeepSeek-V2-Lite checkpoint config; they are not stored in either repository.

  • Attention role. Each layer builds a native attention module. Layer 0 (dense) and every MoE layer use the branches from Step 2: layer 0 gets a RemoteFFNProxy, each MoE layer gets the remote-experts shell on CUDA. No expert weights load, because _checkpoint_weight_roles sends them to the FFN role.
  • FFN role. Each self_attn is a PPMissingLayer, so no attention weights load. Layer 0 builds the native dense MLP; each MoE layer builds the native MoE block with its experts. The attention and embedding weights do not load.
  • Shared weights. The embeddings, the final norm and the LM head are not under a decoder mlp or self_attn, so _checkpoint_weight_roles returns both roles (afd_plugin/model_executor/models/deepseek_v2.py:78-79).

What can go wrong#

Why it matters. The role filter and the branch set are what let two processes load one checkpoint without duplicating the large weights. A wrong branch would load the wrong half, or run the FFN in the wrong process.

Check yourself#

  1. 1
    On the FFN role, what is self_attn?
    easy
    Show answer

    A PPMissingLayer: a placeholder with no weights (afd_plugin/model_executor/models/deepseek_v2.py:520). The FFN role does no attention.

  2. 2
    Which model family must run the gate on the FFN side?
    easy
    Show answer

    Qwen3-MoE. The wrapper requires compute_gate_on_attention=false (afd_plugin/model_executor/models/qwen3_moe.py:84-87).

  3. 3
    On the attention role, which class does a CUDA MoE layer build for mlp, and why not GateOnlyRemoteMoE?
    medium
    Show answer

    AFDDeepseekV2RemoteExpertsMoE (afd_plugin/model_executor/models/deepseek_v2.py:493-500). On CUDA, uses_remote_experts is true for a MoE layer, so the first branch is taken; GateOnlyRemoteMoE is the NPU path (afd_plugin/model_executor/models/deepseek_v2.py:444).

  4. 4
    How does the attention role avoid loading the expert weights?
    medium
    Show answer

    load_weights wraps the weight stream in _iter_role_weights, which keeps a weight only when the role owns it. Expert weights map to the FFN role (afd_plugin/model_executor/models/deepseek_v2.py:964-974, afd_plugin/model_executor/models/deepseek_v2.py:92-95).

  5. 5
    Why does DeepSeek-V4 set afd_requires_input_ids, and where does the value change the run?
    hard
    Show answer

    Its hash-router MoE needs the token ids to route (afd_plugin/model_executor/models/deepseek_v4.py:578). The attention proxy sends the ids (afd_plugin/model_executor/models/deepseek_v4.py:108-112), and the FFN runner reads the flag to request them from the connector (afd_plugin/v1/worker/ffn_model_runner.py:221).