AFD Plugin Explained

What is AFD?

BeginnerLearn the basicsabout 3 min read + 2 min deep dives

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

In one sentence

AFD runs the attention part and the feed-forward part of one model in separate server processes, joined by a connector.

The idea#

A transformer decoder layer has two heavy parts. Attention reads the KV cache and mixes tokens. The feed-forward part (dense FFN, or MoE experts) applies big weight matrices. The two parts are slow for different reasons.

Attention is memory-bound. Each request owns its KV cache, and the step reads a lot of memory for little compute. The FFN and the MoE experts are weight-bound. They reuse the same weights for every token, so a large batch makes them efficient. One batch size and one rank count cannot suit both parts at once.

Attention-FFN Disaggregation splits the two parts into two groups of processes. The attention role keeps the KV cache and the API server. The FFN role holds the experts. A connector moves hidden states from the attention side to the FFN side and back. Each role can then use its own rank count and its own batch shape.

An everyday comparison. Think of a kitchen with a prep station and an oven. Prep works per order; the oven works best full. You staff them apart so each runs at its own pace. The comparison stops here: the two AFD roles must still finish every layer together, once per step, before the next layer starts.

The picture#

Figure: the two parts have different bottlenecks (panel 1), so AFD gives each its own server group and joins them with a connector (panel 2).
Open full size ↗

Read panel 1 as the problem: two parts, two bottlenecks. Read panel 2 as the answer: two server groups, joined by a connector. The attention side sends hidden states and gets the FFN output back, once per layer.

Deep dive#

These blocks compare AFD with two layouts the reader may know, and explain the xAyF topology name.

Deep dive

AFD vs. prefill/decode disaggregation

Prefill/decode disaggregation and AFD both spread one model over more processes, but they cut it in different directions.

  • Prefill/decode disaggregation cuts by phase. A prefill server runs the whole model for the prompt; a decode server runs the whole model for the new tokens. Each process still holds attention and FFN together.
  • AFD cuts by model part. Inside one phase, attention runs on one group of processes and the FFN runs on another.

The two layouts stack. The plugin ships a recipe that combines them: a 2p1a1f topology with two prefill workers, one attention decode worker and one FFN decode worker (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:27, recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:40-50). A proxy server joins the prefill and decode stages (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:50).

Deep dive

AFD vs. expert parallelism (EP)

Expert parallelism spreads the MoE experts of one layer over several ranks. It is a way to place the experts; it does not move attention to another process. Attention and the experts still run inside the same worker and the same step.

AFD is a process split, not an expert placement. It puts attention and the FFN in separate processes. Expert parallelism can still run inside the FFN role to spread its experts. The GPU runtime in fact needs static expert parallelism on the AFD stack, and rejects elastic EP or EPLB (afd_plugin/validation.py:118-125). So AFD and EP are not alternatives: AFD places the two parts, and EP places the experts within the FFN part.

Deep dive

The xAyF topology name and the rank-count rule

A topology is named xAyF: x attention ranks and y FFN ranks. For example 2A2F is two of each. The counts are two AFD config fields, num_attention_ranks and num_ffn_ranks (afd_plugin/config.py:61-64).

The GPU P2P connector sets a rule: the number of attention ranks must be greater than or equal to the number of FFN ranks. The world places the FFN ranks first, then the attention ranks, and spreads the attention ranks over the FFN ranks in blocks that differ in size by at most one (afd_plugin/connectors/gpu/p2p.py:11-20, README.md:59). So one FFN rank can serve more than one attention peer, but not the other way around. The Topology and role ranks: the AFD world page gives the full mapping.

In afd-plugin#

afd-plugin adds AFD to vLLM v0.30.0 without changing the vLLM source. Three pointers connect this page to the code:

Check yourself#

  1. 1
    Why does AFD put attention and the FFN in separate processes?
    easy
    Show answer

    The two parts are slow for different reasons. Attention is memory-bound per request; the FFN and the experts are weight-bound and want a large batch. Separate processes let each part use its own rank count and batch shape.

  2. 2
    What does the name 2A2F mean?
    easy
    Show answer

    Two attention ranks and two FFN ranks. The counts are num_attention_ranks and num_ffn_ranks (afd_plugin/config.py:61-64).

  3. 3
    How does AFD differ from prefill/decode disaggregation?
    medium
    Show answer

    Prefill/decode disaggregation cuts by phase; each process still runs the whole model. AFD cuts by model part: attention and the FFN run in separate processes. The 2p1a1f recipe combines both (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:40-50).

  4. 4
    Is AFD an alternative to expert parallelism?
    medium
    Show answer

    No. AFD places the two model parts in separate processes. Expert parallelism places the experts inside the FFN part. The GPU stack needs static expert parallelism (afd_plugin/validation.py:118-125).

  5. 5
    On the GPU P2P connector, why must num_attention_ranks be at least num_ffn_ranks?
    hard
    Show answer

    The connector maps each attention rank to one FFN peer, and lets one FFN rank serve several attention ranks in near-equal blocks. That only works when there are at least as many attention ranks as FFN ranks (afd_plugin/connectors/gpu/p2p.py:11-20).

  • Attention and FFN roles: what each server ownsAFD runs the attention part and the FFN part as two server processes. This page shows what each role owns, and why the FFN side has no real scheduler, no KV cache and no sampler.
  • Topology and role ranks: the AFD worldThe P2P connector builds one world, [F0..A0..], from the role ranks. This page shows how the role rank, the subgroups and the DP-metadata routes come from the topology sizes.
  • The vLLM v1 mapAFD hooks into a short chain of vLLM v1 processes and classes: the API server, the EngineCore, the executor, the worker, the Model Runner V2 and the model. This page names each box, with its vLLM line and the AFD hook on it.