Deployment topologies: how to place the roles

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

The idea#

AFD splits one model into an attention role and an FFN role. Each role is a separate server process. A topology name xAyF gives the counts: x attention ranks and y FFN ranks (xAyF). The GPU recipes ship five shapes of this split.

Two rules fix the rank counts. For each role, the rank count must equal DP × TP: the attention ranks equal the attention server's data-parallel size times its tensor-parallel size, and the same for FFN (afd_plugin/validation.py:107-117). The attention ranks must also be at least the FFN ranks, A >= F (afd_plugin/distributed/topology.py:52-58).

An everyday comparison. Think of a kitchen with a front counter and a back kitchen. The counter takes orders (attention); the kitchen cooks (FFN). You can add counters or cooks, but you need at least as many counters as cooks, and each team's size is set when you open. The comparison stops at the wire: the two teams talk only through the connector.

The picture#

Read the three panels top to bottom. Panel 1 is the common case, two servers on one host. Panel 2 adds a separate prefill stage and a proxy. Panel 3 splits the same roles across k8s pods. The blue boxes are the attention role, the emerald boxes the FFN role.

In the code#

Step 1: The rank rule (A = DP x TP, F = DP x TP)#

The validator reads the role's rank count and compares it with DP × TP of that server. A mismatch raises before the run starts (afd_plugin/validation.py:107-117).

    configured_ranks = (
        afd_config.num_attention_ranks
        if expected_role == "attention"
        else afd_config.num_ffn_ranks
    )
    distributed_ranks = parallel.data_parallel_size * parallel.tensor_parallel_size
    if configured_ranks != distributed_ranks:
        raise RuntimeError(
            f"AFD ModelRunnerV2 {expected_role} ranks must match DP * TP: "
            f"configured={configured_ranks}, distributed={distributed_ranks}",
        )

So a 2A2F topology has two ways to reach two attention ranks: DP1 TP2 or DP2 TP1. Both give 2 = DP × TP. The Topology and role ranks: the AFD world page covers how a role rank maps to a world rank.

Step 2: Colocation 2A2F (two servers, four GPUs)#

The simplest recipe runs one attention server on GPUs 0 and 1, and one FFN server on GPUs 2 and 3. Each server sets num_attention_ranks: 2 and num_ffn_ranks: 2, so the topology is 2A2F (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:7-32).

CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --additional-config '{
        "afd": {
            "role": "attention",
            "connector": "P2pNcclAFDConnector",
            "host": "127.0.0.1",
            "port": 6269,
            "num_attention_ranks": 2,
            "num_ffn_ranks": 2
        }
    }' \

The DP/TP split is a choice. The dp1tp2 variant uses DP1 TP2 (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:8-9); the dp2tp1 variant uses DP2 TP1 (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_eager_dbo_dp2tp1.sh:8-9). Both give two ranks per role.

Step 3: Colocation 4A4F (double both roles)#

The 4A4F recipe doubles both sides: four attention ranks on GPUs 0-3, four FFN ranks on GPUs 4-7, each server DP2 TP2 (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/4a4f_graph_dbo_dp2tp2.sh:7-18).

CUDA_VISIBLE_DEVICES=0,1,2,3 uv run vllm serve "$MODEL_PATH" \
    --data-parallel-size 2 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --additional-config '{
        "afd": {
            "role": "attention",
            "connector": "P2pNcclAFDConnector",
            "host": "127.0.0.1",
            "port": 6269,
            "num_attention_ranks": 4,
            "num_ffn_ranks": 4

Step 4: Disaggregation 2P1A1F (prefillers and a proxy)#

The disaggregation recipe adds a prefill stage. It runs five processes: two prefillers, one attention decode server, one FFN decode server, and a proxy (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:40-50). The attention and FFN servers each use 1A1F (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_disaggregation/2p1a1f_eager_dbo.sh:56-98).

# ----- Disaggregation proxy (client-facing endpoint, HTTP 18305) -----
# vLLM-native NIXL load-balance proxy. For each request it prefills on a
# prefiller (do_remote_decode), reads back the prefiller's NIXL handshake
# metadata (remote_engine_id / remote_block_ids / remote_host / remote_port) in
# kv_transfer_params, then hands it to the decoder with do_remote_prefill so the
# decoder pulls the KV. Send benchmark/client traffic to 18305, NOT to 18303.
# Wait for the prefill/attn/ffn servers to print "Application startup complete"
# before this proxy starts accepting traffic.
uv run python "$SCRIPT_DIR/../../../../../tools/proxy_server.py" \
    --host 127.0.0.1 \
    --port 18305 \
    --prefiller-hosts 127.0.0.1 127.0.0.1 \
    --prefiller-ports 18301 18302 \
    --decoder-hosts 127.0.0.1 \
    --decoder-ports 18303 \
    > proxy.log 2>&1

Step 5: Multipod on Kubernetes#

The multipod recipes put one role per pod. Each vllm serve is wrapped in an if [ "$POD" == ... ] block, so with POD unset neither block runs (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh:15-16). These scripts are for the deploy-afd-k8s skill, not for a local run (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:62-69).

AFD_CONNECTOR_HOST=${AFD_CONNECTOR_HOST:-vllm-ffn-p2p-service}
AFD_CONNECTOR_PORT=${AFD_CONNECTOR_PORT:-6269}

if [ "$POD" == "ATTENTION_0" ]; then
  CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \

The Qwen3.5 122B FP8 recipe is a three-pod 4A2F split: two attention pods and one FFN pod. The second attention pod sets ATTENTION_HEADLESS=1 and ATTENTION_DP_START_RANK=2 (recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh:6-8). Each attention pod runs --data-parallel-size 4 with --data-parallel-size-local 2 (recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh:33-34). So the two pods together hold the four attention ranks: global DP4, two local ranks per pod, with the second pod starting at rank 2 (recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh:27-28). The FFN pod runs DP2 (recipe/gpu/P2pNcclAFDConnector/qwen3_5_122b_a10b_fp8/prefill_decode_colocation/multipod_2a_2a_2f_graph.sh:70).

Deep dive#

What can go wrong#

• Rank count does not match DP x TP. The validator raises when num_attention_ranks != DP × TP for the attention server, or the same for FFN (afd_plugin/validation.py:107-117).
• A < F. The GPU connector requires num_attention_ranks >= num_ffn_ranks and raises otherwise (afd_plugin/distributed/topology.py:52-58).
• A multipod script run locally. With POD unset, neither serve block runs, so nothing starts (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/multipod_2a_2f_graph_dbo_dp1tp2.sh:15-16).
• A host or port mismatch between the roles. Both roles must use the same AFD host:port, or the group setup hangs until the timeout (afd_plugin/connectors/gpu/p2p.py:271).

Why it matters. The topology fixes the process count, the GPU count and the rank mapping. A wrong rank count or a wrong host stops the run before the first token.