Tour: Run a recipe

1Turn the V2 runner on#

The script exports one environment variable before it starts a server (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:5).

MODEL_PATH=${MODEL_PATH:-/path/model_weights/DeepSeek-V2-Lite}
export VLLM_USE_V2_MODEL_RUNNER=1

AFD runs on Model Runner V2. Setting VLLM_USE_V2_MODEL_RUNNER=1 turns the V2 runner on explicitly. vLLM also picks V2 by default when the variable is unset (vllm:vllm/config/vllm.py:692-694); the recipe sets it so the choice is clear. The validator raises only when V2 resolves to off (afd_plugin/validation.py:90-91).

2The attention serve command and the parallel sizes#

The attention server runs on GPUs 0 and 1. The parallel flags set the per-role rank count (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:7-10).

CUDA_VISIBLE_DEVICES=0,1 uv run vllm serve "$MODEL_PATH" \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \

• CUDA_VISIBLE_DEVICES=0,1 picks the two GPUs for this role.
• --data-parallel-size 1 --tensor-parallel-size 2 gives two attention ranks, because the rank count must equal DP × TP (afd_plugin/validation.py:107-117).
• --enable-expert-parallel turns on static expert parallelism, which AFD requires (afd_plugin/validation.py:118-125).

See Deployment topologies: how to place the roles for the rank rule.

3The afd block in --additional-config#

The --additional-config JSON carries the AFD settings under the "afd" key (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:11-20).

    --additional-config '{
        "afd": {
            "role": "attention",
            "connector": "P2pNcclAFDConnector",
            "host": "127.0.0.1",
            "port": 6269,
            "num_attention_ranks": 2,
            "num_ffn_ranks": 2
        }
    }' \

The plugin reads this inner mapping and builds an AFDConfig (afd_plugin/config.py:42). The role is attention here; the connector is the GPU P2P connector; host and port set the rendezvous where the two roles meet; num_attention_ranks and num_ffn_ranks are both 2. See AFD config reference: the additional_config["afd"] keys.

4The batch, DBO and CUDA graph flags#

The next flags set the batch limits, DBO and the CUDA graph (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:21-29).

    --max-num-seqs 64 \
    --max-num-batched-tokens 64 \
    --enable-dbo \
    --dbo-decode-token-threshold 2 \
    --dbo-prefill-token-threshold 12 \
    --max-cudagraph-capture-size 64 \
    --compilation-config '{
        "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64]
    }' \

• --enable-dbo with --dbo-decode-token-threshold 2 and --dbo-prefill-token-threshold 12 turns on DBO with low example thresholds. AFD needs exactly two microbatches when DBO is on (afd_plugin/validation.py:126-131). See Dual-batch overlap (DBO): two stages per step.
• --max-cudagraph-capture-size 64 and the --compilation-config with cudagraph_mode: FULL_DECODE_ONLY turn on CUDA graphs. GPU AFD allows only FULL_DECODE_ONLY (afd_plugin/v1/worker/cuda_graph.py:20-21). See CUDA graphs in AFD: FULL_DECODE_ONLY on both roles.

5The HTTP port and backgrounding#

The attention server binds an HTTP port and runs in the background (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:30-32).

    --host 127.0.0.1 \
    --port 18305 \
    --trust-remote-code > attn.log 2>&1 &

Clients connect to this HTTP port, 18305. It is a different port from the AFD 6269: the HTTP port carries requests, the AFD port carries the connector setup. The command ends with &, so it runs in the background.

6The FFN server: the same block, role ffn#

The FFN server is the same command with two changes: GPUs 2 and 3, and role: ffn (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:34-59).

CUDA_VISIBLE_DEVICES=2,3 uv run vllm serve "$MODEL_PATH" \
    --data-parallel-size 1 \
    --tensor-parallel-size 2 \
    --enable-expert-parallel \
    --additional-config '{
        "afd": {
            "role": "ffn",
            "connector": "P2pNcclAFDConnector",
            "host": "127.0.0.1",
            "port": 6269,
            "num_attention_ranks": 2,
            "num_ffn_ranks": 2
        }
    }' \

The FFN server reads the same host, port and rank counts, so the two roles meet at the same rendezvous. It runs no KV cache and no real scheduler, only a no-op stand-in; the connector drives it (afd_plugin/compat/patches/engine_core.py:599-623). Send no client traffic here. See Attention and FFN roles: what each server owns and The FFN runtime: a connector-driven server loop.

7Start order: background both, then wait#

The script backgrounds both servers and ends with wait (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:53-61).

    --trust-remote-code > ffn.log 2>&1 &

wait

Both roles start together and meet at the AFD rendezvous. Wait for Application startup complete in each log before you send traffic (recipe/gpu/P2pNcclAFDConnector/README.md:39-43).

export MODEL_PATH=/path/model_weights/DeepSeek-V2-Lite
export VLLM_USE_V2_MODEL_RUNNER=1
bash recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh

8Send a request, then benchmark it#

Send requests only to the attention HTTP server (README.md:258-260). The FFN server has no request path; a request there fails fast (afd_plugin/v1/worker/ffn_worker.py:122-131).

curl http://127.0.0.1:18305/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "'"$MODEL_PATH"'", "prompt": "Hello", "max_tokens": 16}'

To benchmark, drive load against the same port with the repository's wrapper around vllm bench serve (tools/benchmarks/request_generator.sh:44-57). The README shows the benchmark command for this model (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:104-120). See Tuning: the knobs the repo exposes.