Tuning: the knobs the repo exposes

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

The idea#

Tuning AFD means three things: shape the split and the overlap for throughput, set the timeouts and gate for reliable startup, and measure with the profiler and benchmark tools. The repository gives example values for some knobs and says to tune them for the workload. This page uses only numbers the code or the READMEs state.

An everyday comparison. Think of the knobs on an audio mixer. Each one changes one thing, and the factory marks are a starting point, not the final mix. The comparison stops at the meters: here the meters are the profiler and the benchmark, not your ear.

The picture#

Read the three panels as three goals. Panel 1 shapes throughput; panel 2 keeps startup reliable; panel 3 measures. Each card gives the key or flag and the file where the code reads it.

In the code#

Step 1: The A:F rank ratio#

The attention ranks must be at least the FFN ranks. The P2P topology check raises when A < F (afd_plugin/distributed/topology.py:52-58). Within that rule you choose the counts and the DP/TP split per role (afd_plugin/validation.py:107-117). See Deployment topologies: how to place the roles.

Step 2: DBO thresholds#

DBO microbatches a step when the token count passes a threshold. The recipes set a low decode threshold 2 and prefill threshold 12 (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:24-25). The vLLM defaults are higher: 32 for decode and 512 for prefill (vllm:vllm/config/parallel.py:227, vllm:vllm/config/parallel.py:232).

    --enable-dbo \
    --dbo-decode-token-threshold 2 \
    --dbo-prefill-token-threshold 12 \

vLLM reads these thresholds per step to decide whether to microbatch (vllm:vllm/v1/worker/ubatch_utils.py:141-149). AFD requires exactly two microbatches when DBO is on (afd_plugin/validation.py:126-131). See Dual-batch overlap (DBO): two stages per step.

Step 3: CUDA graph capture size#

The graph variants set --max-cudagraph-capture-size and the cudagraph_capture_sizes list together, with cudagraph_mode: FULL_DECODE_ONLY (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/prefill_decode_colocation/2a2f_graph_dbo_dp1tp2.sh:26-29).

    --max-cudagraph-capture-size 64 \
    --compilation-config '{
        "cudagraph_mode": "FULL_DECODE_ONLY", "cudagraph_capture_sizes":[64]
    }' \

On GPU, AFD allows only FULL_DECODE_ONLY (afd_plugin/v1/worker/cuda_graph.py:20-21). The README says to set --max-cudagraph-capture-size and cudagraph_capture_sizes together, based on the batch sizes and GPU memory (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:154-157). See CUDA graphs in AFD: FULL_DECODE_ONLY on both roles.

Step 4: The group timeout#

afd_process_group_timeout_s sets how long the world process-group rendezvous waits; the default is 120 seconds (afd_plugin/config.py:70). The P2P connector reads it when it builds the world group (afd_plugin/connectors/gpu/p2p.py:275). The config comment says the right value depends on rank count, placement and model size, so it is left to the caller (afd_plugin/config.py:67-70).

Step 5: Gate placement#

compute_gate_on_attention moves the MoE gate to the attention side. When it is on, the FFN side runs only the expert layers (afd_plugin/v1/worker/ffn_model_runner.py:211-215). It defaults to false (afd_plugin/config.py:66). Not every model allows it: the Qwen3-MoE wrapper requires it to be false (afd_plugin/model_executor/models/qwen3_moe.py:84-87). See The MoE handoff: where the gate runs.

Deep dive#

What can go wrong#

• DBO on without two microbatches. AFD requires exactly two microbatches when DBO is on, and raises otherwise (afd_plugin/validation.py:126-131).
• A graph mode other than FULL_DECODE_ONLY on GPU. The GPU runner accepts only FULL_DECODE_ONLY (afd_plugin/v1/worker/cuda_graph.py:20-21).
• A timeout too short for the placement. The rendezvous fails when the world group does not form in time (afd_plugin/connectors/gpu/p2p.py:275).
• The routing simulator left on for serving. It changes model outputs, so it gives wrong answers outside a benchmark (afd_plugin/model_executor/routing_simulator.py:10-11).

Why it matters. The knobs trade throughput, memory and startup time. The repository gives example values and the read sites; the right values come from measuring your own workload.