AFD Plugin Explained

FAQ

IntermediateReferenceabout 6 min read

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

In one sentence

Short answers to common questions about afd-plugin, each with a citation. Longer answers link to the concept page that covers the topic.

Where do docs and code disagree?#

The design docs under docs/design/module/ were written against an older plugin commit, so their line numbers are stale. These pages cite the code at the pinned commit. The differences below are the ones a reader is most likely to meet.

The docs sayThe code doesSource
The README says both roles "require VLLM_USE_V2_MODEL_RUNNER=1" (README.md:204). vLLM picks Model Runner V2 by default when the variable is unset. The validator raises only when V2 resolves to off. The recipes still set it to 1 to be explicit. vllm:vllm/config/vllm.py:692-694, afd_plugin/validation.py:90-91
The README model table names "Qwen3.5 / Qwen3.6 MoE" (README.md:51). One architecture string is registered: Qwen3_5MoeForConditionalGeneration, mapped to AFDQwen3_5MoeForConditionalGeneration. afd_plugin/__init__.py:96-131
The design docs cite plugin commit 36dadb4 (docs/design/module/index.md:38). The plugin is pinned at a later commit. Line numbers in the design docs do not match the current code; read the code. 7452d52
Every other connector config key is snake_case. The async CAM connector accepts a camelCase wire key dynamicQuant for its field dynamic_quant. afd_plugin/connectors/npu/async_cam.py:182-184
The DeepSeek-V2-Lite README benchmark step runs bash tools/benchmarks/benchmark.sh (recipe/gpu/P2pNcclAFDConnector/deepseek_v2_lite/README.md:111). No tools/benchmarks/benchmark.sh exists at the pinned commit. The benchmark harness is tools/benchmarks/request_generator.sh, a wrapper around vllm bench serve. tools/benchmarks/request_generator.sh:1-27
The decode-bench script header points to tools/benchmarks/2a2f_graph_dbo_dp2tp1_decode_bench.sh and tools/benchmarks/vllm_bench.sh (tools/benchmarks/decode_bench_server.sh:26-30). Neither file exists at the pinned commit. The present scripts are decode_bench_server.sh and request_generator.sh. tools/benchmarks/

Known open questions#

These facts are not in either repository. The pages that need them mark them as inferred.

  • DeepSeek-V2-Lite model shape. The worked examples use hidden size 2048, dtype bf16, and 27 decoder layers (layer 0 dense). These come from the DeepSeek-V2-Lite checkpoint config, not from afd-plugin or vLLM. A unit test fixture uses a synthetic 40-layer value, which is not the real model (tests/unit/connectors/test_p2p_connector.py:136).
  • The 32,768-byte per-layer transfer. The figure 8 × 2048 × 2 bytes on the connector pages is derived from the inferred hidden size and bf16, for one decode step of 8 tokens. It is an example, not a repository constant.
  • v0.30 NPU hardware validation. The recipe index lists both NPU connectors as "integrated v0.30 hardware validation pending"; the measured numbers are from older runtimes (recipe/README.md:36-38). See Ascend NPU overview: the same contract, different transports.

Practical questions#

Does AFD change the vLLM source?#

No. AFD is a vLLM general plugin. It registers models, a worker class, a routing strategy and a DBO op, and it installs monkey-patches on vLLM functions at import. The vLLM checkout is unchanged (afd_plugin/__init__.py:142-216). See Compat patches: how AFD changes vLLM without editing it.

Why does the FFN server reject a request?#

The FFN worker is connector-driven, not scheduler-driven. If vLLM's scheduler ever calls the FFN worker's execute_model, it raises at once, so a misrouted request fails fast (afd_plugin/v1/worker/ffn_worker.py:122-131). Send requests only to the attention API server (README.md:258-260). See Attention and FFN roles: what each server owns.

Why does the FFN server have no scheduler?#

The FFN role runs no KV cache and no request scheduler. Its EngineCore returns after the executor is built, and it installs a no-op scheduler stand-in so the native readiness handshake still works (afd_plugin/compat/patches/engine_core.py:599-623). The connector receive loop drives the FFN side instead. See The FFN runtime: a connector-driven server loop.

Can the attention ranks be fewer than the FFN ranks (A < F)?#

No, not for the GPU connector. The P2P topology requires num_attention_ranks >= num_ffn_ranks, and raises otherwise (afd_plugin/distributed/topology.py:52-58). See Deployment topologies: how to place the roles.

How do the rank counts relate to DP and TP?#

For each role, the rank count must equal DP × TP: num_attention_ranks for the attention role, num_ffn_ranks for the FFN role. The validator raises when they do not match (afd_plugin/validation.py:107-117). See Deployment topologies: how to place the roles.

Which CUDA graph modes are allowed?#

On GPU, only FULL_DECODE_ONLY (afd_plugin/v1/worker/cuda_graph.py:20-21). The recipes pass --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY", ...}' for the graph variants. See CUDA graphs in AFD: FULL_DECODE_ONLY on both roles and Tuning: the knobs the repo exposes.

Where does sampling run?#

On the attention side. The attention runner does not override sample_tokens, so it samples natively in vLLM's call (vllm:vllm/v1/worker/gpu/model_runner.py:2048-2050). The FFN runner's sample_tokens raises (afd_plugin/v1/worker/ffn_model_runner.py:405-406). See Tour: Life of a request.

Why are there two vLLM version checks?#

The first check, at registration, warns on a version mismatch but does not stop (afd_plugin/compat/vllm.py:29-50). The second, at patch install, is a stricter gate that tolerates a dev suffix and installs the patches only on a matching version (afd_plugin/compat/vllm.py:53-68). See Compat patches: how AFD changes vLLM without editing it.

What happens on a different vLLM version?#

The patch gate fails, so no patch installs and worker_cls is never rewritten; AFD does not take effect. The registration still adds the models, the routing strategy and the DBO op (afd_plugin/__init__.py:162-216). See Reference: compat patches.

Which models does AFD support?#

DeepSeek V2/V3/V3.2, GLM-style DeepSeek, Qwen3-MoE and the Qwen3.5 family, each registered as an AFD<arch> wrapper (afd_plugin/__init__.py:96-131). DeepSeek-V4 and the NPU wrappers are device-gated. See Model registration: the AFD<arch> wrappers.

Which connector is the default, and on which device?#

The default is P2pNcclAFDConnector, the GPU connector (afd_plugin/config.py:52). The two NPU connectors are CAMP2pAFDConnector (sync) and CAMAsyncAFDConnector (async). See The connector contract: the Attention/FFN transport and Ascend NPU overview: the same contract, different transports.

How do I benchmark a recipe?#

Start a recipe, then drive load against the attention HTTP port with tools/benchmarks/request_generator.sh, a wrapper around vllm bench serve (tools/benchmarks/request_generator.sh:44-57). For a Kubernetes run, the repository ships a run-vllm-bench-k8s contributor skill (.agents/skills/run-vllm-bench-k8s/SKILL.md:1-6). See Tuning: the knobs the repo exposes.

Why two servers for one model?#

AFD splits the model into an attention role and an FFN role, each a separate vllm serve process. The attention side owns the request and the KV cache; the FFN side runs the feed-forward and MoE experts and answers over the connector (README.md:258-260). See What is AFD? and Attention and FFN roles: what each server owns.

  • Attention and FFN roles: what each server ownsAFD runs the attention part and the FFN part as two server processes. This page shows what each role owns, and why the FFN side has no real scheduler, no KV cache and no sampler.
  • Deployment topologies: how to place the rolesThe GPU recipes place the attention and FFN roles in five shapes: 2A2F and 4A4F colocation, 2P1A1F prefill/decode disaggregation with a proxy, a k8s multipod split, and the Qwen3.5 4A2F multipod. This page gives the process count, GPUs, ranks, flags, start order and ports of each.
  • Tuning: the knobs the repo exposesWhat afd-plugin lets you tune, and only what its code and READMEs state: the A:F rank ratio, DBO thresholds, CUDA graph capture, the group timeout, gate placement, the NPU CAM capacity, the profiler, the routing simulator and the benchmark tools. Every number is cited.
  • AFD config reference: the additional_config["afd"] keysEvery key of --additional-config "afd", with its alias, type, default and validation, plus the connector_extra_config keys per connector and a worked example from a real recipe.
  • Reference: compat patchesEvery AFD compat patch, its vLLM target at the pinned commit, how and when it installs, what it changes, and the page that explains it. Hand-checked, one row per replaced function.