AFD Plugin Explained

Environment variables reference

IntermediateReferenceabout 4 min read + 3 min deep dives

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

In one sentence

The plugin reads some settings from the environment, not from the AFD config. This page lists every one, with its default and read site.

How the plugin uses the environment#

Most AFD settings go through --additional-config (see AFD config reference: the additional_config["afd"] keys). A smaller set of settings comes from environment variables. The plugin uses them for build-time choices, the profilers, the Ascend operator loader, the benchmark routing strategy, and a few runtime switches. The AFD config values do not change these; you set them in the shell.

The table below is generated from the source, so the names, defaults and lines stay correct at the pinned commit. Run python3 tools/gen_env_vars.py to refresh it. The generator parses the Python with ast only; it never imports the plugin or vLLM.

Every variable the plugin reads or sets#

The Kind column says what the plugin does with the variable: reads it, sets it, or prepends a path to it. A default shows only where the read site gives one. Build-time variables (in setup.py) run at install time, not at serve time.

Generated by tools/gen_env_vars.py from afd_plugin/** and setup.py. Each variable is listed with its default (when the read site gives one), whether the plugin reads, sets or prepends it, and every source line. Do not edit between the markers.

51 environment variables.

VariableDefaultKindRead at
AFD_ASYNC_CAM_LAYERED_GMM''readsafd_plugin/envs.py:19
AFD_ASYNC_MOE_LAYOUT_LOG''readsafd_plugin/model_executor/models/npu/async_cam_layout.py:187
AFD_BENCHMARK_FORCE_LB_TOPN_PER_RANK'0'readsafd_plugin/model_executor/routing_simulator.py:127
AFD_BUILD_ASCEND_OPS—readssetup.py:44
AFD_CAM_OP_IO_LOG''readsafd_plugin/connectors/npu/async_cam.py:955
AFD_CUST_OPAPI_LIB_PATH—setsafd_plugin/compat/npu/ops.py:54
AFD_FORCE_BALANCED_TOPK_IDS''readsafd_plugin/envs.py:15
AFD_FORCE_SPAWN_MULTIPROCESSING—readsafd_plugin/__init__.py:60
AFD_GPU_ATTENTION_PROFILER_ACTIVE10readsafd_plugin/compat/profiler.py:54
AFD_GPU_ATTENTION_PROFILER_DIR'./profiler_logs/attn'readsafd_plugin/compat/profiler.py:60
AFD_GPU_ATTENTION_PROFILER_ENABLEFalsereadsafd_plugin/compat/profiler.py:51
AFD_GPU_ATTENTION_PROFILER_REPEAT1readsafd_plugin/compat/profiler.py:55
AFD_GPU_ATTENTION_PROFILER_SKIP_FIRST0readsafd_plugin/compat/profiler.py:56
AFD_GPU_ATTENTION_PROFILER_WAIT2500readsafd_plugin/compat/profiler.py:52
AFD_GPU_ATTENTION_PROFILER_WARMUP1readsafd_plugin/compat/profiler.py:53
AFD_GPU_FFN_PROFILER_ACTIVE10readsafd_plugin/compat/profiler.py:54
AFD_GPU_FFN_PROFILER_DIR'./profiler_logs/ffn'readsafd_plugin/compat/profiler.py:60
AFD_GPU_FFN_PROFILER_ENABLEFalsereadsafd_plugin/compat/profiler.py:51
AFD_GPU_FFN_PROFILER_REPEAT1readsafd_plugin/compat/profiler.py:55
AFD_GPU_FFN_PROFILER_SKIP_FIRST0readsafd_plugin/compat/profiler.py:56
AFD_GPU_FFN_PROFILER_WAIT2500readsafd_plugin/compat/profiler.py:52
AFD_GPU_FFN_PROFILER_WARMUP1readsafd_plugin/compat/profiler.py:53
AFD_NPU_ATTENTION_PROFILER_ACTIVE10readsafd_plugin/compat/npu/profiler.py:61
AFD_NPU_ATTENTION_PROFILER_DIR'/tmp/profile/attn'readsafd_plugin/compat/npu/profiler.py:67
AFD_NPU_ATTENTION_PROFILER_ENABLEFalsereadsafd_plugin/compat/npu/profiler.py:58
AFD_NPU_ATTENTION_PROFILER_REPEAT1readsafd_plugin/compat/npu/profiler.py:62
AFD_NPU_ATTENTION_PROFILER_SKIP_FIRST1500readsafd_plugin/compat/npu/profiler.py:63
AFD_NPU_ATTENTION_PROFILER_WAIT2readsafd_plugin/compat/npu/profiler.py:59
AFD_NPU_ATTENTION_PROFILER_WARMUP1readsafd_plugin/compat/npu/profiler.py:60
AFD_NPU_ATTENTION_PROFILER_WITH_STACKFalsereadsafd_plugin/compat/npu/profiler.py:68
AFD_NPU_FFN_PROFILER_ACTIVE20readsafd_plugin/compat/npu/profiler.py:61
AFD_NPU_FFN_PROFILER_DIR'/tmp/profile/ffn'readsafd_plugin/compat/npu/profiler.py:67
AFD_NPU_FFN_PROFILER_ENABLEFalsereadsafd_plugin/compat/npu/profiler.py:58
AFD_NPU_FFN_PROFILER_REPEAT1readsafd_plugin/compat/npu/profiler.py:62
AFD_NPU_FFN_PROFILER_SKIP_FIRST1500readsafd_plugin/compat/npu/profiler.py:63
AFD_NPU_FFN_PROFILER_WAIT2readsafd_plugin/compat/npu/profiler.py:59
AFD_NPU_FFN_PROFILER_WARMUP1readsafd_plugin/compat/npu/profiler.py:60
AFD_NPU_FFN_PROFILER_WITH_STACKFalsereadsafd_plugin/compat/npu/profiler.py:68
AFD_SKIP_ACLNN_BUILD'0'readssetup.py:66
ASCEND_CUSTOM_OPP_PATH—prependsafd_plugin/compat/npu/ops.py:48
ASCEND_HOME_PATH—readssetup.py:38 setup.py:98 setup.py:99
ASCEND_OPP_PATH—readssetup.py:38
ASCEND_TOOLKIT_HOME—readssetup.py:38
BATCH_SIZE_FACTOR'1'readsafd_plugin/compat/npu/feature_validation.py:247 afd_plugin/validation.py:52
CMAKE_BUILD_TYPE'Release'readssetup.py:85
LD_LIBRARY_PATH—prependsafd_plugin/compat/npu/ops.py:51
MAX_JOBS—readssetup.py:104
SOC_VERSION'910c'readssetup.py:67 setup.py:88
TORCH_NPU_PATH—readssetup.py:38 setup.py:100 setup.py:101
VLLM_ASCEND_ENABLE_FLASHCOMM1'0'readsafd_plugin/compat/npu/feature_validation.py:190 afd_plugin/connectors/npu/async_cam.py:427
VLLM_TORCH_PROFILER_DIR—readsafd_plugin/compat/npu/profiler.py:149 afd_plugin/compat/profiler.py:132

What each group does#

The generated table gives the default and the read site. This section gives the effect, with a citation. The variables fall into five groups.

Deep dive

Multiprocessing

Deep dive

Build-time (setup.py)

These run at install time, when you build the plugin. They decide whether to build the Ascend operators and how. They have no effect on a running server.

  • AFD_BUILD_ASCEND_OPS — forces the Ascend operator build on or off. When unset, the build auto-detects an Ascend platform (setup.py:43-54).
  • AFD_SKIP_ACLNN_BUILD — when not 1, the build runs csrc/npu/build_aclnn.sh (setup.py:66-71).
  • SOC_VERSION — the Ascend SoC version passed to the operator build; default 910c (setup.py:67, setup.py:88).
  • CMAKE_BUILD_TYPE — the CMake build type; default Release (setup.py:85).
  • ASCEND_HOME_PATH, TORCH_NPU_PATH — passed to CMake when set (setup.py:98-101); also two of the four variables that mark an Ascend platform (setup.py:18-23, setup.py:38).
  • ASCEND_OPP_PATH, ASCEND_TOOLKIT_HOME — the other two platform markers (setup.py:18-23, setup.py:38).
  • MAX_JOBS — the parallel build job count; defaults to the CPU count (setup.py:104).
Deep dive

Profilers

The plugin owns a torch profiler for each GPU runner role and each NPU runner role. Each family has its own prefix: AFD_GPU_ATTENTION_PROFILER, AFD_GPU_FFN_PROFILER, AFD_NPU_ATTENTION_PROFILER, AFD_NPU_FFN_PROFILER. The suffix picks the setting (_ENABLE, _WAIT, _WARMUP, _ACTIVE, _REPEAT, _SKIP_FIRST, _DIR; the NPU profiler adds _WITH_STACK). The GPU profiler reads them in afd_gpu_profiler_config (afd_plugin/compat/profiler.py:46-61); the NPU profiler in afd_npu_profiler_config (afd_plugin/compat/npu/profiler.py:52-70).

Deep dive

Benchmark and routing

Deep dive

Ascend (NPU) runtime

These matter only on an Ascend NPU. The Ascend NPU overview: the same contract, different transports page covers the NPU path.

vLLM variables that AFD depends on#

These variables belong to vLLM, not to the plugin, but AFD needs them set correctly. The recipes set the first one.

VariableWhy AFD needs itSource (vLLM)
VLLM_USE_V2_MODEL_RUNNER AFD runs on Model Runner V2. The validator raises when V2 resolves to off (afd_plugin/validation.py:90-91). When the variable is unset, vLLM still picks V2 by default; the recipes set it to 1 to be explicit. vllm:vllm/envs.py:2037-2038, vllm:vllm/config/vllm.py:692-694
VLLM_PLUGINS vLLM's plugin loader filters by this list. AFD loads only when afd is in the list, or the variable is unset (vllm:vllm/plugins/__init__.py:64). An empty string loads no plugin. vllm:vllm/envs.py:1146-1150
VLLM_MOE_ROUTING_SIMULATION_STRATEGY Selects a routing strategy. Set it to afd_balanced to use the plugin's deterministic benchmark routing (afd_plugin/model_executor/routing_simulator.py:5-8). vllm:vllm/envs.py:1743-1745
VLLM_TORCH_PROFILER_DIR The plugin profilers fall back to this directory for traces when the per-role ..._DIR is unset (afd_plugin/compat/profiler.py:131-132). Not declared in vLLM v0.30.0 (see note).

Refreshing this page#

Run the generator from the explainer folder:

python3 tools/gen_env_vars.py          # rewrite the table
python3 tools/gen_env_vars.py --check   # exit 1 if the table is out of date
python3 tools/gen_env_vars.py --report  # print the extracted facts
  • Tuning: the knobs the repo exposesWhat afd-plugin lets you tune, and only what its code and READMEs state: the A:F rank ratio, DBO thresholds, CUDA graph capture, the group timeout, gate placement, the NPU CAM capacity, the profiler, the routing simulator and the benchmark tools. Every number is cited.
  • Ascend NPU overview: the same contract, different transportsAFD on Ascend NPU keeps the same attention/FFN contract as GPU but uses two CAM connectors: CAMP2p (sync, with a control plane over HCCL and Gloo) and CAMAsync (async, no control plane, driven by its receive loop). This page maps the NPU workers, runners, model wrappers, patches and the operator loader, and points to the NPU docs.
  • AFD config reference: the additional_config["afd"] keysEvery key of --additional-config "afd", with its alias, type, default and validation, plus the connector_extra_config keys per connector and a worked example from a real recipe.
  • Recipes referenceEvery recipe script under recipe/, generated from the source, with hardware, connector, model, topology, role, DP/TP, mode, DBO thresholds and ports. Hand notes cover the parts the parser cannot read.