Compat patches: how AFD changes vLLM without editing it

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

The idea#

AFD is an external afd-plugin. It must not edit the vLLM source. Instead, it replaces a few vLLM functions at run time with its own versions. Each replacement is a patch, in afd_plugin/compat/patches/. register_afd() imports the patch modules once at start-up (afd_plugin/__init__.py:172-181).

Every patch copies the matching vLLM function from the pinned release and marks only the AFD difference between PATCH START and PATCH END comments. The patched signature must match the original (AGENTS.md:161-168). A version gate decides whether a patch installs, so AFD does not break a vLLM it was not built for.

An everyday comparison. Think of a train that swaps one carriage for a custom one before it leaves, and swaps the original back when it returns. The comparison stops here: a patch checks the train model first, and some patches swap the carriage only for the length of one trip.

The picture#

Read panel 1 as two separate checks. One warns at start-up; the other gates each install. Panel 2 lists the GPU patch modules and how each one installs.

In the code#

Step 1: Patch discipline#

The patches package is a closed namespace: it raises on any unexpected attribute, so nothing leaks (afd_plugin/compat/patches/__init__.py:12-13). Each patch function carries comments that give the reason, the change, and the upstream target (AGENTS.md:165-170), and marks the AFD lines with PATCH START and PATCH END.

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the AFD plugin project
"""Isolated monkey-patch namespace.

Patches in this package must remain idempotent, version-aware, documented, and
covered by CPU-safe tests whenever possible.
"""

__all__: list[str] = []


def __getattr__(name: str):
    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

Step 2: Two version checks#

The first check runs in register_afd. It is not strict, so a vLLM that is not exactly the target release only warns (afd_plugin/compat/vllm.py:38-50, afd_plugin/__init__.py:162-170).

The second check is the install gate, is_target_vllm_compatible. It allows the target release and any dev build, and returns true when vLLM is absent so CPU imports still work (afd_plugin/compat/vllm.py:53-68).

def is_target_vllm_compatible() -> bool:
    """Return whether compatibility patches may install.

    Tolerates development builds of the target release and any import
    failure (no vLLM installed) so CPU-safe imports keep working.
    """
    try:
        import vllm

        version_value = vllm.__version__
    except (AttributeError, ImportError):
        return True
    version_text = str(version_value)
    if "dev" in version_text:
        return True
    return version_text.startswith(TARGET_VLLM_VERSION)

Step 3: config_validation#

This patch replaces two vLLM functions: EngineArgs.create_engine_config (create_engine_config) and VllmConfig.__post_init__ (VllmConfig.__post_init__). It installs at import time, behind the gate (afd_plugin/compat/patches/config_validation.py:235-261).

What it changes: it rewrites worker_cls to the AFD role worker, but only when the value was "auto" (afd_plugin/compat/patches/config_validation.py:92-93, afd_plugin/compat/patches/config_validation.py:179-197). It also relaxes one vLLM ubatching backend check for AFD configs (afd_plugin/compat/patches/config_validation.py:200-214). See Worker selection: choosing the role worker.

Step 4: engine_core#

This patch replaces EngineCore.__init__, _initialize_kv_caches and shutdown, plus the busy loops of EngineCoreProc and DPEngineCoreProc (afd_plugin/compat/patches/engine_core.py:722-727). The vLLM targets are in vllm/v1/engine/core.py (EngineCore.__init__, EngineCoreProc.run_busy_loop).

What it changes: for an FFN config, __init__ returns right after the executor is built, so the FFN EngineCore has no real scheduler (a no-op stand-in) and no KV cache (afd_plugin/compat/patches/engine_core.py:56-68, afd_plugin/compat/patches/engine_core.py:614). The busy loop instead runs the connector loop (afd_plugin/compat/patches/engine_core.py:411-419). See Attention and FFN roles: what each server owns.

Step 5: gpu_mrv2_ubatching#

This patch is different: it installs through scoped context managers, not at import time (afd_plugin/compat/patches/gpu_mrv2_ubatching.py:263-282). It replaces maybe_build_ubatch_runner (maybe_build_ubatch_runner) and dispatch_cg_and_sync_dp (dispatch_cg_and_sync_dp) only inside AFD's own initialize_kv_cache and execute_model calls. It re-enables DBO at one data-parallel rank. See Dual-batch overlap (DBO): two stages per step.

Step 6: async_dp patches#

Two patches support async-DP, which is NPU-only. They install at import time but only change behavior for async configs, gated at run time (afd_plugin/config.py:303-308). On GPU, async_dp must be false, so these patches run as transparent re-implementations.

• async_dp_engine replaces EngineCoreProc.run_engine_core (run_engine_core), launch_core_engines (launch_core_engines) and DPAsyncMPClient.add_request_async (add_request_async), selecting a non-DP engine for async attention (afd_plugin/compat/patches/async_dp_engine.py:117-120).
• async_dp_forward_context replaces set_forward_context (set_forward_context) and skips DP metadata only for async configs (afd_plugin/compat/patches/async_dp_forward_context.py:88-127).

Step 7: NPU patches#

The patches under afd_plugin/compat/patches/npu/ install only when the device is NPU (afd_plugin/__init__.py:204-209). They are covered on Ascend NPU overview: the same contract, different transports. In one line each:

• npu/ascend_config: keep AFD settings out of the strict Ascend config namespace (afd_plugin/compat/patches/npu/ascend_config.py:3).
• npu/ascend_platform: patch the Ascend platform config normalization for AFD DBO (afd_plugin/compat/patches/npu/ascend_platform.py:3-5).
• npu/mla_graph: route Ascend MLA ACL graph parameters through AFD forward contexts (afd_plugin/compat/patches/npu/mla_graph.py:3-5).
• npu/mrv2_ubatching: scope Ascend seams for the native ubatch runner (afd_plugin/compat/patches/npu/mrv2_ubatching.py:3-8).

Deep dive#

The blocks below give the install gate and its guards, the monkey-patch rule, and a pointer to the full table.

What can go wrong#

• A vLLM that is not the target release. The start-up check only warns (afd_plugin/compat/vllm.py:48-50), but every install gate is false, so no patch installs (afd_plugin/compat/patches/config_validation.py:235).
• A patch that imports but cannot apply. The patch imports are wrapped in a try that only logs on failure (afd_plugin/__init__.py:172-181).
• A worker set to a non-AFD class. The patch rewrites worker_cls only when it was "auto"; an explicit non-AFD worker is rejected later (afd_plugin/validation.py:357-367). See Worker selection: choosing the role worker.
• A signature drift after a vLLM upgrade. A patched function must match the vLLM signature, or the call fails. The patch comments record the exact upstream target (AGENTS.md:165-170).

Why it matters. The patches are how AFD runs inside unmodified vLLM. The version gate keeps AFD from breaking a vLLM it was not built for, and the scoped patches keep AFD changes out of native requests.