How vLLM plugins work

Code version: afd-plugin @ 7452d52 · 2026-10-27 · vLLM @ ced6857

The idea#

vLLM can load extra code that is not in its own source tree. The extra code is a plugin. A plugin declares an entry point in its packaging metadata. vLLM finds the entry point by its group name and calls it. afd-plugin uses this to add AFD, and it changes no vLLM file.

A general plugin is the simplest kind. It is one function. vLLM calls the function with no arguments. The function then registers models, patches vLLM functions, or sets up whatever the plugin needs. For afd-plugin the function is register_afd().

An everyday comparison. Think of a start-up checklist with named tasks. The host reads the list and runs each task once. The comparison stops here: vLLM runs the list again in every process it starts, and each run starts fresh. So one plugin function runs many times, in many processes.

The picture#

Read the top panel left to right. Process 0 starts first, then the engine core, then the workers. Each process loads the plugins once. The lower panel shows why a plugin must run safely more than once.

In the code#

Step 1: The five entry-point groups#

vLLM defines five plugin groups. Each group loads in a different set of processes. The comments in the file say where (vllm:vllm/plugins/__init__.py:16-30). afd-plugin uses only the first group, vllm.general_plugins.

# Default plugins group will be loaded in all processes(process0, engine core
# process and worker processes)
DEFAULT_PLUGINS_GROUP = "vllm.general_plugins"
# IO processor plugins group will be loaded in process0 only
IO_PROCESSOR_PLUGINS_GROUP = "vllm.io_processor_plugins"
# Platform plugins group will be loaded in all processes when
# `vllm.platforms.current_platform` is called and the value not initialized,
PLATFORM_PLUGINS_GROUP = "vllm.platform_plugins"
# Stat logger plugins group will be loaded in process0 only when serve vLLM with
# async mode.
STAT_LOGGER_PLUGINS_GROUP = "vllm.stat_logger_plugins"
# Endpoint plugins group is loaded in the API server front end process only.
# Each entry point resolves to a factory returning an `EndpointPlugin`
# (see `vllm/plugins/endpoint_plugins/interface.py`).
ENDPOINT_PLUGINS_GROUP = "vllm.endpoint_plugins"

Step 2: load_plugins_by_group and load_general_plugins#

Two functions do the work. load_plugins_by_group finds the entry points of one group, filters them, and loads each one (vllm:vllm/plugins/__init__.py:36-74). load_general_plugins loads the general group, then calls each loaded function (vllm:vllm/plugins/__init__.py:87-90).

def load_general_plugins():
    """WARNING: plugins can be loaded for multiple times in different
    processes. They should be designed in a way that they can be loaded
    multiple times without causing issues.
    """
    global plugins_loaded
    if plugins_loaded:
        return
    plugins_loaded = True

    plugins = load_plugins_by_group(group=DEFAULT_PLUGINS_GROUP)
    # general plugins, we only need to execute the loaded functions
    for func in plugins.values():
        func()

The filter is the VLLM_PLUGINS environment variable. vLLM parses it in envs.py. When the variable is not set, the value is None and vLLM loads every plugin (vllm:vllm/envs.py:1146-1150). When the variable is an empty string, the value is [""], which matches no plugin name, so vLLM loads none (vllm:vllm/plugins/__init__.py:63-64).

    # if this is not set, it means all plugins will be loaded
    # if this is set to an empty string, no plugins will be loaded
    "VLLM_PLUGINS": lambda: (
        None
        if "VLLM_PLUGINS" not in os.environ
        else os.environ["VLLM_PLUGINS"].split(",")
    ),

Step 3: When and in which processes#

vLLM calls load_general_plugins() from five call sites. The diagram above shows the four kinds of process. Each call site runs in the process that first needs the plugins. The excerpt below is the engine core.

        # plugins need to be loaded at the engine/scheduler level too
        from vllm.plugins import load_general_plugins

        load_general_plugins()

The deep dive below lists all five call sites and the process of each one.

Step 4: How a plugin registers models, workers and config#

A general plugin registers things through vLLM APIs that run in the same process.

• Models. The plugin calls ModelRegistry's register_model(name, model_cls). The class can be a string in the "module:Class" form, so vLLM imports it only when it is needed (vllm:vllm/model_executor/models/registry.py:1126-1141).
• Workers. There is no register API for a worker. vLLM reads the worker class name from parallel_config.worker_cls and imports it by its qualified name (vllm:vllm/v1/worker/worker_base.py:273-277). A plugin changes the worker by setting this field.
• Config. vLLM has one open config field, additional_config, for out-of-tree config (vllm:vllm/config/vllm.py:434). afd-plugin reads its "afd" key.

    def register_model(
        self,
        model_arch: str,
        model_cls: type[nn.Module] | str,
    ) -> None:
        """
        Register an external model to be used in vLLM.

        `model_cls` can be either:

        - A [`torch.nn.Module`][] class directly referencing the model.
        - A string in the format `<module>:<class>` which can be used to
          lazily import the model. This is useful to avoid initializing CUDA
          when importing the model and thus the related error
          `RuntimeError: Cannot re-initialize CUDA in forked subprocess`.
        """

Step 5: What a plugin can and cannot change#

A general plugin runs real Python in the vLLM process. So it can change vLLM behaviour by monkey-patching: it replaces a vLLM function or method with its own. afd-plugin does this with its compat patches.

A plugin cannot do some things:

• It cannot change a name that another module already bound with from ... import. That name keeps the original object. A patch that replaces a module attribute (module.func) changes only later attribute lookups. So the patch must run before the code that reads the attribute.
• It cannot change compiled C++ or CUDA code. It can only change the Python that calls it.
• It cannot add a new command-line flag on its own. The plugin function gets no parser.

There is one near-exception for the command line. vLLM calls load_general_plugins() at the start of add_cli_args (vllm:vllm/engine/arg_utils.py:3003-3009). So a plugin that registers a new quantization method or a new platform makes it a valid choice for the existing --quantization or --device flag.

Deep dive#

In afd-plugin#

afd-plugin is a single general plugin. Three facts connect it to this page:

• The entry point is afd = "afd_plugin:register_afd" in the vllm.general_plugins group (pyproject.toml:45-46). See The plugin entry point: register_afd.
• The function register_afd applies the patches and registers the models and the routing strategy (afd_plugin/__init__.py:142-216). See The plugin entry point: register_afd and afd_plugin/__init__.py: the plugin entry module.
• One compat patch sets the worker class. It rewrites parallel_config.worker_cls, but only when it is "auto" (afd_plugin/compat/patches/config_validation.py:60, afd_plugin/compat/patches/config_validation.py:92). See Worker selection: choosing the role worker and Compat patches: how AFD changes vLLM without editing it.

What can go wrong#

• VLLM_PLUGINS set to empty. An empty string loads no plugins, so AFD does not activate (vllm:vllm/envs.py:1146-1150).
• A plugin that is not idempotent. The function runs in many processes. State kept in one process does not reach the others.
• A patch that runs too late. A monkey-patch cannot change a name that another module already bound with from ... import. The patch must run before that import, or before the attribute lookup it changes.

Why it matters. AFD is installed only through this plugin mechanism. If the plugin does not load, or a patch runs too late, the two roles do not form and the run fails.