What plugin discovery actually checks

The door check at a party#

Think of a host checking guests at a party. They look for your name on the list. They let you in. Then they check your ID, and find you’re not old enough. By then you’ve already been in the kitchen.

That’s the order a typical Python plugin loader works in. Its list is installed package metadata, letting a guest in is importing the plugin, and checking ID is the version check, which often can’t happen until the guest is inside, because the version is written on the plugin’s own class. Each check below is one step at that door.

Five one-file plugins, installed in fresh virtual environments with real wheels, hit every outcome:

Check 1: find the entry point#

A plugin is a separate distribution, the thing pip installs (textpipe-future), which contains an importable package (textpipe_future). It announces itself with an entry point, a line of package metadata that says “I’m here”: a group the host searches, and the object to load. It’s declared in pyproject.toml:

1
[project.entry-points."textpipe.filters"]
2
summarize = "textpipe_future:Summarize"

So textpipe.filters is the group, summarize the plugin’s name, and textpipe_future:Summarize the class to load. The build tool writes that into an entry_points.txt file, and pip installs it with the rest of the distribution’s metadata in a .dist-info folder next to the code. The host reads it with importlib.metadata:

1
>>> from importlib.metadata import entry_points
2
>>> for ep in entry_points(group="textpipe.filters"):
3
...     print(ep.name, ep.value, ep.dist.name)
4
summarize textpipe_future:Summarize textpipe-future
5
shout textpipe_shout:Shout textpipe-shout

Nothing is imported yet: an EntryPoint is those strings plus ep.dist, the distribution that declared it. This is the guest list, read without letting anyone in.

It’s also why a plugin you’re developing can vanish. A source checkout on PYTHONPATH has code but no installed metadata, so it isn’t on the list, unless an earlier build left an *.egg-info folder (setuptools’ metadata in the source tree) beside it. Then it shows up, with that stale metadata.

Checks 2 to 4: the loop#

Here’s textpipe’s discovery method, the whole door. _loader is importlib.metadata.entry_points; tests hand it fake entry points instead, which the last section uses:

1
def _discover(self, group):
2
    claims = {}                                   # name -> (package, class)
3
    for ep in self._loader(group=group):          # 1. find: read installed metadata
4
        if not self._admit(group, ep):
5
            continue
6
        try:
7
            cls = ep.load()                       # 2. import: runs the plugin's module
8
            self._check_version(cls)              # 3. version
9
        except PluginIncompatibleError as exc:
10
            self._skip(group, ep.name, str(exc))
11
            continue
12
        except Exception as exc:
13
            self._skip(group, ep.name, repr(exc), verb="could not import plugin")
14
            continue
15
        package = getattr(ep.dist, "name", None)  # 4. name: which package claims it
16
        earlier = claims.get(ep.name)
17
        if earlier and earlier[0] and package and earlier[0] != package:
18
            raise PluginCollisionError(
19
                collision_message(group, ep.name, earlier, (package, cls)))
20
        claims[ep.name] = (package, cls)
21
    return {name: cls for name, (_package, cls) in claims.items()}

Read it in order:

1. Find. Loop over every entry point in the group. _admit is a place for checks that need only metadata, which a subclass can fill in; the basic registry admits everything.
2. Import. ep.load() imports the plugin’s module and returns its class. If that raises, _skip logs a warning and moves on.
3. Version. _check_version reads the class’s declared API version, and a too-new plugin is skipped the same way.
4. Name. claims maps each name to the distribution that claimed it. If a second distribution claims a name that’s taken, the loop raises rather than choosing.

The pattern behind it: when a plugin can’t work here, the loader logs it and moves on, so one broken plugin can’t take down an app that doesn’t use it. When the loader would have to guess, two distributions with one name, it raises. Otherwise sys.path order, the order Python searches folders for code, then the order files happen to be listed in, would pick the winner, and neither says anything about which plugin you meant.

Check 2: letting a plugin in runs it#

ep.load() imports the module named in the entry point, and importing a module runs everything at its top level: other imports, a print, opening a connection, starting a thread. Here’s the whole of textpipe-future:

1
from textpipe import Filter
2

3
print("textpipe-future: warming up the model cache")  # runs on import
4

5

6
class Summarize(Filter):
7
    api_version = "2.0"
8

9
    def apply(self, text):
10
        return text[:40]

That print is the guest in the kitchen. It runs the moment the plugin is imported, before anything has asked whether it fits.

If the import fails, the plugin is skipped. textpipe-broken imports a module that isn’t installed, so the first discovery logs a warning, and every later request for it gets a bare “no usable plugin”:

WARNING textpipe.plugins: could not import plugin 'translate': ModuleNotFoundError("No module named 'language_tables'")
PluginNotFoundError: No usable plugin named 'translate' in 'textpipe.filters' (usable: shout).

Check 3: checking ID once they’re inside#

textpipe-future declares the API version it targets as a class attribute, api_version = "2.0". Reading it needs the class, and the class needs the import, so this check always runs after the plugin’s code. A version on the class tells the host what the plugin claims to target; it can’t protect the host from code that runs at import.

How the versions are compared matters too. textpipe’s parse_version_prefix keeps the leading number of each dotted part, so "2.0" becomes (2, 0), and refuses the plugin when its tuple is bigger than the host’s. That mostly works, with two traps:

The third row is wrong by Python packaging’s own rules. Its version spec (PEP 440) pads the shorter version with zeros: “X.Y and X.Y.0 are not considered distinct release numbers.”

The fourth row is the one I’d fix before a host’s first breaking release. A plugin declaring 1.0 loads on a host at 2.0, after exactly the breaking changes a new major version, the first number, exists to signal. The fix is to refuse other major versions too. But the check compares against textpipe’s package version, so once it refuses other majors, every major release of textpipe would count as a plugin-API break, whether the plugin interface changed or not. A separate constant, say textpipe.PLUGIN_API = "1.0", bumped only when that interface changes, avoids it.

Check 4: two distributions, one name#

Two guests give the door the same name. The entry points spec only asks for names to be unique within one distribution’s group, and leaves the rest to the host: “If different distributions provide the same name, the consumer decides how to handle such conflicts.” Install textpipe-shout and textpipe-loud, which both register shout:

PluginCollisionError: Two packages register 'shout' in 'textpipe.filters': textpipe-loud (textpipe_loud:Loud) and textpipe-shout (textpipe_shout:Shout). Rename the entry point in one of them, or uninstall one.

It names both distributions and both fixes, and takes the whole group down: nothing is cached, so every later request re-runs discovery, retries any import that failed, re-logs the warnings, and raises again. The check also only sees plugins that passed checks 2 and 3: if one shout distribution can’t import, the collision is never reported.

entry_points: shout -> textpipe-shout 0.2.0
distributions: textpipe-shout 0.2.0, textpipe-shout 0.1.0

Choosing between two plugins that both work comes after discovery. Plugins subclass textpipe’s base classes, and Casing is the one for case-changing filters. When the app asks for any plugin of a kind, select(Casing), and two qualify, textpipe raises until the app pins one, names the plugin it wants:

select(Casing): AmbiguousPluginError: 2 plugins are a Casing: caps, shout. Pin one: select(Casing, pin=...).
select(Casing, pin='caps'): textpipe_caps:Caps

The same failures in pytest, Flask and stevedore#

The same kinds of plugin behave like this in pytest 9.1.1 on pluggy 1.6.0, Flask 3.1.3 and stevedore 5.9.1, each tried with small test plugins in a fresh environment:

ExtensionManager is stevedore’s manager for loading every plugin in a group, and DriverManager the one for loading exactly one. Every cell reproduces with sh real_hosts/run.sh in plugin-discovery/, which pins each version.

Two cases come out the same everywhere, by how Python finds metadata rather than by anything run.sh tests: a plugin with no installed metadata is never seen, and a distribution installed twice silently resolves to the first copy on sys.path. In the first row, “not found” means “skipped, read the log” in textpipe and stevedore’s ExtensionManager, while pytest and Flask stop at startup with the traceback.

What I’d change: check ID at the door#

The demo’s stricter.py makes three changes through the registry’s hooks.

Check what you can before letting them in#

If the plugin’s distribution declares which host versions it needs (dependencies = ["textpipe>=2.0"] in its pyproject.toml), that requirement is in ep.dist.requires, and _admit can check it before ep.load(). That’s checking ID at the door:

python↩ WrapCopy

1
def _admit(self, group, ep):
2
    try:
3
        spec = declared_host_requirement(ep)      # textpipe's range, from ep.dist.requires
4
    except UNREADABLE as exc:                     # an unparseable line or marker
5
        self._skip(group, ep.name, f"unreadable requirement ({exc}); not imported")
6
        return False
7
    if spec is not None and not spec.contains(self._host_version, prereleases=True):
8
        self._skip(group, ep.name, f"its package requires textpipe{spec}, and "
9
                                   f"textpipe {self._host_version} is installed; not imported")
10
        return False
11
    return True

declared_host_requirement finds the plugin’s requirement on textpipe, parsed with the packaging library. It skips any requirement whose marker, the condition after ;, doesn’t apply, such as extra == "gpu" for an optional extra. If the host’s version isn’t in the declared range, the plugin is skipped without being imported; prereleases=True lets a pre-release host like 2.0rc1 count as in range. UNREADABLE is the set of errors a requirement can raise when it can’t be parsed or its marker can’t be evaluated, so one bad line skips one plugin, rather than breaking discovery for all of them. With this, the warming-up line from the opening never prints:

textCopy

WARNING textpipe.plugins: skipping plugin 'summarize': its package requires textpipe>=2.0, and textpipe 1.4 is installed; not imported

That’s the most this check can do. It isn’t a safety boundary: a plugin that passes every check still runs whatever is at its top level.

Doesn't pip already check this?

The same requirement lets pip stop a mismatched install, but only for the packages in the current command. Without --no-deps, pip won’t install textpipe-future next to textpipe 1.4. Yet with textpipe 2.0 and textpipe-future installed properly, pip install textpipe==1.4 downgrades anyway. It prints the conflict under an ERROR: header but exits 0, and after that only pip check notices:

textCopy

$ pip install textpipe==1.4          # with textpipe 2.0 and textpipe-future installed
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
textpipe-future 0.1.0 requires textpipe>=2.0, but you have textpipe 1.4 which is incompatible.
Successfully installed textpipe-1.4
$ pip check
textpipe-future 0.1.0 has requirement textpipe>=2.0, but you have textpipe 1.4.

So there are three places a version check can run, and the earlier, the more it prevents:

PluginNotFoundError: 'summarize' is installed but not usable: its package requires textpipe>=2.0, and textpipe 1.4 is installed; not imported.

If you write or install plugins#

• Keep your module’s top level free of side effects; do the work in the plugin’s methods.
• Declare the host range in dependencies ("textpipe>=1.4,<2"), so pip, and a host that checks before importing, can refuse a mismatch.
• Develop with an editable install (pip install -e .), not PYTHONPATH: it writes the metadata the host needs to find you.
• Run pip check after installing or upgrading anything next to a host.
• When a host says “not found”, read the warnings above the error. In textpipe, and in stevedore’s ExtensionManager if the app configures logging, that’s where the reason is.

Three checks for your own host#

Each failure above is a question you can ask of your own loader, without installing anything, using a stand-in entry point: an object with name, dist and load(), all textpipe’s registry reads, whose load() counts how many times it’s called:

1
from types import SimpleNamespace
2

3

4
class StandIn:
5
    """Enough of importlib.metadata.EntryPoint for textpipe's registry to use."""
6

7
    def __init__(self, name, cls, package, requires=()):
8
        self.name, self.cls, self.loads = name, cls, 0
9
        self.dist = SimpleNamespace(name=package, requires=list(requires))
10

11
    def load(self):
12
        self.loads += 1
13
        return self.cls

textpipe takes its entry points as a constructor argument, so a test just hands it stand-ins. (Patching the name wouldn’t reach textpipe’s default, which is bound when the registry is created.) If your host looks up importlib.metadata.entry_points itself and calls it during discovery, patch that name in a pytest test instead:

1
# if your host's module does `from importlib.metadata import entry_points`:
2
monkeypatch.setattr(myhost.plugins, "entry_points", lambda **kw: stand_ins)
3
# or, if it calls importlib.metadata.entry_points(...) directly, as Flask's CLI does:
4
monkeypatch.setattr(importlib.metadata, "entry_points", lambda **kw: stand_ins)

That only works for a loader that calls entry_points: pluggy walks distributions() instead, so for pytest you’d fake whole distributions, as pluggy’s own tests do, or test with real installed plugins, as the demo’s run_scenarios.sh does.

Then ask three questions, using your registry’s discovery call (available() here) and its lookup by name (get() here):

1. Is a too-new plugin refused, and was it imported first? Give it a stand-in whose class targets a newer API and whose distribution declares textpipe>=2.0. Run discovery, see whether the plugin came back, and read loads.
2. Does “not found” say why? Ask for that plugin by name, and read the message.
3. What does a collision do, and does it stay broken? Give it two stand-ins with one name from different distributions, and run discovery twice.

Against textpipe’s two registries:

Registry
  1. too-new plugin: refused: yes, imported: yes
  2. lookup: PluginNotFoundError: No usable plugin named 'new' in 'textpipe.filters' (usable: none).
  3. collision, two requests: PluginCollisionError, PluginCollisionError
StricterRegistry
  1. too-new plugin: refused: yes, imported: no
  2. lookup: PluginNotFoundError: 'new' is installed but not usable: its package requires textpipe>=2.0, and textpipe 1.4 is installed; not imported.
  3. collision, two requests: PluginCollisionError, PluginCollisionError

Read your own answers the same way:

The demo#

The host, the five plugins and 31 tests are in plugin-discovery/. run_scenarios.sh builds every package as a wheel and runs each scenario in a fresh virtual environment, which is where the traces here come from (the version-table cases and the import corners come from the unit tests); real_hosts/run.sh does the same for pytest, Flask and stevedore at pinned versions.

cd mainlog-code/plugin-discovery
sh run_scenarios.sh /tmp/textpipe-work           # every trace above, in fresh environments
python3 -m venv .venv && .venv/bin/pip install packaging pytest /tmp/textpipe-work/dist/textpipe-1.4-py3-none-any.whl
.venv/bin/python -m pytest tests
sh real_hosts/run.sh /tmp/real-hosts-work        # the pytest, Flask and stevedore table

Installing the built wheel, rather than pip install ./textpipe, keeps pip from building inside the source tree, which would leave an *.egg-info folder behind: the stale-metadata trap from check 1.