Green on my laptop, red in CI

The plugin, and the line pytest reads#

Here’s the project:

scenariokit/
├── pyproject.toml                    # declares the pytest plugin
├── src/scenariokit/
│   ├── __init__.py
│   ├── registry.py                   # the @scenario decorator
│   ├── context.py                    # talks to the development server
│   └── pytest_plugin.py              # teaches pytest to run scenarios
└── tests/
    ├── test_registry.py
    ├── test_pytest_plugin.py         # runs pytest inside a test
    └── examples/
        └── test_greeting_example.py  # the README example

A scenario looks like this:

1
@scenario("greeting")
2
async def greets_new_client(ctx):
3
    assert await ctx.request("HELLO") == "WELCOME"

This one is the README example. It sits under tests/ in a test_ file because that’s how users run scenarios: put them in a test file and point pytest at a running server.

pytest wouldn’t normally run that function. Its name doesn’t start with test_, and pytest doesn’t run async functions by itself. The plugin changes that. pytest is the host: it owns the test run and calls a plugin’s hooks at fixed points, such as “here’s an object from a test file; is it a test?” That’s inversion of control.

But pytest can only call a plugin it knows about. The package tells it through an entry point in pyproject.toml:

1
[project.entry-points.pytest11]
2
scenariokit = "scenariokit.pytest_plugin"

pytest11 is the group pytest searches for installed plugins. scenariokit is the plugin’s name, and the value is the module to import.

Here’s the part I had missed: pytest doesn’t read that declaration from pyproject.toml. Building the package copies it into a small metadata file that is installed next to the code, and pytest reads the installed metadata when it starts:

site-packages/
├── scenariokit/                      # the same .py files as src/scenariokit/
└── scenariokit-0.1.0.dist-info/
    ├── METADATA
    ├── RECORD
    ├── WHEEL
    ├── entry_points.txt              # [pytest11]
    │                                 # scenariokit = scenariokit.pytest_plugin
    └── …

A checkout on PYTHONPATH has the .py files but no .dist-info directory. As far as Python’s package metadata is concerned, the package isn’t installed at all. One line shows the difference:

python -c "from importlib.metadata import entry_points; print(entry_points(group='pytest11'))"

From my source checkout it prints (). With the wheel installed, it prints:

(EntryPoint(name='scenariokit', value='scenariokit.pytest_plugin', group='pytest11'),)

You can also see it at the top of any pytest run. When pytest loads a plugin from an entry point, its session header includes it in a line like plugins: scenariokit-0.1.0. The run from my checkout doesn’t list it. That’s a five-second comparison between a CI log and a local run.

So on my laptop, the main test run never loaded the plugin. The one test that needed it loaded it by hand, and that’s where the second failure comes from.

Failure 1: the README example became a test#

When pytest collects a test file, it imports the file and offers each object in it to the loaded plugins. Our plugin claims the scenarios:

1
@pytest.hookimpl(tryfirst=True)
2
def pytest_pycollect_makeitem(collector, name, obj):
3
    scenario_name = getattr(obj, "_scenario_name", None)
4
    module = getattr(collector, "obj", None)
5
    if scenario_name is None or obj.__module__ != getattr(module, "__name__", None):
6
        return None  # not a scenario defined in this file: normal collection
7
    return ScenarioItem.from_parent(
8
        collector, name=f"scenario[{scenario_name}]", fn=obj)

@scenario only tags the function with _scenario_name. The pytest_pycollect_makeitem hook turns tagged functions into test items and returns None for everything else, which leaves normal collection alone. tryfirst=True lets it answer before other async-aware plugins do, and the __module__ check skips scenarios that a test file merely imports. Each item runs its scenario with asyncio.run(), using a context that connects to 127.0.0.1:7357 unless you pass --scenario-server.

Here is the whole path in CI, from startup to the failure:

On my laptop, pytest still imports tests/examples/test_greeting_example.py, because the file name matches test_*.py. Without the plugin, it finds nothing it recognizes as a test and moves on. Nothing ran, so nothing failed.

In CI, the plugin is loaded before collection starts, so it claims greets_new_client. pytest runs it, and the scenario tries to connect to a development server that CI never started. Here’s that failure, trimmed and reproduced on macOS:

FAILED tests/examples/test_greeting_example.py::scenario[greeting]
E   ConnectionRefusedError: [Errno 61] Connect call failed ('127.0.0.1', 7357)

(Error 61 is macOS’s number for a refused connection; Linux reports 111.)

The plugin was doing its job. CI was just the first place anything ran that example without a server.

Those examples are documentation, meant for someone who has started a server. So they stay out of this repository’s default run:

1
[tool.pytest.ini_options]
2
testpaths = ["tests"]
3
addopts = ["--ignore=tests/examples"]

Naming the directory still runs them, which is what I wanted. With pytest 9.1.1, pytest tests/examples --scenario-server HOST:PORT collects the scenario despite the ignore option.

Failure 2: the plugin’s own test loaded it twice#

The plugin needs tests of its own, like “does a @scenario function become a test item?” A normal test can’t answer that, so pytest provides the pytester fixture. It writes a throwaway test file and runs a second pytest session on it, inside the same Python process:

1
pytest_plugins = ["pytester"]  # enables the pytester fixture
2

3

4
def test_scenario_becomes_a_test_item(pytester):
5
    pytester.makepyfile(test_demo=SCENARIO_FILE)
6
    result = pytester.runpytest("-p", "scenariokit.pytest_plugin",
7
                                "--collect-only", "-q")
8
    result.stdout.fnmatch_lines(["*scenario[[]demo[]]*"])

The nested session loads plugins the same way a normal run does. On my laptop there was no metadata to find scenariokit with, so the test had to name it: -p scenariokit.pytest_plugin means “import this module and register it as a plugin.” Without that flag, the nested session wouldn’t know about scenarios at all.

In CI, the nested session loaded it both ways:

First, -p registered the module under its module name. Then automatic discovery found the entry point and tried to register the same module object under the entry point’s name. Pluggy, pytest’s plugin manager, won’t hold one plugin under two names. Because pytester catches the nested session’s crash, the outer test doesn’t fail with that error. It fails because the expected output never appeared, and the real cause is in the captured output (trimmed):

FAILED tests/test_pytest_plugin.py::test_scenario_becomes_a_test_item
E       Failed: remains unmatched: '*scenario[[]demo[]]*'
----------------------------- Captured stderr call -----------------------------
ValueError: Plugin already registered under a different name: scenariokit=<module 'scenariokit.pytest_plugin' from '.../site-packages/scenariokit/pytest_plugin.py'>

If a plugin test fails in CI with an unmatched pattern, read the captured stderr before you debug the pattern.

The fix is to pass -p only when pytest won’t find the plugin by itself:

1
from importlib.metadata import entry_points
2

3

4
def plugin_args():
5
    """Load the plugin explicitly only when pytest won't find it by itself."""
6
    installed = {ep.name for ep in entry_points(group="pytest11")}
7
    return [] if "scenariokit" in installed else ["-p", "scenariokit.pytest_plugin"]

The test then calls pytester.runpytest(*plugin_args(), "--collect-only", "-q"), and the same test works against a checkout or an installed package.

I also checked another way out: name the entry point after the module ("scenariokit.pytest_plugin" = "scenariokit.pytest_plugin"). pytest then sees a plugin with that name is already registered and skips the second one, so -p stops colliding. But the entry-point name is also what users type to turn a plugin off (-p no:scenariokit), so I’d rather keep the short name and fix the test.

Run what CI runs, before CI does#

Both fixes were small once I could see the failures. The expensive part was not being able to see them on my laptop. The fix that lasts is a check that tests what CI tests (the built wheel) in a fresh environment that has only the wheel and pytest installed. With build installed:

python -m build --wheel                          # the artifact CI installs
python -m venv --clear /tmp/wheel-check          # a fresh, empty environment
/tmp/wheel-check/bin/pip install pytest dist/scenariokit-0.1.0-py3-none-any.whl
/tmp/wheel-check/bin/python -c "import scenariokit; print(scenariokit.__file__)"
/tmp/wheel-check/bin/python -m pytest            # the suite, against the installed copy

--clear matters on the second run. Without it, venv reuses the old environment, and pip keeps the previously installed wheel if the version number hasn’t changed. The check would quietly test yesterday’s build.

The fourth command is there because a passing run only means something if it used the installed copy. It should print a path inside /tmp/wheel-check/lib/.../site-packages. That isn’t automatic: python -m pytest adds the current directory to sys.path. With the package under src/, it isn’t importable from the project root, so the tests get the installed copy. If the package sat at the top level of the repository, the same commands would quietly test the checkout again. The same happens if your pytest configuration sets pythonpath = ["src"]; add -o pythonpath= to the last command to switch it off for this run.

Here’s what the check reports for each version of the example, next to the run I used to trust:

The left column never changes. That’s the whole problem in one table: the run I relied on couldn’t tell these three versions apart.

An editable install (pip install -e .) also writes the entry-point metadata, and it reproduced both failures here. For day-to-day runs, I’d use one instead of bare PYTHONPATH. It still runs the files in your checkout, though, so it can’t tell you whether the wheel is missing something. The wheel check can.

When I run the wheel check#

The wheel check is slower than a normal test run, so I don’t run it on every change. I run it when a change touches how the package presents itself to other programs:

• anything under [project.entry-points.*] or [project.scripts]: pytest plugins, console commands, other tools’ plugin groups;
• tests that start pytest themselves, or code that reads package metadata through importlib.metadata;
• what goes into the wheel (package data, include and exclude rules, the build backend), which only the wheel check sees;
• any CI failure I can’t reproduce locally. That was the signal I had here, and it’s the cheapest one to act on.

Neither CI failure was flaky, and neither was a CI quirk. Both were accurate reports of what the suite does once the package is really installed. The environment that was wrong was my laptop: it was testing a package that doesn’t exist outside my checkout. So when a change touches any of those, I build the wheel, install it into a clean environment, and run the suite there before CI does.