Keeping optional Python imports optional

What Python remembers after an import#

It helps to separate a package being available to import from a module being loaded into a running interpreter. The package’s files can be sitting on disk while the program has never imported them.

For an ordinary Python source module, the first import executes the code at module level. That creates its functions and variables, and runs any imports written there. Importing one function with from module import function still loads the module containing it. Python’s introduction to modules walks through this distinction.

Defining a function doesn’t execute its body. An import inside that body waits until execution reaches it during a call. That’s the small fact we’re relying on when we make an optional import lazy.

Our test runner is a Python program running in an operating-system process. Inside it, an interpreter executes the tests and holds their module objects in memory.

The interpreter keeps a dictionary called sys.modules. Module names are the keys, and module objects are the values. Test functions running in that interpreter share this state. A new test function doesn’t get a new import history.

An ordinary repeat import finds the existing module in that cache and reuses it. It doesn’t start the module’s initialization over again. The module-cache reference also explains what happens if code deliberately removes or replaces an entry; here we’re concerned with normal imports.

That gives us somewhere to look for the optional dependency. It also explains why where we look will matter.

A small library with one optional feature#

Let’s make up a tiny library, runtime_helpers. Its basic job is trimming whitespace. Its optional feature adds some formatting through another made-up module, optional_toolkit.

Here’s runtime_helpers.py:

1
"""A basic string operation and an optional formatting feature."""
2

3

4
def normalize(value):
5
    return value.strip()
6

7

8
def decorate(value):
9
    import optional_toolkit
10

11
    return optional_toolkit.decorate(normalize(value))

Notice where the import sits. Loading this file creates normalize and decorate; the optional import waits inside decorate.

For optional_toolkit.py, all we need is something recognizable:

1
"""A tiny optional formatter for the example."""
2

3

4
def decorate(value):
5
    return f"*** {value} ***"

Now we can follow the same interpreter as it does a little more work each time:

These are the observed results in a fresh interpreter with both files available. The third row is allowed: the caller has chosen the optional feature. We’re trying to keep that import out of the first row.

This is also why the test should have the optional dependency available. We want an accidental eager import to succeed so we can detect what it loaded. Otherwise a missing-package error could obscure the particular behavior we’re checking.

Give the import a fresh process#

Suppose another test has already called decorate. Checking the main test runner’s sys.modules would now find optional_toolkit, even if our library’s import boundary is correct. Conversely, if the library is already cached, importing it again doesn’t exercise its initialization under the conditions we meant to test.

I find a new Python process easier to reason about here. The parent stays in charge of the test; the child starts a separate interpreter and reports what happened during one import. Python’s subprocess module gives us that boundary and the pipes to collect the result.

A fresh interpreter still has startup modules loaded. So the child first checks that the optional namespace is absent. It then imports our library and checks again. We’re testing the whole namespace: optional_toolkit and names starting with optional_toolkit.. A name such as optional_toolkit_extra doesn’t belong to it.

Here’s the successful path:

The result comes from a completed check in the child’s interpreter; earlier imports in the parent don’t determine it.

The last two messages are both necessary. An exception can stop the child before it reaches the check. Less obviously, a module can call sys.exit(0). The process can exit “successfully” while skipping the rest of our probe. That’s why the parent requires an exact completion marker as well as exit status zero.

A cache entry isn’t proof that initialization finished. Python registers a module before executing its body, so recursive imports can find the object being initialized. We wait for the import call to return and the dependency check to finish before printing our marker.

Turning that sequence into a test#

With that model in place, the helper mostly follows the picture. The string named child is the little program we ask the new interpreter to run. importlib.import_module lets it import the target by name.

Everything on the blue side runs in the child. Nothing an earlier test imported can reach the check there, and the parent judges the result only by what comes back along the dashed arrow.

Here’s the complete assert_no_import.py:

1
"""Check optional import boundaries in a fresh Python interpreter."""
2
from pathlib import Path
3
import subprocess
4
import sys
5

6

7
def assert_import_does_not_load(module, forbidden, *, source_root=None):
8
    """Check one import in a child using this interpreter and installed environment."""
9
    child = r"""
10
import contextlib
11
import importlib
12
import sys
13

14
module, forbidden, source_root = sys.argv[1:]
15
if source_root:
16
    sys.path.insert(0, source_root)
17

18
def loaded():
19
    return sorted(name for name in tuple(sys.modules)
20
                  if name == forbidden or name.startswith(forbidden + "."))
21

22
if loaded():
23
    raise RuntimeError("Invalid test setup: forbidden module already loaded")
24

25
with contextlib.redirect_stdout(sys.stderr):
26
    importlib.import_module(module)
27
if loaded():
28
    raise AssertionError("Import loaded forbidden modules: " + ", ".join(loaded()))
29
print("IMPORT_CONTRACT_OK")
30
"""
31
    root = str(Path(source_root).resolve()) if source_root is not None else ""
32
    result = subprocess.run(
33
        [sys.executable, "-I", "-c", child, module, forbidden, root],
34
        capture_output=True,
35
        text=True,
36
        timeout=10,
37
    )
38
    if result.returncode != 0 or result.stdout != "IMPORT_CONTRACT_OK\n":
39
        raise AssertionError(
40
            f"Import contract failed for {module!r} (exit {result.returncode})\n"
41
            f"{result.stdout}{result.stderr}"
42
        )

There are a few choices here I’d keep even in a small test.

The tuple(sys.modules) gives us a snapshot of the names to iterate, following the dictionary’s documented guidance. The namespace comparison catches a dependency loaded indirectly through another helper, too.

The output redirection keeps ordinary import-time print() calls out of our completion signal. They go to standard error instead, where we can still read them if the test fails. Without that separation, a harmless startup message would make the exact-output check fail.

Finally, subprocess.run waits for the child and captures its output. A nonzero exit becomes an assertion with the diagnostics attached. If waiting times out, TimeoutExpired fails the test. Silence never becomes a pass.

Now we can express the actual boundary we care about in test_import_contract.py:

1
from pathlib import Path
2

3
from assert_no_import import assert_import_does_not_load
4

5

6
def test_runtime_import():
7
    assert_import_does_not_load(
8
        "runtime_helpers",
9
        "optional_toolkit",
10
        source_root=Path(__file__).resolve().parent,
11
    )
12

13

14
if __name__ == "__main__":
15
    test_runtime_import()
16
    print("PASS: runtime_helpers leaves optional_toolkit unloaded")

To try it, the four files go side by side: runtime_helpers.py, optional_toolkit.py, assert_no_import.py, and test_import_contract.py. From that directory:

python3 test_import_contract.py

The result is:

PASS: runtime_helpers leaves optional_toolkit unloaded

That’s the test script speaking after the helper succeeds. The child’s IMPORT_CONTRACT_OK stays inside the captured output.

Make sure it fails when it should#

The passing result is only half the demonstration. Let’s make the edit that worried us: move import optional_toolkit out of decorate and up to module level.

Run the same command and the failure includes:

AssertionError: Import loaded forbidden modules: optional_toolkit

Move it back, and the test passes again. Now we’ve seen it distinguish the two versions of our library, rather than merely accept the one we started with.

The other cases are useful for checking the test itself. These outcomes were exercised with small example modules:

The submodule-only case deliberately inserts a cache entry to check name matching. A normal package import isn’t being modeled there. The example and these controls were run on Python 3.12.12 and 3.9.6.

The unrelated-exception and early-exit rows are the ones I’d pay particular attention to. A test for something not happening needs to establish that it reached the point where it could have observed that thing.

What I’d take into a real library#

The helper’s two import names are the part you’ll change first: your target module, perhaps your_library.runtime_helpers, and the optional dependency’s top-level name. Use absolute import names.

Then choose what code the child should see. sys.executable selects the parent’s Python executable. The -I option ignores PYTHON* environment variables and leaves the current directory and user site-packages out of the child’s search path. Our explicit source_root adds the example directory back. In a source-tree test, that might instead be the project’s src directory.

If the question is about an installed distribution, omit source_root and use an interpreter where that distribution is available under -I. A package reachable only through PYTHONPATH or user site-packages won’t be found that way. Starting fresh and choosing an installation are separate decisions.

I’d keep this assertion’s promise narrow. It checks which module names remain after one import completes. It can’t show every import attempt that was caught, or a module that was loaded and later removed from the cache. It’s a regression test for cooperating code, not a security boundary.

There is also one more test I’d want before promising callers that the dependency is optional: run the basic operation in an environment where that dependency is absent. That environment test hasn’t been run here. Our probe checks whether importing the library leaves the extra module unloaded; the separate test checks whether callers can actually do their basic work without it.

For the original one-line edit, though, we now have a precise guard: the import must finish, and it must leave the optional feature’s dependency alone.