importlib: load a module from a file path

Introduction

import name looks for the module in the directories of sys.path: the current directory, the installed packages, the paths of the virtual environment. A file outside those directories has no import name, and a file whose directory is not on sys.path cannot be imported by name.

The standard library can still load it by its path. importlib.util takes a path, builds a module object and runs the file inside it.

Load a module by path

Three calls do the work: spec_from_file_location(name, path) reads the file header and returns a spec, module_from_spec(spec) creates the empty module object, and spec.loader.exec_module(module) runs the body of the file, so the names of the file land in the module.

The file plugins/hello.py:

NAME = "hello" def greet(who): return f"hello, {who}"

The file loader.py:

import importlib.util import sys from pathlib import Path path = Path("plugins/hello.py") spec = importlib.util.spec_from_file_location("hello_plugin", path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) print(module.__name__) print(module.NAME) print(module.greet("world")) print("hello_plugin" in sys.modules)

python loader.py

hello_plugin hello hello, world False

The first argument of spec_from_file_location is the name the module gets inside the program; it does not have to match the file name. The path can be a str or a pathlib.Path.

The module is not registered

The last line of the output is False: the new module is not put into sys.modules. That is deliberate. The import system keeps one module object per name there, and a second import of the same name returns the cached object. A module loaded from a path is usually a private copy, and the caller decides whether it should be visible under its name.

If the module must be importable later, register it by hand:

import sys sys.modules["hello_plugin"] = module

Do this only when the name is unique in the process. Replacing an entry that another part of the program already imported gives two different module objects for one name, and the classic symptom is a isinstance check that fails for an object of the same class.

When a path beats an import

A file loaded by path is useful when the program must read a file that is not a package: a test helper next to the tests, a plugin in a user directory, a script in the tools folder of the repository. The test suite of this site does exactly that: it loads container/tools/analyze_logs/log_paths.py by path, because the tools folder is not on sys.path and the test must check the module that the shell scripts use, not a copy of it.

Two details make that reliable. The path is built from Path(__file__).resolve(), so the test works from any working directory, and the call is wrapped in a small function, so every test gets its own module object and no test can change the module of another one.

The related article shows the test that does this: Testing a module loaded from a path .

Article author: Arthur Isaev