Git-ignored images: a committed list file

Images are the part of a site that git does not carry. A .gitignore of the usual kind keeps *.jpg, *.jpeg, *.png, *.gif, *.webp, *.svg, *.ico, *.bmp, *.avif, *.tif and *.tiff out of the repository, so an image that is added, replaced or deleted changes nothing in git status, and a worktree or a Docker container that holds only the tracked files has almost no images. The pages render, the pictures do not, and a test that looks for the file behind every img tag has nothing to check.

This page shows a small Python script that keeps one committed text file instead: the SHA-1 hash, the size in bytes and the path of every image. The list does not put the images into git; it makes git show when one of them appears, changes or disappears.

The idea

One text file, container/docs/image-files.txt, holds one line per image: its sha1, its size in bytes and its path relative to container/src. The file is small enough to commit, the images stay out. On this tree:

The list is under a tenth of a percent of the bytes it describes and a hundredth of the repository. Two numbers per file are enough: the size changes when an image grows or shrinks, and the sha1 changes when it is replaced at the same size, under the same name.

The script

The script is container/tools/images/image_list.py, next to the other image tools. It takes its defaults from its own location, so it runs from any directory of the checkout; the paths on this page are written from the repository root. Three modes:

Four options: --src DIR scans another tree (default container/src), --out FILE writes another list (default container/docs/image-files.txt), --cache FILE keeps the cache elsewhere (default container/reports/image-list-cache.tsv), and --no-cache hashes every file and writes no cache. A bad argument, such as a --src that is not a directory, exits 2.

An image here is a regular file whose extension, in any letter case, is jpg, jpeg, png, gif, webp, svg, ico, bmp, avif, tif or tiff; symlinks are skipped. The lines are sorted by path in byte order, so the same tree always gives the same file. The whole script:

#!/usr/bin/env python3 # version 0.1.1 """Track the image files under container/src in a committed text file. The images are git-ignored (about 19,800 files, 2.3 GB), so git never shows them and a worktree or the OpenCode container holds almost none. This script writes container/docs/image-files.txt, a small file that is committed instead: # Image files under container/src: sha1, size in bytes, path relative to container/src. # Generated by container/tools/images/image_list.py; do not edit by hand. <sha1> <size> <path> One line per image file on disk, tracked or not, sorted by path in byte order. An image is a regular file whose extension (any letter case) is jpg jpeg png gif webp svg ico bmp avif tif tiff; symlinks are skipped. A path may hold spaces, so a reader splits a line on whitespace at most twice. Usage (any working directory): python3 container/tools/images/image_list.py --write # regenerate the list python3 container/tools/images/image_list.py --check # 1 when the list is stale python3 container/tools/images/image_list.py --hook # pre-commit gate --src DIR scans another tree, --out FILE writes another list, --cache FILE keeps the local sha1 cache (container/reports/, git-ignored). Hashing 2.3 GB on every commit is too slow, so a file whose size and mtime_ns match its cache entry reuses the cached sha1; --no-cache hashes everything. --hook works like --write and exits 1 when the file changed, so the commit stops and the new list can be staged. In a checkout without the images (a worktree, the OpenCode container) fewer than half of the listed paths exist on disk: --hook prints one line saying so and exits 0 without writing. Exit codes: 0 fine, 1 the list changed or is stale, 2 bad arguments. """ from __future__ import annotations import argparse import hashlib import os import sys import tempfile from dataclasses import dataclass from pathlib import Path CONTAINER = Path(__file__).resolve().parents[2] # The defaults: container/src is the tree, container/docs holds the list and # container/reports is git-ignored scratch space for the cache. DEFAULT_SRC = CONTAINER / "src" DEFAULT_OUT = CONTAINER / "docs" / "image-files.txt" DEFAULT_CACHE = CONTAINER / "reports" / "image-list-cache.tsv" # The extensions .gitignore drops, in any letter case. EXTENSIONS = frozenset({"jpg", "jpeg", "png", "gif", "webp", "svg", "ico", "bmp", "avif", "tif", "tiff"}) HEADER = "# Image files under container/src: sha1, size in bytes, path relative to container/src.\n# Generated by container/tools/images/image_list.py; do not edit by hand.\n" MAX_SHOWN = 20 # paths per category in the summary @dataclass(frozen=True) class Image: """What the list records about one file: its sha1 and its size in bytes.""" sha1: str size: int def _is_image_name(name: str) -> bool: """True when a file name ends in one of the image extensions, any case.""" return "." in name and name.rsplit(".", 1)[-1].lower() in EXTENSIONS def scan(src: Path) -> list[tuple[str, int, int, Path]]: """(path relative to src, size, mtime_ns, file) per image, in byte order. os.scandir is used instead of Path.rglob: 20,000 images must stay a second-scale job, and scandir reads the type of an entry from the directory itself. Symlinks (to a file or a folder) are not listed. """ found: list[tuple[str, int, int, Path]] = [] stack = [(str(src), "")] while stack: directory, prefix = stack.pop() with os.scandir(directory) as entries: for entry in entries: rel = prefix + entry.name if entry.is_symlink(): continue if entry.is_dir(follow_symlinks=False): stack.append((entry.path, rel + "/")) elif _is_image_name(entry.name) and entry.is_file(follow_symlinks=False): stat = entry.stat(follow_symlinks=False) found.append((rel, stat.st_size, stat.st_mtime_ns, Path(entry.path))) found.sort(key=lambda item: item[0].encode("utf-8")) return found def sha1_of(path: Path) -> str: """The SHA-1 of a file; file_digest reads in chunks, so a large image stays cheap.""" with path.open("rb") as handle: return hashlib.file_digest(handle, "sha1").hexdigest() CacheKey = tuple[int, int] # size, mtime_ns Cache = dict[str, tuple[int, int, str]] # path relative to src -> size, mtime_ns, sha1 def load_cache(path: Path) -> Cache: """The cache file as a dict; a missing or damaged file is an empty cache.""" cache: Cache = {} if not path.is_file(): return cache for line in path.read_text(encoding="utf-8").splitlines(): if not line or line.startswith("#"): continue fields = line.split("\t") if len(fields) != 4 or not fields[3]: continue try: cache[fields[0]] = (int(fields[1]), int(fields[2]), fields[3]) except ValueError: continue return cache def render_cache(entries: dict[str, Image], stats: dict[str, CacheKey]) -> str: """The cache file text: path, size, mtime_ns, sha1, one file per line.""" lines = ["# path\tsize\tmtime_ns\tsha1\n"] for rel in sorted(entries, key=lambda item: item.encode("utf-8")): size, mtime_ns = stats[rel] lines.append(f"{rel}\t{size}\t{mtime_ns}\t{entries[rel].sha1}\n") return "".join(lines) def collect(src: Path, cache: Cache | None) -> tuple[dict[str, Image], dict[str, CacheKey]]: """Every image under src with its size and sha1, plus the stat keys for the cache.""" entries: dict[str, Image] = {} stats: dict[str, CacheKey] = {} for rel, size, mtime_ns, path in scan(src): stats[rel] = (size, mtime_ns) hit = cache.get(rel) if cache is not None else None sha1 = hit[2] if hit is not None and hit[0] == size and hit[1] == mtime_ns else sha1_of(path) entries[rel] = Image(sha1, size) return entries, stats def render(entries: dict[str, Image]) -> str: """The list file text: the header, then one sha1, size and path per line.""" lines = [HEADER] for rel in sorted(entries, key=lambda item: item.encode("utf-8")): lines.append(f"{entries[rel].sha1} {entries[rel].size} {rel}\n") return "".join(lines) def parse_list(text: str) -> dict[str, Image]: """The entries of a list file text; comments, blank and damaged lines are skipped.""" entries: dict[str, Image] = {} for line in text.splitlines(): if not line or line.startswith("#"): continue fields = line.split(maxsplit=2) # a path may hold spaces if len(fields) != 3: continue try: entries[fields[2]] = Image(fields[0], int(fields[1])) except ValueError: continue return entries def load_list(path: Path) -> dict[str, Image]: """The entries of the list file; a missing file is an empty list.""" return parse_list(path.read_text(encoding="utf-8")) if path.is_file() else {} def relate(old: dict[str, Image], new: dict[str, Image]) -> tuple[list[str], list[str], list[str]]: """(added, removed, changed) paths, sorted, between the list and the disk.""" added = sorted(new.keys() - old.keys(), key=lambda item: item.encode("utf-8")) removed = sorted(old.keys() - new.keys(), key=lambda item: item.encode("utf-8")) changed = sorted((rel for rel in new.keys() & old.keys() if new[rel] != old[rel]), key=lambda item: item.encode("utf-8")) return added, removed, changed def _show(label: str, paths: list[str]) -> None: """One summary line: the first MAX_SHOWN paths of a category.""" more = f", ... ({len(paths) - MAX_SHOWN} more)" if len(paths) > MAX_SHOWN else "" print(f" {label}: {', '.join(paths[:MAX_SHOWN])}{more}") def report(out: Path, entries: dict[str, Image], differs: bool, old: dict[str, Image]) -> None: """Print the summary: the file count, then what changed and, in short, where.""" print(f"{out}: {len(entries)} image(s) on disk") if not differs: print(" unchanged") return added, removed, changed = relate(old, entries) print(f" {len(added)} added, {len(removed)} removed, {len(changed)} changed") for label, paths in (("added", added), ("removed", removed), ("changed", changed)): if paths: _show(label, paths) if not (added or removed or changed): print(" the text differs only in order or header") def write_atomic(path: Path, text: str) -> None: """Write `text` to a temp file next to `path`, then rename it into place.""" path.parent.mkdir(parents=True, exist_ok=True) fd, tmp = tempfile.mkstemp(dir=path.parent, prefix=f".{path.name}.", suffix=".tmp") try: with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle: handle.write(text) os.chmod(tmp, 0o664) # mkstemp creates 0600 os.replace(tmp, path) finally: Path(tmp).unlink(missing_ok=True) # gone already when the rename worked def lacks_images(out: Path, src: Path) -> tuple[bool, str]: """True when this checkout lacks the images. The list names the files of a full checkout. When fewer than half of them exist here (a worktree, the OpenCode container), regenerating the list would drop the rest, so --hook leaves it alone. """ listed = load_list(out) if not listed: return False, "" present = sum(1 for rel in listed if (src / rel).is_file()) if present * 2 >= len(listed): return False, "" return True, (f"{out}: this checkout holds {present} of {len(listed)} listed images, so it has no images; the list is left alone") def _parse_args(argv: list[str]) -> argparse.Namespace: parser = argparse.ArgumentParser( prog="image_list.py", description="Write or check the tracked list of the image files under container/src.", ) mode = parser.add_mutually_exclusive_group(required=True) mode.add_argument("--write", action="store_true", help="regenerate the list") mode.add_argument("--check", action="store_true", help="exit 1 when the list is stale, never write") mode.add_argument("--hook", action="store_true", help="pre-commit: write, exit 1 when the file changed") parser.add_argument("--src", type=Path, default=DEFAULT_SRC, metavar="DIR", help="scanned tree (default: container/src)") parser.add_argument("--out", type=Path, default=DEFAULT_OUT, metavar="FILE", help="list file (default: container/docs/image-files.txt)") parser.add_argument("--cache", type=Path, default=DEFAULT_CACHE, metavar="FILE", help="sha1 cache (default: container/reports/image-list-cache.tsv)") parser.add_argument("--no-cache", action="store_true", help="hash every file, ignore and keep no cache") return parser.parse_args(argv) def main(argv: list[str] | None = None) -> int: args = _parse_args(sys.argv[1:] if argv is None else argv) src: Path = args.src out: Path = args.out if not src.is_dir(): print(f"error: {src} is not a directory", file=sys.stderr) return 2 if args.hook: lacks, message = lacks_images(out, src) if lacks: print(message) return 0 cache_path: Path | None = None if args.no_cache else args.cache cache = load_cache(cache_path) if cache_path is not None else None entries, stats = collect(src, cache) text = render(entries) differs = not out.is_file() or out.read_text(encoding="utf-8") != text report(out, entries, differs, load_list(out)) if args.check: if differs: print("--check: the list is stale, run --write to regenerate it") return 1 return 0 if differs: write_atomic(out, text) print(f"wrote {out}") if cache_path is not None: write_atomic(cache_path, render_cache(entries, stats)) return 1 if args.hook and differs else 0 if __name__ == "__main__": sys.exit(main())

First run

The demo repository of this page has two images, img/logo.png and img/photo.jpg, and no list yet. The first run hashes everything and writes it:

python3.12 container/tools/images/image_list.py --write

/home/me/site/container/docs/image-files.txt: 2 image(s) on disk 2 added, 0 removed, 0 changed added: img/logo.png, img/photo.jpg wrote /home/me/site/container/docs/image-files.txt

The first line counts the images on disk, the second says what changed against the old list, the third names the first twenty paths of each kind and adds , ... (N more) beyond that. The real tree says 19764 image(s) on disk and takes about two minutes the first time. The file it wrote:

cat container/docs/image-files.txt

# Image files under container/src: sha1, size in bytes, path relative to container/src. # Generated by container/tools/images/image_list.py; do not edit by hand. 9c71545958fd6fde8eb0ae49e8f056fdbfcbd411 2048 img/logo.png ec43fa5da56b5e4684d9046320dcafd29766c3ac 4096 img/photo.jpg

Two comment lines, then one line per image: a 40-character sha1, two spaces, the size in bytes, two spaces, the path. A path may hold spaces, so a reader splits a line on whitespace at most twice. A list that matches the disk is what --check wants: unchanged and exit code 0; the section on the hook shows it on a tree with three images.

The pre-commit hook

A list helps only while it is no older than the images. The pre-commit hook of the pre-commit framework is one entry in .pre-commit-config.yaml at the repository root; pre-commit install writes the hook file once per clone:

vi .pre-commit-config.yaml

- repo: local hooks: - id: image-list name: image list (container/docs/image-files.txt) entry: python3.12 container/tools/images/image_list.py --hook language: system pass_filenames: false always_run: true

pass_filenames: false and always_run: true are the two lines that matter. The staged files almost never include an image, because git ignores them (the upper-case names of the last section are the exception), so a hook called only for changed files would almost never run. This one runs on every commit and looks at the disk. A second local hook, image-list-tests, runs only the tests of the script, pytest container/tests/tools/test_image_list.py -q, and only when container/tools/images/image_list.py changes. It does not test the image paths.

While the list matches the disk, the hook passes:

git commit -m "First commit"

image list (container/docs/image-files.txt)..............................Passed

Now an image is added and a page uses it. The image is ignored, so only the page is staged:

git status --short

M container/src/index.html

git commit -m "Adds the banner"

image list (container/docs/image-files.txt)..............................Failed - hook id: image-list - exit code: 1 - files were modified by this hook /home/me/site/container/docs/image-files.txt: 3 image(s) on disk 1 added, 0 removed, 0 changed added: img/banner.png wrote /home/me/site/container/docs/image-files.txt

The commit stopped, and this is the formatter pattern, not a fault of the script. The hook ran --write, the list on disk changed, and a hook that changes a file fails the run with files were modified by this hook: the change has to be seen and staged, otherwise it would slip into the history unseen, in a commit whose message says nothing about it. The same rule stops an unrelated commit while the list is stale, because the hook runs on every commit and a stale list means an image is already missing from it.

The hook left the list changed on disk but did not stage it:

git status --short

M container/docs/image-files.txt M container/src/index.html

The list is sorted by path, so img/banner.png lands first, before img/logo.png and img/photo.jpg; the last lines of the diff:

git diff -- container/docs/image-files.txt

+ba36c6c91209c9d7969297162a6ade0486ca9f96 3000 img/banner.png 9c71545958fd6fde8eb0ae49e8f056fdbfcbd411 2048 img/logo.png ec43fa5da56b5e4684d9046320dcafd29766c3ac 4096 img/photo.jpg

Staging the list and committing again is the whole fix:

git add container/docs/image-files.txt

git commit -m "Adds the banner"

image list (container/docs/image-files.txt)..............................Passed

The last lines of the log:

git log --oneline --stat -1

container/docs/image-files.txt | 1 + container/src/index.html | 1 + 2 files changed, 2 insertions(+)

The commit carries two files: the page and one line of text. The banner itself is still out of git, and the line proves it was there. On this tree the check agrees:

python3.12 container/tools/images/image_list.py --check

/home/me/site/container/docs/image-files.txt: 3 image(s) on disk unchanged

The exit code is 0, so a hook on the next commit passes.

The cache

Hashing 2.3 GB on every commit is too slow: the first --write takes 120.5 seconds, and a hook that costs two minutes is a hook people turn off. The script keeps the sha1 of the last run in container/reports/image-list-cache.tsv, a git-ignored file, and trusts it while the key of an entry still matches the file on disk. The key is the path plus two numbers: the size and the modification time in nanoseconds. When both match, the cached sha1 is reused and the file is not read; when either differs, the file is hashed again and the entry is rewritten. With a warm cache --check takes 0.66 seconds.

--no-cache hashes every file and writes no cache. Use it after a restore that may keep sizes and timestamps, or when the cache is in doubt. One case escapes the cache by design: a file replaced by one of the same size whose modification time was set back to the old value keeps the old sha1 until something hashes it again. --no-cache is that something. The cache is never committed; the list is.

Worktrees and containers

A second working tree has the tracked files and almost none of the images. Make one, step into it and run the hook there:

git worktree add ../site-wt -b wt

cd ../site-wt

python3.12 container/tools/images/image_list.py --hook

/home/me/site-wt/container/docs/image-files.txt: this checkout holds 0 of 3 listed images, so it has no images; the list is left alone

Fewer than half of the listed paths are on disk, so the hook prints one line and exits 0 without writing. Regenerating the list here would drop all 3 listed paths; in a worktree of the real tree, which holds 242 of the 19,764, it would drop 19,522. Either way the commit would carry a gutted list, so the hook leaves the file alone and the commit goes through untouched:

git commit -am "Edits the page in a worktree"

image list (container/docs/image-files.txt)..............................Passed

The same idea rescues the test of img paths. In a checkout without images it needs no image root: it reads container/docs/image-files.txt and counts a path that the list names as present. Without the list that test reports 4,048 missing files; with it, 243, the same 243 that a full checkout with the real images reports. The list says the file exists somewhere, not that the picture in it is the right one.

Variants and limits

The list records names, sizes and sha1 hashes, no pixels. What that catches is a replacement of the same size, which is the case two numbers exist for. img/photo.jpg replaced by another file of 4,096 bytes:

python3.12 container/tools/images/image_list.py --check

/home/me/site/container/docs/image-files.txt: 3 image(s) on disk 0 added, 0 removed, 1 changed changed: img/photo.jpg --check: the list is stale, run --write to regenerate it

The exit code is 1, which is what a CI job or a hook needs.

• No content. A fresh clone still has no images: the list says what should be there and what it should hash to, not where to get it. A backup or the copy on the hosting is checked against the same sha1 values.
• Case of the extension. On Linux the patterns of .gitignore are matched case-sensitively, so *.jpg does not match .JPG. On this site 120 .JPG files ended up tracked in git, which is why they are in the repository while their neighbours are not. The script lowercases the extension before it compares, so those files are in the list like any other.
• Git LFS. Large File Storage keeps a pointer in git and the bytes in a separate store. The images do enter the repository history that way, but every clone has to fetch 2.3 GB from the LFS server, and the server has to keep it. The list costs two megabytes and no server.
• A hook that only warns. The same script with exit 0 after a message is easier to live with and easy to miss in a long commit output; the list goes stale until someone reads the line.
• No hook. Run --check in CI, where a stale list fails the build. That reports the drift late but reports it every time, and the same command is the manual check.

Article author: Arthur Isaev