Two-level cache

Each project gets a .db (SQLite — chunks + metadata + reference graph) plus a matching .tq (TurboQuant — dense vectors) under ~/.pydocs-mcp/. The SQLite filename is {dirname}_{path_hash}.db, where path_hash is a 10-char slug of the absolute project path, so two projects in different directories never share state.

Skip when nothing changed

Subsequent indexing runs do a quick metadata scan and skip when nothing changed (typically <100 ms):

• For every package (your project + each dep), the indexer collects (file_path, mtime) pairs, joins them into one buffer, and hashes it — xxh3-64 in the Rust build, a truncated MD5 fingerprint in the pure-Python fallback — → stored in packages.content_hash.

• Before re-indexing a package, it recomputes the hash and compares. Match → skip the whole package (no parsing, no chunking, no embedding, no writes). Mismatch → re-extract that package only.

• mtime + path (not file contents) is the signal: cheap to read in bulk, and the file tree is the source-of-truth. Reading contents would defeat the speed goal.

Chunk-level diff-merge

When a package is re-extracted, work happens at chunk granularity. Each chunk carries a content_hash:

content_hash = SHA-256( length-prefixed fields:
                        package, module, title, text, "{pipeline_hash}|tier:{tier}" )

Each field is length-prefixed (len(field)\0field) rather than joined on a bare separator, so two distinct identity tuples can’t realign into a collision; the |tier:… suffix is the embed-tier fold described under Selective dependency embedding.

IndexingService.reindex_package diffs incoming chunks against the persisted set by this hash: unchanged chunks keep their row and their vector, removed chunks are wiped atomically from both stores via the CompositeUnitOfWork, and only added chunks are re-embedded.

The pipeline_hash slot is what makes model swaps automatic:

pipeline_hash = SHA-256( embedder identity (provider, model_name, dim, bit_width,
                           max_seq_length, normalize — plus backend /
                           model_file_name / pooling when non-default)
                         |  search-backend identity
                         |  late-interaction identity (only when the ingestion
                            YAML uses the multi-vector embed stage)
                         |  ingestion.yaml raw bytes )

Any change to the embedder config or any edit to ingestion.yaml (even whitespace — raw bytes are hashed, deliberately conservative) changes pipeline_hash, which changes every chunk’s content_hash. The diff-merge then sees all chunks as “added” and re-embeds through the normal add path — no separate “force re-embed” code path, no manual cache wipe.

Clearing

pydocs-mcp index . --force calls IndexingService.clear_all, which wipes SQLite + TurboQuant atomically through the same CompositeUnitOfWork — no half-deleted state if a crash lands mid-clear. Both files are always rebuildable from source, so deleting ~/.pydocs-mcp/*.db and the matching .tq is always safe.