Paperless to Qdrant ingestion worker

Problem

My scanned documents sit in Paperless, but my AI assistants could not search them. I wanted every document to be searchable from the shared assistant memory, without paying for an API call on each one.

Approach

• I wrote a worker that runs once, does its job and exits. It has no scheduler and no restart policy on purpose.
• It reads the text that Paperless already extracted, so it does no OCR of its own.
• It calls no language model. A local embedding model turns each piece of text into a vector, so indexing runs on the home server's CPU and costs nothing per document.
• I split the code into small pieces with one job each, so I can test and change them separately.
• I run it every hour from the host's cron.

Architecture and design decisions

• The worker pulls documents from the Paperless REST API, cuts the text into chunks, embeds the chunks and stores them in the shared Qdrant memory collection.
• PaperlessClient handles paged and retried access to the API. EmbeddingClient sends chunks in batches. QdrantStore writes the vectors. StateStore keeps the checkpoint.
• IngestionMap decides which memory domain a document belongs to. It maps Paperless tags to domains through a YAML file.
• The Chunker counts tokens with the tokenizer of the embedding model itself. It packs whole sentences, splits very long OCR sentences by force and adds overlap between chunks.
• The embedding model is BAAI/bge-m3. Each chunk becomes a vector with 1024 dimensions.
• Every point gets an ID built from the document number and the chunk number. The same chunk always gets the same ID.

Reliability and performance

• Because the IDs never change, running the same document twice gives the same result. Re-indexing is safe.
• When a document gets shorter, the worker deletes exactly the chunks it no longer produces.
• Incremental runs only look at documents newer than the last checkpoint.
• The checkpoint is written to a temporary file, flushed to disk and then swapped in as one step. An interrupted run cannot leave a half written file.
• A failed run never moves the checkpoint forward, so the missed documents are tried again and never skipped.
• A full resync compares the document IDs in Paperless with the points in Qdrant and removes what was deleted.
• A dry run with a small limit shows what would happen without writing anything.
• The cron entry runs under flock, so two runs can never overlap.

Outcomes

• My whole document archive can be found by the shared assistant memory.
• Indexing costs nothing per document, because the embeddings run locally.
• Re-running, failing and resuming all leave the index in a correct state.
• The index stays in line with Paperless, including deleted documents.
• The repository is public.

Known gaps

• I do not give numbers for how many documents or chunks it handles, or how long a run takes.
• I have not measured how good the search results are.
• I do not say which chunk size and overlap I use or why.
• I do not describe how a change to an existing document is picked up by an incremental run.
• It only works as well as the text that Paperless extracted. Bad OCR means bad search.
• It runs on one machine for one household. I have not tested it at larger scale.