clair-algorithm

The indexing algorithm behind Clair's semantic news API: it turns news articles into the vectors that queries are matched against.

It is published so the matching can be checked rather than taken on trust. Every score the API returns comes from the embeddings this worker writes, using the open model and the chunking rules in this repository — so the same text, embedded the same way, yields the same vector and the same score for anyone who runs it.

What it does

A long-lived process that scans Postgres for articles which have been collected but not yet embedded, and embeds them. Per article it writes:

1. A whole-document vector (news_embeddings) — the coarse candidate stage.
2. Sentence-level chunk vectors (news_chunk_embeddings) — used to rerank those candidates against the best-matching sentence.

There is no LLM in the loop, no entity extraction, and no classification. Nothing is inferred about an article beyond its vectors.

The embedding contract

Matching only means something if both sides embed identically. The rules are fixed in src/embed.py:

Property	Value
Model	BAAI/bge-m3 (open weights)
Output	dense, 1024 dimensions
Normalization	L2 (unit length) — cosine == dot product
Instruction	none — queries and documents embed identically

Chunk boundaries are part of the contract too, and live in src/chunking.py: the title is always chunk 0, body sentences follow (capped at 60), fragments under 25 characters are glued onto the next sentence, and duplicates are dropped. Change any of this and vectors computed before the change stop corresponding to the same text — which is why every stored vector records the model tag it was produced under, and why changing that tag forces a re-embed instead of silently mixing two spaces.

Layout

main.py              tick loop
health.py            liveness endpoint
src/embed.py         the embedding contract (bge-m3)
src/chunking.py      sentence segmentation
src/indexer.py       the index stage — whole-doc + chunk vectors
src/work.py          which articles are pending
src/stage_state.py   per-article stage progress
src/embed_server.py  optional text -> vector endpoint
src/db.py            connection pool
src/config.py        env-driven config

Run it

Requires Postgres with pgvector and the tables the worker reads and writes (news_items, news_embeddings, news_chunk_embeddings, ingest_stage_state).

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
export DATABASE_URL=postgres://...
python main.py

Or with Docker, which bakes the model into the image so it boots offline:

docker build -t clair-algorithm .
docker run -e DATABASE_URL=postgres://... clair-algorithm

Configuration

Variable	Default	Meaning
DATABASE_URL	required	Postgres connection string
TICK_SECONDS	30	pause between scans
BATCH_SIZE	20	articles processed per scan
EMBED_BACKEND	local	local (in-process) or http
EMBED_ENDPOINT	—	bge-m3 endpoint, when EMBED_BACKEND=http
ST_BATCH	32	sentence-transformers encode batch size
PORT	8080	liveness endpoint
EMBED_PORT	8090	text -> vector endpoint
EMBED_SERVICE_SECRET	—	required to enable that endpoint; unset leaves it off

Embedding bge-m3 on a throttled shared CPU is slow. For a large backfill, give it dedicated cores and raise ST_BATCH.

Scope

This repository is the indexing algorithm only. Which sources are collected, and the archive of articles built up over time, are not part of it — the code here is deliberately ordinary, and it's the corpus that takes years to accumulate.

Support

Published as a reference for how Clair scores news, not as a general-purpose library. It's maintained for Clair's own use: issues and pull requests aren't actively monitored, and there are no release or compatibility guarantees. Fork it freely.

License

Apache-2.0