Fix hybrid-mode scoring: log a cosine relevance score, not an RRF rank score

[jpr5]

Production runs search_mode: hybrid (deploy/copilotkit-docs.yaml), and in that mode query_log.top_score was being fed a Reciprocal Rank Fusion score instead of a relevance score. rrfMerge overwrites ChunkResult.similarity with the fused rank value, whose ceiling is 2/(RRF_K+1) ≈ 0.0328, and the tools then did Math.max(...results.map(r => r.similarity)).

Three things broke off that one line:

1. The dashboard's "Avg Score" column rendered a unitless rank score as if it were a cosine — every row read 0.02–0.03, carrying no information.
2. LOW_CONFIDENCE_SCORE_THRESHOLD = 0.5 was compared against a metric capped at 0.033, so every scored query was flagged low-confidence — a 100% false-positive rate.
3. min_score was documented as "Minimum similarity score (0-1). Results below this threshold are filtered out", which is not what it does in hybrid mode: it gates only the vector candidates, before fusion, so a keyword-only hit can be returned below the floor.

What changed

Scoring is a shared mechanism (tool response → query_log → dashboard → low-confidence flag → min_score semantics), so the fix names the two scales rather than patching one call site. RRF is untouched as a ranking device.

• ChunkResult now carries two numbers. similarity stays the per-retriever RANKING score (cosine / ts_rank / RRF). New cosine_similarity is the RELEVANCE score — a real cosine in [0,1] from the vector retriever, explicitly null for keyword-only hits — and it survives rrfMerge untouched.
• topCosineScore() (new src/relevance.ts) reduces a result set to the best cosine, or null. Both the search and knowledge tools log that as top_score. Keyword mode consequently logs NULL rather than a ts_rank; the analytics layer already reads NULL as "no score", never as "a low score", so this records an honest absence instead of a number on the wrong scale.
• LOW_CONFIDENCE_SCORE_THRESHOLD is derived — COSINE_SCORE_MAX * 0.5 — so it cannot drift onto a scale the metric doesn't live on. Value unchanged at 0.5.
• The dashboard column is relabelled "Avg Cosine" with a tooltip; blank now means "no cosine recorded", not "zero".
• min_score's description states what it actually gates, per mode.

src/relevance.ts exists rather than parking the reducer in db/queries.ts for two reasons: it is a pure function with no DB dependency, and several suites mock db/queries.js wholesale — a helper living there would have been silently undefined inside them.

Historical query_log rows

Existing rows hold RRF-scaled (and, in keyword mode, ts_rank-scaled) values under the old semantics. Those scales are not recoverable per-row — an RRF 0.0164 carries no derivable cosine — and inferring one from the magnitude would be exactly the silent reinterpretation to avoid.

So: a nullable score_kind column declares the scale of top_score, and there is deliberately no backfill. New rows are tagged 'cosine' (derived at the write boundary from top_score, so the two can never disagree); legacy rows stay NULL = "unknown scale". Both score-based readers — the low-confidence FILTER in getAnalyticsSummary and avg_top_score in getTopQueries — now require score_kind = 'cosine', so history is excluded rather than misread.

Operator-visible consequence: on the existing deployment the low-confidence counter and the Avg Cosine column start empty and refill as new traffic lands. That is intended — an empty honest metric beats a full dishonest one. Every other card (totals, empty-result rate, latency, by-source, per-day) is unaffected.

The migration is an additive nullable ADD COLUMN IF NOT EXISTS, matching the existing request_source and v1.15.2 attribution columns. The GREEN run below exercised it against a database created before the column existed, so the in-place upgrade path is covered.

Gap Reports

I traced this before assuming it. scripts/gap-analysis/monthly-gap-analysis.ts builds its LLM prompt from total_queries_window, empty_result_rate_window, empty_result_count_window, and the top-queries / empty-queries clusters — it never reads low_confidence_* or avg_top_score (both are carried in the TypeScript contract in cluster.ts / weekly-search-report.ts but are not rendered). The bi-weekly Gap Reports were therefore not contaminated by the false-positive low-confidence signal and do not need a re-run. The blast radius was the /analytics dashboard and /api/analytics/* consumers.

---

LOCAL RED-GREEN PROOF

Real surface, not a fake: real OpenAI text-embedding-3-small embeddings → real PGlite+pgvector index over the actual production docs corpus (CopilotKit/showcase/shell-docs/src/content/docs, 656 .mdx files → 2,999 chunks) → the real registered MCP search tool in search_mode: hybrid with min_score: 0.3 (mirroring deploy/copilotkit-docs.yaml exactly) → real hybridSearchChunks/rrfMerge → real logQuery → real query_log → real getAnalyticsSummary / getTopQueries.

Identical index, identical queries, identical harness across both runs. true top cosine is computed independently of the tool and is the ground truth the metric should report.

RED (before the fix)

search_mode        : hybrid
min_score (config) : 0.3
chunks indexed     : 2999
low-conf threshold : 0.5 (src/db/analytics.ts)

[GOOD-A ] "useCopilotAction"
   true top cosine    : 0.59517   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.01639
   score_kind         : NULL
   low-confidence?    : YES  <-- flagged

[GOOD-B ] "how do I stream agent state to the frontend"
   true top cosine    : 0.69179   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.01639
   score_kind         : NULL
   low-confidence?    : YES  <-- flagged

[POOR   ] "how do I configure kubernetes ingress TLS certificates for m…"
   true top cosine    : 0.49535   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.01639
   score_kind         : NULL
   low-confidence?    : YES  <-- flagged

[GARBAGE] "The sourdough starter should be fed a 1:2:2 ratio of starter…"
   true top cosine    : 0.18586   <-- ground truth
   result_count       : 0
   top_score (logged) : NULL
   score_kind         : NULL
   low-confidence?    : no

--- getAnalyticsSummary (real reader) ---
   total_queries_window        : 4
   low_confidence_count_window : 3
   low_confidence_rate_window  : 75.0%

--- getTopQueries (dashboard "Avg Score" column) ---
     0.02  "how do I configure kubernetes ingress TLS certificates …"
     0.02  "how do I stream agent state to the frontend"
     0.02  "useCopilotAction"

An excellent match (useCopilotAction, true cosine 0.595) logs 0.01639 and is flagged low-confidence. So does a mediocre one, and so does a genuinely poor one — all three log the identical value. The metric carries zero information, and 100% of scored queries are flagged.

GREEN (after the fix, same index, same queries)

search_mode        : hybrid
min_score (config) : 0.3
chunks indexed     : 2999
low-conf threshold : 0.5 (src/db/analytics.ts)

[GOOD-A ] "useCopilotAction"
   true top cosine    : 0.59517   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.59517
   score_kind         : cosine
   low-confidence?    : no

[GOOD-B ] "how do I stream agent state to the frontend"
   true top cosine    : 0.69179   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.69177
   score_kind         : cosine
   low-confidence?    : no

[POOR   ] "how do I configure kubernetes ingress TLS certificates for m…"
   true top cosine    : 0.49535   <-- ground truth
   result_count       : 5
   top_score (logged) : 0.49534
   score_kind         : cosine
   low-confidence?    : YES  <-- flagged

[GARBAGE] "The sourdough starter should be fed a 1:2:2 ratio of starter…"
   true top cosine    : 0.18586   <-- ground truth
   result_count       : 0
   top_score (logged) : NULL
   score_kind         : NULL
   low-confidence?    : no

--- getAnalyticsSummary (real reader) ---
   total_queries_window        : 4
   low_confidence_count_window : 1
   low_confidence_rate_window  : 25.0%

--- getTopQueries (dashboard "Avg Cosine" column) ---
     0.50  "how do I configure kubernetes ingress TLS certificates …"
     0.69  "how do I stream agent state to the frontend"
     0.60  "useCopilotAction"

The logged score now tracks ground truth to five decimals and discriminates: 0.60 / 0.69 for the good matches (not flagged) vs 0.50 for the off-topic one (flagged). Low-confidence rate drops from a meaningless 75% to a defensible 25% — the one query that genuinely has no good answer in the corpus. The dashboard column shows three distinct, interpretable numbers instead of three identical 0.02s.

Regression tests

src/__tests__/relevance-score-scale.test.ts (17 tests) pins the invariant at three levels: the reducer, the live hybrid tool → logQuery path driven through the real rrfMerge, and the SQL readers against real Postgres via PGlite.

The load-bearing one is "DISCRIMINATES between an excellent match and a poor one": reintroducing Math.max(...similarity) collapses a 0.87 match and a 0.21 match to the same ~0.0164 and fails on both the inequality and the classification. There is also an explicit scale assertion — the RRF ceiling must be more than an order of magnitude below the threshold — which states the arithmetic of this bug as a test.

Mutation-checked: reverting the tool to Math.max(...similarity) and dropping the score_kind guards fails 7 of these tests. They are not vacuous.

Local gate

prettier --check · tsc --noEmit · tsc --noEmit -p tsconfig.scripts.json · check-version-sync.sh · npm test (3573 passed, 1 skipped, 179 files) · npm run build — all green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WX4mtASFe6K4TnJBhRhMNE