- Focus
- Trace-based observability and failure-level diagnosis for retrieval-augmented generation
- Inputs
- Versioned JSON traces containing queries, retrieved evidence, answers, metrics, and provenance
- Outputs
- Validated traces, failure labels, comparisons, benchmark summaries, and inspectable Markdown / HTML reports
- Status
- Research prototype — core reporting loop implemented and exercised on 40 public BEIR SciFact traces
Problem
A single aggregate score rarely explains why a RAG system failed. The source may be retrieval, reranking, evidence use, unsupported generation, or the evaluation setup itself. Without a shared trace contract, these failures are difficult to inspect across runs and easy to hide behind averages.
What I built
A local-first research prototype that turns RAG execution traces into inspectable artifacts. It validates trace structure, applies manual or heuristic failure labels, compares runs, and produces reports that preserve the evidence and provenance behind each conclusion.
The project is deliberately not another RAG pipeline or chatbot framework. Its role is the diagnostic layer around experiments: capture what happened, make failures reviewable, and keep comparisons reproducible.
Technical components
- Trace contract
- Versioned JSON schema for queries, retrieved passages, generated answers, metrics, labels, and run provenance
- Failure analysis
- Manual and heuristic categories spanning retrieval, evidence use, and generation errors
- Evidence inspection
- Claim-level support views and attribution-oriented diagnostics
- Run comparison
- Trace-to-trace comparisons, benchmark summaries, and conversation-level reports
- Reports
- Portable Markdown and HTML outputs with SVG previews for local inspection
- Evaluation
- Utilities for checking failure labels and quality dimensions rather than relying only on aggregate metrics
- Interoperability
- Offline OTLP/HTTP JSON ingestion for OpenInference retriever, reranker, and LLM spans
- Quality gates
- 107 automated tests plus Ruff, mypy, pre-commit, and container checks
Reproduce a small run
make reproduce-smallThe workflow produces normalized traces, human-readable reports, run comparisons, and a manifest suitable for review. Checked-in synthetic examples make the output format inspectable without requiring a full dataset-scale experiment.
Controlled SciFact retrieval-depth run
I also ran an actual retrieval-stage comparison over 20 fixed BEIR SciFact test queries. Deterministic BM25, the top-1 context policy, extractive generator, and qrels evaluator stayed fixed while retrieval_top_k changed from 1 to 5, producing 40 public traces.
| Configuration | Retrieval hit | Failure-labelled traces | Retrieval-stage failures |
|---|---|---|---|
top_k=1 |
13/20 | 7/20 | 7/20 |
top_k=5 |
14/20 | 20/20 | 20/20 |
The extra retrieval depth recovered one additional relevant result, while the current retrieval_noise heuristic fired on every top_k=5 trace because any irrelevant retrieved document triggers the label. The useful result is therefore not a clean victory metric: it exposes both a retrieval-depth trade-off and a limitation in the diagnostic heuristic.
Current status
Research prototype. Public trace conversion, OpenTelemetry/OpenInference JSON ingestion, stage-aware reports, configuration comparison, and a real SciFact smoke run are in place. The next study will examine chunk-boundary sensitivity under fixed evidence and evaluation settings.