Aletheia
LLM orchestration and evaluation platform with grounded retrieval
An AI engineering system for orchestrating, evaluating, and comparing LLM reasoning paths with LangGraph/LangChain, provenance tracking, retrieval, and measurable cost/quality tradeoffs.
Scientific LLM systems can produce confident reasoning without making retrieval, evidence use, uncertainty, and citation integrity independently inspectable.
Grounds claims in live scientific retrieval through Biolab MCP and stores provenance for each step.
Supports a single-call default path plus an explicit Advocate-Skeptic-Synthesizer evaluation path.
Rejects citations that are not present in retrieved evidence instead of silently accepting them.
Includes a real evaluation harness comparing the multi-agent design against a single-call baseline.
On the current 10-claim evaluation, the debate pipeline underperformed the single-call baseline on every measured metric while costing about 7.4x more; the system was changed to default to the measured-better path instead of hiding the negative result.