FlowCast
Pipeline diagnostics and observability for data-intensive workflows
A diagnostics system that combines execution traces, metrics, anomaly detection, and evidence-grounded explanations to make complex pipeline failures easier to debug.
RNA-seq pipeline diagnostics often jump from a suspicious metric to a confident root-cause story even when the execution evidence does not support that conclusion.
Parses execution_trace.txt and MultiQC STAR metrics from real nf-core/rnaseq runs.
Uses a modified z-score rule for unmapped-too-short outliers and a separate trace-only rule for FAILED tasks.
Constrains the LLM narrator to Observed, Reported, or Unknown claims tied to explicit evidence sources.
Writes Go and Python narrator events into one SQLite log and replays them in timestamp order; OpenTelemetry spans instrument pipeline stages.
Verified on a real nf-core/rnaseq run with 208 tasks and 5 STAR samples: WT_REP1 was flagged at modified z-score 5.46 while all others were <= 0.86. A deliberately memory-capped STAR_ALIGN run produced a genuine exit-137 failure after 4m33s, which the failure classifier and narrator diagnosed end to end. CI build, vet, and tests are green.