workflow-Engine
Durable Go workflow engine with idempotency and crash recovery
A distributed-systems project built around exactly-once step execution, retries, saga compensation, Redis Streams, and event-backed recovery after worker crashes.
A worker can crash after performing a side effect but before acknowledging the queue, causing a naive retry system to repeat completed business operations or lose workflow progress.
Derives deterministic SHA-256 deduplication keys from step inputs and checks the PostgreSQL event log before executing business logic.
Reclaims unacknowledged Redis Stream messages after SIGKILL and reconstructs completed state without rerunning finished steps.
Implements reverse-order Saga compensation with durable before/after checkpoints so rollback itself can resume after a crash.
Includes CLI inspection, a lightweight HTTP dashboard, Prometheus metrics, rate limiting, Docker deployment, and deterministic fault-injection tooling.
Passed 60/60 deterministic SIGKILL fault-injection runs across six crash points with zero double executions and zero lost steps; also passed 10/10 four-worker contention tests. Benchmarks on an Apple M2 measured 962 sequential and 2,884 concurrent steps/sec, 4.4ms p50 / 19.1ms p99 for a 5-step workflow, and about 90ms steady-state crash recovery.