CellForge: My Trust Layer Said Everything Was Unreliable, and It Was Wrong

Perturbation-response models promise to predict how a cell changes when a gene is knocked down. Benchmarks for them are easy to misread. A model can look strong on a random cell split while failing on a perturbation it has never seen, and any prediction is only as trustworthy as the measurement it is scored against.

CellForge is my attempt to put that second point into software: a trust layer that sits between a perturbation screen and an experimental decision. This post covers the first real model run through it and the mistake it exposed in my own code.

The pipeline

Before any model earns trust, CellForge makes four things explicit:

• Validated data. Checks on matrix and index alignment, gene and guide identifiers, control presence, and which out-of-distribution holdouts the metadata can actually support.
• A deterministic unseen-perturbation split. Test perturbations are excluded from training and model selection.
• Simple baselines as first-class participants. A control-mean predictor, a training-mean predictor, and an embedding ridge model run under the same evaluation contract as the advanced model.
• A reliability gate. Each perturbation's measured response is graded before a model-versus-baseline comparison can raise trust.

I chose GEARS as the single advanced-model adapter for the first milestone because it documents a workflow on the Norman dataset.

The first run

The data was the official GEARS processed Norman file, Norman et al. 2019, GSE133344. I kept single-target perturbations with at least 50 cells, sampled them with a fixed seed, and capped the matrix at 20,000 cells: 19,977 cells, 5,045 genes, 7,353 controls. The split held out 10 perturbations. No model metric was used to pick any of this.

I ran GEARS for one epoch on a CPU. That is a deliberate verification of the whole path, not a tuned benchmark, and training plus inference took about 100 seconds.

• GEARS: mean Pearson 0.3286 on expression change, mean squared error 0.00650.
• Training-mean baseline: Pearson 0.5997, squared error 0.00536.
• Embedding ridge baseline: Pearson 0.6031, squared error 0.00538.
• Control mean: Pearson undefined, squared error 0.00664.

GEARS lost to both non-trivial baselines. It was below ridge on 9 of the 10 held-out perturbations, and FOXF1 was the worst case at 0.048 against 0.509. The control predictor has an undefined correlation because it predicts no change, so CellForge reports it as non-informative rather than inventing a number.

That is a sobering result, and exactly the kind CellForge exists to surface. A model whose training finishes is not a model that adds information.

The part I did not expect

The reliability layer classified all ten held-out perturbations as unreliable, so every GEARS trust record came out limited and none was trusted.

That should have made me suspicious. Ten out of ten is not what a measurement gate looks like when it is working. My rule divided the mean absolute effect by control-cell variance and applied a threshold of 1.5, computed directly on the processed matrix. It was a convenient heuristic, and I had labelled it as reliability without checking it against the published approach.

Replacing the heuristic

I reimplemented reliability as a split-half estimator. It splits a perturbation's cells in half many times, correlates the two halves' responses, and applies a Spearman-Brown correction. The settings were 100 seeded repetitions over the top 1,000 control-expressed genes, a reliability threshold of 0.5, and a shared-axis threshold of the cosine of 45 degrees. A perturbation is called specific, shared, or unreliable. No threshold was chosen after looking at GEARS scores or rankings.

On the same small matrix, which contains 38 unique single-gene targets after canonicalizing the labels, the new method gave:

• Specific: 30 of 38 targets (78.9%) on the small matrix and 76 of 105 (72.4%) on the larger one.
• Shared: 7 (18.4%) and 27 (25.7%).
• Unreliable: 1 (2.6%) and 2 (1.9%).

It did not reproduce the first run's ten out of ten. The earlier labels were an artifact of a mismatched heuristic, not a defensible replication of the published method.

The method was also reasonably stable. Across the 38 targets present in both preparations, class agreement was 92.1% and the reliability statistics correlated at 0.868. A cell-resampling bootstrap on four selected perturbations left the reliability gate unchanged in every case. Reliability does depend on sampling depth, and I would rather report that than hide it.

What changed for trust

I re-derived trust from the original, unchanged GEARS predictions using the new reliability labels. Only the reliability and trust interpretation were recomputed, and the original artifacts and decision package hash were left untouched.

One perturbation changed. CEBPA became trusted: its response is specific, with a split-half reliability of 0.99, and GEARS beats both baselines on it. The other nine stayed limited. CDKN1B and SAMD1 are classified as shared, and the remaining seven are ones where GEARS loses to ridge or the training mean.

That is a better outcome than either extreme. A broken gate that rejects everything is as unhelpful as one that accepts everything. The point of the gate is that reliability should condition trust, so a model can earn it on a perturbation where the measurement is solid and where it actually beats a simple method.

What this does and does not show

• It is a ten-target, one-epoch, CPU verification run. It does not show that GEARS is generally untrustworthy, and it is not a tuned benchmark.
• Only single-target perturbations were evaluated. Norman's combinations are deferred.
• No live Open Targets, DepMap, Reactome, or literature retrieval was used, and the external evidence in the decision package is fixture-only.
• The downloaded artifact has no machine-readable program labels, so there is no external ground-truth validation of the reliability classes. CEBPA is a reasonable qualitative sanity check, not a formal label.
• The validation conclusion is "valid with limitations".

What I took away

I built a layer to doubt other people's models and nearly trusted my own heuristic without the same scrutiny. The most valuable thing the first run produced was not the GEARS number. It was a result that looked too clean to believe, which sent me back to check the measurement rule.

The next steps are the full gene-by-perturbation prediction matrices, Norman's combinatorial perturbations, and a tuned run on a GPU. Until then, the project only claims what it measured. The code and experiment notes are in the repository.