Blog Details

A genomics lab we worked with had a sequencing pipeline that took the better part of a week to go from raw signal files to a variant call file a clinician could actually read. The wet lab work, the actual sequencing, took a day. The rest was queueing, babysitting, and manually re-running steps that had failed in ways nobody noticed until three stages later. Our job was to compress that timeline into hours. This sounds like a straightforward infrastructure problem: more parallelism, better schedulers, GPU basecalling instead of CPU. It is not a straightforward infrastructure problem. Automating a pipeline that scientists used to babysit by hand removes the human who used to catch garbage before it propagated.

That turned out to be the real project. Every stage of a nanopore sequencing workflow, basecalling raw electrical signal into nucleotide sequences, demultiplexing reads back to the samples they came from, aligning and haplotyping to call variants, can fail silently. Not crash-and-log-an-error fail, but produce plausible-looking output that is quietly wrong. A basecaller with a corrupted model checkpoint still emits reads. A demultiplexer with a misconfigured barcode set still assigns reads to samples, just the wrong ones. If you only measure wall-clock time and whether the pipeline exited with status zero, you will ship a much faster pipeline that is worse than the slow manual one it replaced. This is the story of the checkpoints we built to prevent that.

Why 'the Job Finished' Was the Wrong Success Metric

The original pipeline was a chain of scripts kicked off by hand, each one written by a different postdoc over several years, each with its own idea of logging. When something went wrong, a human noticed because the output file was empty, or absurdly small, or a colleague flagged that a sample's results looked off compared to a previous run of the same cell line. That human-in-the-loop suspicion was, unglamorously, the pipeline's actual QC system.

When we started automating stage-to-stage handoffs with a workflow orchestrator, the first version we shipped was, by every infrastructure metric, a success: jobs completed, exit codes were clean, runtime dropped from days to hours. It was also wrong in ways we didn't discover for two weeks. A basecalling run on a partially degraded flow cell produced reads that were shorter and lower-quality than normal, but not zero. Nothing downstream was checking for that. The haplotyping step happily called variants from garbage data and produced a VCF file that looked completely normal on inspection.

That incident reframed the whole project for us. 'The job finished' is a necessary condition for a good pipeline, not a sufficient one. We needed the automated pipeline to reproduce the suspicion a tired postdoc would have felt looking at intermediate output, not just the willingness to continue. That meant instrumenting every stage boundary with a QC gate that could halt the pipeline and flag a human, rather than trusting that downstream stages would compensate for upstream problems.

Mapping Failure Modes Stage by Stage

Before writing any checkpoint code, we spent a couple of weeks with the wet lab and bioinformatics team cataloguing every failure mode they'd personally seen or heard about anecdotally, going back years. This felt slow at the time, but it paid for itself many times over. It turned 'add some quality control' into a concrete, prioritized list of specific things that had actually gone wrong in production, rather than a generic checklist copied from a paper.

Basecalling failures clustered around flow cell degradation, pore blocking, and model mismatches. For example, using a basecalling model trained for one chemistry version against reads from another produces syntactically valid but statistically off sequences. Demultiplexing failures clustered around barcode misassignment under low signal-to-noise conditions and index hopping, where reads bleed across samples that were sequenced together. Haplotyping and variant calling failures were subtler still: low coverage in specific genomic regions, reference genome mismatches, and heterozygous calls that were artifacts of alignment ambiguity rather than real biology.

We wrote each failure mode down alongside what its downstream symptom looked like, which was the more useful column. A basecalling problem doesn't announce itself at the basecalling stage. It shows up as unexpectedly low mapping rates two stages later, or as an implausible variant allele frequency three stages after that. Mapping failure mode to downstream symptom is what let us decide where each check actually needed to live, which was not always at the stage where the problem originated.

“A fast pipeline that nobody trusts isn't an improvement on a slow one. It's the same problem with better marketing.”

The Checkpoints That Made It Into Production

We settled on a small number of checkpoints per stage rather than an exhaustive battery of metrics, deliberately. Every additional check risks a false-positive halt that trains the team to ignore alerts. Alert fatigue was a real risk given how frequently the lab ran samples. The goal was the smallest set of metrics that would have caught every failure mode in our catalogue, tuned against a backlog of historical runs we knew the outcomes of.

At basecalling, we check read length distribution and mean quality score against a rolling baseline for that flow cell chemistry. We flag any run that falls more than a set number of standard deviations below recent history, rather than against a fixed global threshold, since 'normal' shifts with reagent batches. At demultiplexing, we check the ratio of successfully barcoded reads to total reads and cross-check barcode balance across the multiplexed samples, since a skew usually means one barcode is being systematically misread. At alignment and haplotyping, we check mean and per-region coverage against the regions of clinical interest specifically, not just genome-wide average coverage, because genome-wide numbers can look fine while the region a clinician actually cares about is thin.

Each checkpoint produces a machine-readable verdict: pass, warn, fail. It also logs the underlying numbers centrally, so a run's full QC history is visible in one place instead of scattered across log files from five different tools. A fail halts the pipeline and pages the on-call bioinformatician. A warn lets the run continue but flags the output for review before it's treated as clinically final, which turned out to be the more commonly used state in practice.

• Basecalling: read length distribution and mean Phred quality vs. rolling per-chemistry baseline
• Demultiplexing: percentage of reads successfully barcoded, plus barcode balance across the multiplexed pool
• Alignment: mapping rate and per-target-region coverage depth, not just genome-wide averages
• Haplotyping: variant allele frequency sanity bounds and concordance against a known-truth reference sample run periodically through the pipeline
• Cross-stage: read count reconciliation, confirming the number of reads entering each stage roughly matches what the prior stage reported emitting

What Didn't Work: Thresholds Borrowed From Papers and One Global Dashboard

Our first pass at thresholds came from published QC recommendations for nanopore sequencing, and it was a mistake to lean on them as heavily as we did. Published thresholds assume a generic setup. This lab's specific combination of flow cell chemistry, sample prep protocol, and target panel produced baseline numbers that were meaningfully different from the literature defaults in both directions. Using literature thresholds gave us false alarms on entirely normal runs for weeks. This was exactly the alert fatigue we were trying to avoid, and it undermined trust in the new system right when we needed people to trust it.

We also initially built a single dashboard that aggregated QC status across all stages into one green/yellow/red light per run. It looked good in demos but was actively unhelpful in practice. A yellow status collapsed 'basecalling quality was borderline' and 'coverage in a specific clinically relevant region was thin' into a single flat signal that told the reviewing scientist nothing about what to actually go check. They ended up digging into raw logs anyway, defeating the point of the dashboard.

The fix for both problems was the same underlying move: stop trying to abstract away domain expertise and instead surface it. We rebuilt thresholds from the lab's own historical run data, computing baselines per chemistry version and updating them as new reagent batches came in, with the bioinformatics team signing off on any threshold change. The dashboard became stage-specific, showing the actual metric, its trend against baseline, and a link to the specific reads or regions responsible for a warning, so a reviewer's first click already lands on the right piece of data instead of a generic status light.

From Days to Hours, Measured Against the Right Baseline

The infrastructure changes, GPU-accelerated basecalling, parallel demultiplexing across sample batches, and an orchestrator that ran independent stages concurrently instead of serially, got total pipeline time down from four to six days to under eight hours for a typical run. Most of that time is now basecalling itself, rather than anything we could realistically compress further. That number is the one that makes for a good headline, but it's not the number that mattered most to the lab.

The number they cared about was how many bad runs reached a clinician before someone caught them, and how much manual review time each run required. Before the checkpoint system, roughly one in eight runs needed manual rescue after a downstream problem was noticed late. This often took a full day or more of a bioinformatician's time retracing which stage had gone wrong. After, checkpoint fails or warns caught the equivalent issue at the stage where it originated in the overwhelming majority of cases, and manual review time per run dropped even as the number of runs per week went up substantially.

It's worth being honest that the checkpoints add real overhead. A run that would otherwise take seven hours now takes closer to eight because of the QC computation and the human-in-the-loop pause on warnings. Nobody on the team considers that a bad trade. An hour of added latency on every run is a much better deal than a full day of forensic debugging on one run in eight, and it's a better deal than the alternative universe where speed silently bought back the trust problem we'd started with.

Automating a Pipeline Means Automating Its Skepticism, Too

The mistake we almost made, and the mistake we see other teams make in adjacent domains, is treating automation and quality control as separable concerns. They build the fast pipeline first, then bolt on monitoring later if there's time. In any workflow where a domain expert used to eyeball intermediate results out of professional habit, that eyeballing was load-bearing, even if nobody had ever written it down as a requirement. Removing the human without replacing what the human was actually doing doesn't make the pipeline more automated. It makes it less trustworthy while looking more automated.

This generalizes well beyond genomics. Any multi-stage pipeline where errors compound silently, document processing feeding into a legal review, sensor data feeding into a predictive maintenance model, or image preprocessing feeding into a diagnostic classifier, has the same shape of risk. The speed gains from automation are real and worth pursuing, but they need to be earned stage by stage, with checkpoints calibrated against the specific system's own historical behavior rather than borrowed from someone else's.

The practical takeaway we now carry into every pipeline project is to interview the people currently doing the manual version before writing any orchestration code. We specifically ask what they've noticed go wrong and how they noticed it, not just what the official spec says should happen. That conversation is where the real requirements for validation checkpoints live, and skipping it is how you end up with a very fast pipeline that nobody who understands the domain actually trusts.