Validation & Evidence

What we’ve tested.

We tested whether the Experiments model could put versions of one protein in a useful order. We gave it part of a published dataset and checked its predictions against measurements we had set aside.

Model
Additive ridge v1
Dataset
GB1 · Wu et al., 2016
Evaluated
Evidence level
Comparison with previously published measurements

Ranking results and prediction limits

On average, the model did better at putting versions in order. For predicting the actual measured value, two simpler methods were closer on average. One used the most similar known sequence and one used the average training value. A new lab experiment would be needed to see whether the rankings help researchers choose what to test.

01 / Within the same scaffold

Could the model rank versions it had never seen during training?

For each of three fixed random seeds, we trained on 384 variants and evaluated 1,000 different variants from the same GB1 dataset. Each test version was left out of training. Similar versions of GB1 appear in both groups. This tells us how well the model ranked relatives of proteins it had already seen.

Averages across three ways of splitting the data. The original study measured how strongly each version was selected compared with the starting protein. Both folding and binding can affect that reading.
MethodRank correlation
Spearman; higher is better
Absolute error
MAE; lower is better
Top-24 outcome
Mean measured enrichment
Additive ridge0.3710.1450.664
Nearest training sequence0.3330.1120.495
Training meanUndefinedConstant predictions0.1300.125
Existing composition heuristic0.143Not comparableDifferent units0.287
Random-selection expectation——0.0777

Ridge rank correlation ranged from 0.323 to 0.420; top-24 measured outcome ranged from 0.481 to 0.925. The three seeds reuse one historical dataset and are not independent biological replications. Display values are rounded; downloads retain full precision.

MAE is expressed in relative-enrichment units. The top-24 column evaluates each method’s selected candidates using the held-out measurements. Ties use a fixed sequence hash, not the outcome. Random-selection expectation is the test-set mean, not a separately performed experiment.

02 / Beyond the measured mutation range

What happened when more letters changed?

We also trained on 384 variants with up to two substitutions and evaluated 1,000 variants with three or four substitutions, across the same three seeds.

Mutation-distance stress test. Means across three seeds.
MethodSpearmanMAE
Additive ridge0.2820.320
Nearest training sequence0.2450.285
Training meanUndefined0.335
Existing composition heuristic0.141Not comparable

03 / Errors worth seeing

The averages miss individual failures.

In one held-out split, variant FWAG measured 6.509 while the model predicted 0.438. In another, IYGL measured 3.708 while the prediction was −0.014. These four-letter codes refer to the study’s positions 39, 40, 41, and 54.

The model adds individual mutation effects but cannot capture interactions between mutations and can miss strong variants. Its unbounded estimates can be negative even for a nonnegative assay. Predictions include neither calibrated probabilities nor confidence intervals.

04 / Methods & provenance

A small model you can inspect.

What went into the benchmark

Only the 149,361 measured records in Supplementary file 1 of Wu et al. (2016), including zero values. We did not use the imputed outcomes in Supplementary file 2. The original workbook is identified by a fixed SHA-256 checksum.

The published measurement compares each version with the original protein (wild type). Both how the protein folds and how it binds to part of an antibody (IgG-Fc) can affect this measurement.

How the model was evaluated

Parent-referenced amino-acid indicators with L2 regularization, alpha 10, and an unpenalized intercept. Features come only from training sequences. We fixed the samples, seeds and alpha before inspecting test performance. The protocol documents a later correction to the numerical tolerance; the reported results did not change.

The implementation is licensed under MIT. There are no pretrained weights or inference-time external requests. In the workbench, each fit uses that project’s own assay measurements.

This evaluation concerns only the additive ridge model in Experiments and does not validate the existing sequence generator, structure predictions, or Disease Research output.

05 / Reproduce & review

Download the evidence.

These downloads contain only public benchmark data and code, with no private project records. Select a link to download a file.

Experimental data: Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun, Adaptation in protein fitness landscapes is facilitated by indirect paths, eLife 5:e16965 (2016), Supplementary file 1. Used under CC BY 4.0. DeltaFold extracted measured records, expanded sequence codes, sampled splits, and calculated model results. No author endorsement is implied.

Open Experiments

06 / Evidence Scope

Match the claim to its evidence.

A software check shows whether a tool worked as intended. Comparing predictions with published measurements shows how a model did on that dataset. To learn how a newly chosen protein behaves, a lab has to test it.

Recorded observations

Researchers supply the lab records. DeltaFold cannot verify how the lab obtained them. Campaigns keeps measured numbers separate from missing results, failed tests and readings above or below a limit. The spread of repeated readings describes those repeats. Experiments combines imported rounds and repeats. In Campaigns you can select one training batch and one type of repeat.

Predictions and heuristics

The GB1 evidence above concerns within-scaffold additive variant ranking only. Composition scores, sequence generation and evolution are heuristics without demonstrated functional improvement. Predicted structures and geometric contacts do not establish binding. No calibrated binding probabilities or therapeutic benefits are provided.

Source integrity

New structure responses must match the submitted sequence and pass bounded coordinate and identity checks before saving. Target evidence must match the selected target identifier. These checks prevent specific mismatches; they do not establish structural accuracy or biological truth. Historical results retain their original files and interpretation.

Evidence review

GammaFold checks supplied records and declared rules, including consistent frozen predictions and raw-value semantics. Its narrative screen remains heuristic and requires human review. A checked record or content hash is not a mathematical proof, authenticated experiment or permission to execute a proposal. Older checker verdicts remain historical.

No prospective laboratory validation is established here. New binding, functional or next-batch utility claims require a separately designed study with suitable controls, independent replication and analysis fixed before outcomes. Previously inspected GB1 results cannot become a fresh blind test for later model selection.

07 / Sources & Methods

Where the reference data and methods come from.

DeltaFold brings these resources into the research workflow. Their source identities stay with saved references and results. A database annotation, prediction or geometric check is distinct from a measurement of your candidate.

Reference proteins

Protein entries and annotations come from UniProt Consortium, under CC BY 4.0. DeltaFold selects fields for display and saves a dated snapshot with the chosen sequence, source links and annotation evidence. Reviewed entries can still contain computational inference.

A reference search sends the public terms and filters you enter to UniProt through DeltaFold. Inspection requests use the selected identifiers. Your working sequence is not used as search input.

Structure predictions

Structure prediction uses ESMFold through Meta's public ESM Metagenomic Atlas service. Selecting Predict Structure sends your working sequence to that service. The result retains its method identity; pLDDT describes local structure confidence, not measured function.

The ESM code is MIT licensed. DeltaFold does not distribute its model weights. The hosted service's availability and terms remain under the provider's control.

Target and disease evidence

Disease Research uses Open Targets for human target–disease associations. Only target searches and identifiers are sent there, not your protein sequence. Association scores describe evidence about the target; they do not establish activity or therapeutic benefit for a generated candidate.

Open Targets releases its platform data under CC0 and provides citation guidance. Individual evidence sources retain their own provenance and applicable terms.

Molecular execution

The own-worker pilot uses RFD4-Proteina. It supports four fixed public profiles: 1UBQ or 1VII at seed 42 or 43, with one 50-residue binder, one sample, the low preset and no hotspots. Each job requires your consent for transfer to your qualified worker. Other targets support preparation and import.

Runtime and checkpoint versions stay in the execution record. Code, weights and dependencies have separate license notices; this website does not distribute the weights. A completed run or plausible geometry is not laboratory validation.

Welcome illustration

The background double helix is a decorative drawing. The rotating protein beside the slogan comes from the published structure named in its PDB caption. Neither comes from the sequence you generate in Design. The helix is drawn in your browser without external requests. Its still image uses the same shape.

Earlier welcome film

The previous welcome background adapted Super-resolved Reflectance Confocal Microscopy time-lapse imaging of a living MEF cell lamellipod, by Alberto Aguilar, Adeline Boyreau and Pierre Bon (2021), under CC BY 4.0.

The footage and still frames are cropped, resized and compressed, with a crossfade added to the loop. These archived adaptations provide visual context, not DeltaFold experimental results or a basis for measurements. No creator endorsement is implied. The full media notice records the source and adaptations.