Validation & Evidence
What we’ve tested.
We tested whether the Experiments model could put versions of one protein in a useful order. We gave it part of a published dataset and checked its predictions against measurements we had set aside.
- Model
- Additive ridge v1
- Dataset
- GB1 · Wu et al., 2016
- Evaluated
- Evidence level
- Comparison with previously published measurements
Ranking results and prediction limits
On average, the model did better at putting versions in order. For predicting the actual measured value, two simpler methods were closer on average. One used the most similar known sequence and one used the average training value. A new lab experiment would be needed to see whether the rankings help researchers choose what to test.
01 / Within the same scaffold
Could the model rank versions it had never seen during training?
For each of three fixed random seeds, we trained on 384 variants and evaluated 1,000 different variants from the same GB1 dataset. Each test version was left out of training. Similar versions of GB1 appear in both groups. This tells us how well the model ranked relatives of proteins it had already seen.
| Method | Rank correlation Spearman; higher is better | Absolute error MAE; lower is better | Top-24 outcome Mean measured enrichment |
|---|---|---|---|
| Additive ridge | 0.371 | 0.145 | 0.664 |
| Nearest training sequence | 0.333 | 0.112 | 0.495 |
| Training mean | UndefinedConstant predictions | 0.130 | 0.125 |
| Existing composition heuristic | 0.143 | Not comparableDifferent units | 0.287 |
| Random-selection expectation | — | — | 0.0777 |
Ridge rank correlation ranged from 0.323 to 0.420; top-24 measured outcome ranged from 0.481 to 0.925. The three seeds reuse one historical dataset and are not independent biological replications. Display values are rounded; downloads retain full precision.
MAE is expressed in relative-enrichment units. The top-24 column evaluates each method’s selected candidates using the held-out measurements. Ties use a fixed sequence hash, not the outcome. Random-selection expectation is the test-set mean, not a separately performed experiment.
02 / Beyond the measured mutation range
What happened when more letters changed?
We also trained on 384 variants with up to two substitutions and evaluated 1,000 variants with three or four substitutions, across the same three seeds.
| Method | Spearman | MAE |
|---|---|---|
| Additive ridge | 0.282 | 0.320 |
| Nearest training sequence | 0.245 | 0.285 |
| Training mean | Undefined | 0.335 |
| Existing composition heuristic | 0.141 | Not comparable |
03 / Errors worth seeing
The averages miss individual failures.
In one held-out split, variant FWAG measured 6.509 while the model predicted 0.438. In another, IYGL measured 3.708 while the prediction was −0.014. These four-letter codes refer to the study’s positions 39, 40, 41, and 54.
The model adds individual mutation effects but cannot capture interactions between mutations and can miss strong variants. Its unbounded estimates can be negative even for a nonnegative assay. Predictions include neither calibrated probabilities nor confidence intervals.
04 / Methods & provenance
A small model you can inspect.
What went into the benchmark
Only the 149,361 measured records in Supplementary file 1 of Wu et al. (2016), including zero values. We did not use the imputed outcomes in Supplementary file 2. The original workbook is identified by a fixed SHA-256 checksum.
The published measurement compares each version with the original protein (wild type). Both how the protein folds and how it binds to part of an antibody (IgG-Fc) can affect this measurement.
How the model was evaluated
Parent-referenced amino-acid indicators with L2 regularization, alpha 10, and an unpenalized intercept. Features come only from training sequences. We fixed the samples, seeds and alpha before inspecting test performance. The protocol documents a later correction to the numerical tolerance; the reported results did not change.
The implementation is licensed under MIT. There are no pretrained weights or inference-time external requests. In the workbench, each fit uses that project’s own assay measurements.
This evaluation concerns only the additive ridge model in Experiments and does not validate the existing sequence generator, structure predictions, or Disease Research output.
05 / Reproduce & review
Download the evidence.
These downloads contain only public benchmark data and code, with no private project records. Select a link to download a file.
- Full results & split IDs JSON
Inspect fitted coefficients and individual errors alongside the metrics. Includes input hashes and convergence records.
- Held-out predictions CSV
All 6,000 test predictions and corresponding published outcomes.
- Evaluation Protocol Markdown
How the samples and comparisons were chosen, with eligibility rules, seeds and the numerical amendment.
- Model Card Markdown
What the model accepts, how it was evaluated and where its use is limited.
- Results & interpretation Markdown
A portable summary of these findings and failure cases.
- Reproduction source bundle ZIP
Python source, tests, MIT license, and instructions. No third-party Python packages required.
Experimental data: Nicholas C Wu, Lei Dai, C Anders Olson, James O Lloyd-Smith, and Ren Sun, Adaptation in protein fitness landscapes is facilitated by indirect paths, eLife 5:e16965 (2016), Supplementary file 1. Used under CC BY 4.0. DeltaFold extracted measured records, expanded sequence codes, sampled splits, and calculated model results. No author endorsement is implied.
Open Experiments06 / Evidence Scope
Match the claim to its evidence.
A software check shows whether a tool worked as intended. Comparing predictions with published measurements shows how a model did on that dataset. To learn how a newly chosen protein behaves, a lab has to test it.
Recorded observations
Researchers supply the lab records. DeltaFold cannot verify how the lab obtained them. Campaigns keeps measured numbers separate from missing results, failed tests and readings above or below a limit. The spread of repeated readings describes those repeats. Experiments combines imported rounds and repeats. In Campaigns you can select one training batch and one type of repeat.
Predictions and heuristics
The GB1 evidence above concerns within-scaffold additive variant ranking only. Composition scores, sequence generation and evolution are heuristics without demonstrated functional improvement. Predicted structures and geometric contacts do not establish binding. No calibrated binding probabilities or therapeutic benefits are provided.
Source integrity
New structure responses must match the submitted sequence and pass bounded coordinate and identity checks before saving. Target evidence must match the selected target identifier. These checks prevent specific mismatches; they do not establish structural accuracy or biological truth. Historical results retain their original files and interpretation.
Evidence review
GammaFold checks supplied records and declared rules, including consistent frozen predictions and raw-value semantics. Its narrative screen remains heuristic and requires human review. A checked record or content hash is not a mathematical proof, authenticated experiment or permission to execute a proposal. Older checker verdicts remain historical.
No prospective laboratory validation is established here. New binding, functional or next-batch utility claims require a separately designed study with suitable controls, independent replication and analysis fixed before outcomes. Previously inspected GB1 results cannot become a fresh blind test for later model selection.
07 / Sources & Methods
Where the reference data and methods come from.
DeltaFold brings these resources into the research workflow. Their source identities stay with saved references and results. A database annotation, prediction or geometric check is distinct from a measurement of your candidate.
Reference proteins
Protein entries and annotations come from UniProt Consortium, under CC BY 4.0. DeltaFold selects fields for display and saves a dated snapshot with the chosen sequence, source links and annotation evidence. Reviewed entries can still contain computational inference.
A reference search sends the public terms and filters you enter to UniProt through DeltaFold. Inspection requests use the selected identifiers. Your working sequence is not used as search input.
Structure predictions
Structure prediction uses ESMFold through Meta's public ESM Metagenomic Atlas service. Selecting Predict Structure sends your working sequence to that service. The result retains its method identity; pLDDT describes local structure confidence, not measured function.
The ESM code is MIT licensed. DeltaFold does not distribute its model weights. The hosted service's availability and terms remain under the provider's control.
Target and disease evidence
Disease Research uses Open Targets for human target–disease associations. Only target searches and identifiers are sent there, not your protein sequence. Association scores describe evidence about the target; they do not establish activity or therapeutic benefit for a generated candidate.
Open Targets releases its platform data under CC0 and provides citation guidance. Individual evidence sources retain their own provenance and applicable terms.
Molecular execution
The own-worker pilot uses RFD4-Proteina. It supports four fixed public profiles: 1UBQ or 1VII at seed 42 or 43, with one 50-residue binder, one sample, the low preset and no hotspots. Each job requires your consent for transfer to your qualified worker. Other targets support preparation and import.
Runtime and checkpoint versions stay in the execution record. Code, weights and dependencies have separate license notices; this website does not distribute the weights. A completed run or plausible geometry is not laboratory validation.
Welcome illustration
The background double helix is a decorative drawing. The rotating protein beside the slogan comes from the published structure named in its PDB caption. Neither comes from the sequence you generate in Design. The helix is drawn in your browser without external requests. Its still image uses the same shape.
Earlier welcome film
The previous welcome background adapted Super-resolved Reflectance Confocal Microscopy time-lapse imaging of a living MEF cell lamellipod, by Alberto Aguilar, Adeline Boyreau and Pierre Bon (2021), under CC BY 4.0.
The footage and still frames are cropped, resized and compressed, with a crossfade added to the loop. These archived adaptations provide visual context, not DeltaFold experimental results or a basis for measurements. No creator endorsement is implied. The full media notice records the source and adaptations.