In a new study, Apple researchers detail SimpleDesign, a streamlined AI model that can jointly generate protein sequences and structures. Here are the details.
A bit of context
Last September, Apple researchers published a study titled “SimpleFold: Folding Proteins is Simpler than You Think,” which detailed a streamlined approach to predicting a protein’s 3D structure from its amino acid sequence.
In a nutshell, SimpleFold uses a flow-matching model to generate a protein’s 3D structure directly from its amino acid sequence.
We explained flow matching in more detail here, but the short of it is that the technique starts with a noisy, random basis and learns a relatively direct path toward the final result. That contrasts with diffusion models, which typically work by iteratively removing noise until they reach the final output.
Both techniques are most commonly (or at least historically) associated with image generation, although researchers (including those at Apple) have also explored diffusion models for text and code generation.
Back to SimpleFold, Apple basically paired flow matching with general-purpose Transformer blocks (commonly used in text generation), allowing the model to avoid some of the more computationally expensive techniques typically used by protein-folding models, such as DeepMind’s famous AlphaFold.
Now, Apple researchers have unveiled SimpleDesign, which applies the same push toward simpler, general-purpose architectures explored with SimpleFold to the broader problem of protein design, rather than just predicting a protein’s 3D structure.
SimpleDesign
As Apple’s researchers explain in a new study titled “SimpleDesign: A Joint Model for Protein Sequence and Structure Codesign”:
Existing models often rely on a multi-stage training process where autoencoders that tokenize data into latent representations are trained in a first stage. Secondly, a generative model is trained on the latent representation of the autoencoder(s), i.e. generative modeling in a latent space. We hypothesize that this multi-stage training is not necessary to obtain performant co-design models and thus present SimpleDesign, an effective multi-modal protein design model trained directly in the data space.
In other words, while many existing protein design models rely on a multi-stage process, SimpleDesign learns to generate amino acid sequences and continuous 3D structures in a single end-to-end training process.
Many current protein co-design models work as follows: first, they train a separate model to convert protein structures into discrete representations, or “tokens.” Then, they train a generative model to work with those representations to generate new protein sequences and structures.
SimpleDesign skips that intermediate step, learning directly from paired amino acid sequences and 3D coordinates rather than first compressing protein structures into a separate tokenized representation.
The way Apple researchers trained SimpleDesign is quite interesting.
They started with more than 2 million protein sequence-and-structure pairs, taken primarily from the AFESM dataset, which combines predicted structures from the AlphaFold Database and additional samples.
During training, they corrupted both parts of each pair: while amino acids in the sequence were randomly hidden behind masked tokens, the corresponding 3D structure also had noise added to it.
The researchers would also vary the extent to which each side was scrambled. If the sequence was mostly intact but the structure was heavily corrupted, the task resembled protein folding, and the model had to recover a structure from a known sequence.
By contrast, if the structure was mostly intact but the sequence was heavily masked, it resembled inverse folding. The model then had to generate a sequence capable of producing a given structure.
And when both were partially scrambled, the model learned to work on the two problems simultaneously, effectively training it for protein co-design.
According to the study, SimpleDesign delivered competitive results across protein co-design, structure generation, and sequence generation benchmarks, despite using a much simpler training pipeline.
The researchers also found that SimpleDesign could generate plausible protein structures, and that the amino acid sequences it produced were generally as good as or better than those generated by most competing multimodal models.
Finally, the researchers noted that SimpleDesign’s results are still limited to computer-based evaluations, since the generated proteins were not experimentally tested to confirm that they would actually fold, function, or behave safely in actual biological systems.
Still, the results were quite promising, and the entire study (which naturally goes much more in depth on SimpleDesign’s architecture, training process, benchmarks, and results) is well worth a look.
To read the full study, follow this link.
Worth checking out on Amazon
FTC: We use income earning auto affiliate links. More.