Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
124 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Diffusion on language model encodings for protein sequence generation (2403.03726v2)

Published 6 Mar 2024 in cs.LG, cs.AI, and q-bio.BM

Abstract: Protein sequence design has seen significant advances through discrete diffusion and autoregressive approaches, yet the potential of continuous diffusion remains underexplored. Here, we present DiMA, a latent diffusion framework that operates on protein LLM representations. Through systematic exploration of architectural choices and diffusion components, we develop a robust methodology that generalizes across multiple protein encoders ranging from 8M to 3B parameters. We demonstrate that our framework achieves consistently high performance across sequence-only (ESM-2, ESMc), dual-decodable (CHEAP), and multimodal (SaProt) representations using the same architecture and training approach. We extensively evaluate existing methods alongside DiMA using multiple metrics across two protein modalities, covering quality, diversity, novelty, and distribution matching of generated proteins. DiMA consistently produces novel, high-quality and diverse protein sequences and achieves strong results compared to baselines such as autoregressive, discrete diffusion and flow matching LLMs. The model demonstrates versatile functionality, supporting conditional generation tasks including protein family-generation, motif scaffolding and infilling, and fold-specific sequence design. This work provides a universal continuous diffusion framework for protein sequence generation, offering both architectural insights and practical applicability across various protein design scenarios.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Protein generation with evolutionary diffusion: sequence is all you need. bioRxiv, 2023. doi: 10.1101/2023.09.11.556673. URL https://www.biorxiv.org/content/early/2023/09/12/2023.09.11.556673.
  2. Basic local alignment search tool. Journal of molecular biology, 215(3):403–410, 1990.
  3. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
  4. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  22669–22679, 2023.
  5. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, 2020.
  6. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202, 2022.
  7. Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022. ISSN 0036-8075, 1095-9203. doi: 10.1126/science.add2187. URL https://www.science.org/doi/10.1126/science.add2187.
  8. What is hidden in the darkness? deep-learning assisted large-scale protein family curation uncovers novel protein families and folds. bioRxiv, 2023. doi: 10.1101/2023.03.14.532539. URL https://www.biorxiv.org/content/early/2023/03/19/2023.03.14.532539.
  9. Intrinsically unstructured proteins and their functions. Nature Reviews Molecular Cell Biology, 6(3):197–208, March 2005. ISSN 1471-0072, 1471-0080. doi: 10.1038/nrm1589. URL https://www.nature.com/articles/nrm1589.
  10. ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7112–7127, October 2022. ISSN 0162-8828, 2160-9292, 1939-3539. doi: 10.1109/TPAMI.2021.3095381. URL https://ieeexplore.ieee.org/document/9477085/.
  11. Difformer: Empowering diffusion model on embedding space for text generation. arXiv preprint arXiv:2212.09412, 2022.
  12. A kernel two-sample test. Journal of Machine Learning Research, 13(25):723–773, 2012. URL http://jmlr.org/papers/v13/gretton12a.html.
  13. Likelihood-based diffusion language models. arXiv preprint arXiv:2305.18619, 2023.
  14. Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control. arXiv preprint arXiv:2210.17432, 2022.
  15. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp.  6629–6640, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
  16. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  17. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 34:12454–12465, 2021.
  18. simple diffusion: End-to-end diffusion for high resolution images. arXiv preprint arXiv:2301.11093, 2023.
  19. Learning the protein language of proteome-wide protein-protein binding sites via explainable ensemble deep learning. Communications Biology, 6(1):1–15, January 2023. ISSN 2399-3642. doi: 10.1038/s42003-023-04462-5. URL https://www.nature.com/articles/s42003-023-04462-5.
  20. Learning inverse folding from millions of predicted structures. 162:8946–8970, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/hsu22a.html.
  21. Interproscan 5: genome-scale protein function classification. Bioinformatics, 30(9):1236–1240, 2014.
  22. Evaluation metrics for protein structure generation. ICML, 12(1), June 2023. ISSN 2041-1723. doi: 10.1101/2023.09.11.556673. URL https://icml.cc/virtual/2023/28971.
  23. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, August 2021. ISSN 1476-4687. doi: 10.1038/s41586-021-03819-2. URL https://www.nature.com/articles/s41586-021-03819-2.
  24. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):2577–2637, 1983.
  25. Karpathy, A. nanoGPT, 2023. URL https://github.com/karpathy/nanoGPT.
  26. Score-based generative modeling for de novo protein design. Nature Computational Science, 3(5):382–392, May 2023. ISSN 2662-8457. doi: 10.1038/s43588-023-00440-3. URL https://www.nature.com/articles/s43588-023-00440-3.
  27. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343, 2022.
  28. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. 2023.
  29. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023a. doi: 10.1126/science.ade2574. URL https://www.science.org/doi/abs/10.1126/science.ade2574.
  30. Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise. In International Conference on Machine Learning, pp.  21051–21064. PMLR, 2023b.
  31. Latent diffusion for language generation. arXiv preprint arXiv:2212.09462, 2022.
  32. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 41(8):1099–1106, August 2023. ISSN 1087-0156, 1546-1696. doi: 10.1038/s41587-022-01618-2. URL https://www.nature.com/articles/s41587-022-01618-2.
  33. Scop: a structural classification of proteins database for the investigation of sequences and structures. Journal of molecular biology, 247(4):536–540, 1995.
  34. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp.  8162–8171. PMLR, 2021.
  35. The superfamily 1.75 database in 2014: a doubling of data. Nucleic acids research, 43(D1):D227–D233, 2015.
  36. Structure-based protein design with deep learning. Current Opinion in Chemical Biology, 65:136–144, 2021. ISSN 1367-5931. doi: https://doi.org/10.1016/j.cbpa.2021.08.004. URL https://www.sciencedirect.com/science/article/pii/S1367593121001125. Mechanistic Biology * Machine Learning in Chemical Biology.
  37. Interpro in 2022. Nucleic acids research, 51(D1):D418–D427, 2023.
  38. Mobidb 3.0: more annotations for intrinsic disorder, conformational diversity and interactions in proteins. Nucleic acids research, 46(D1):D471–D476, 2018.
  39. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pp.  8599–8608. PMLR, 2021.
  40. Expanding functional protein sequence spaces using generative adversarial networks. Nature Machine Intelligence, 3(4):324–333, March 2021. ISSN 2522-5839. doi: 10.1038/s42256-021-00310-5. URL https://www.nature.com/articles/s42256-021-00310-5.
  41. Pseudolikelihood reranking with masked language models. CoRR, abs/1910.14659, 2019. URL http://arxiv.org/abs/1910.14659.
  42. Prot-vae: Protein transformer variational autoencoder for functional protein design. bioRxiv, 2023. doi: 10.1101/2023.01.23.525232. URL https://www.biorxiv.org/content/early/2023/01/24/2023.01.23.525232.
  43. Protein design and variant prediction using autoregressive generative models. Nature Communications, 12(1):2403, April 2021. ISSN 2041-1723. doi: 10.1038/s41467-021-22732-w. URL https://www.nature.com/articles/s41467-021-22732-w.
  44. Intrinsic structural dynamics dictate enzymatic activity and inhibition. Proceedings of the National Academy of Sciences, 120(41):e2310910120, October 2023. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.2310910120. URL https://pnas.org/doi/10.1073/pnas.2310910120.
  45. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp.  2256–2265. PMLR, 2015.
  46. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  47. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  48. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35(11):1026–1028, November 2017. ISSN 1546-1696. doi: 10.1038/nbt.3988. URL https://www.nature.com/articles/nbt.3988.
  49. Self-conditioned embedding diffusion for text generation. arXiv preprint arXiv:2211.04236, 2022.
  50. Highly accurate protein structure prediction for the human proteome. Nature, 596(7873):590–596, August 2021. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-021-03828-1. URL https://www.nature.com/articles/s41586-021-03828-1.
  51. Uversky, V. N. Functional roles of transiently and intrinsically disordered regions within proteins. The FEBS journal, 282(7):1182–1189, 2015.
  52. Fast and accurate protein structure search with Foldseek. Nature Biotechnology, May 2023. ISSN 1087-0156, 1546-1696. doi: 10.1038/s41587-023-01773-0. URL https://www.nature.com/articles/s41587-023-01773-0.
  53. Self-play reinforcement learning guides protein engineering. Nature Machine Intelligence, 5(8):845–860, 2023.
  54. De novo design of protein structure and function with RFdiffusion. Nature, 620(7976):1089–1100, August 2023. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-023-06415-8. URL https://www.nature.com/articles/s41586-023-06415-8.
  55. Protein structure generation via folding diffusion. 2022.
  56. Protein sequence design with deep generative models. Current Opinion in Chemical Biology, 65:18–27, 2021. ISSN 1367-5931. doi: https://doi.org/10.1016/j.cbpa.2021.04.004. URL https://www.sciencedirect.com/science/article/pii/S136759312100051X. Mechanistic Biology * Machine Learning in Chemical Biology.
  57. Dinoiser: Diffused conditional sequence learning by manipulating noises. arXiv preprint arXiv:2302.10025, 2023.
  58. Seqdiffuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:2212.10325, 2022.
  59. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702–710, December 2004. ISSN 0887-3585, 1097-0134. doi: 10.1002/prot.20264. URL https://onlinelibrary.wiley.com/doi/10.1002/prot.20264.
  60. Planner: Generating diversified paragraph via latent language diffusion model. arXiv preprint arXiv:2306.02531, 2023.
Citations (4)
View on Semantic Scholar

Summary

  • The paper introduces a diffusion-based framework that uses language model embeddings to generate realistic protein sequences.
  • It combines diffusion processes with advanced language representations to effectively capture complex protein sequence patterns.
  • Experimental results demonstrate significant performance gains over traditional methods, highlighting its potential in protein design research.

A Comprehensive Overview of Submission and Formatting Instructions for ICML 2024

Introduction

The annual International Conference on Machine Learning (ICML) stands as a pivotal gathering for scholars, researchers, and professionals within the machine learning domain to exchange insights, progress, and forecasts about the discipline's trajectory. The 2024 submission guidelines serve as a beacon of structure for prospective contributors, delineating the requisite formatting, submission procedures, and ethical considerations indispensable for deliberation in the conference proceedings. This blog post aims to simplify and elucidate these cardinal regulations, ensuring authors navigate the submission landscape with efficacy and compliance.

Electronic Submission and Preparations

Submissions for ICML 2024 pivot entirely on an electronic interface, dismissing any form of email or hard copy submissions. In a novel twist, appendices must be amalgamated with the main manuscript and references into a singular file for submission, adhering strictly to a PDF format. This consolidation aims at steering clear of oversight during the review process. Paramount details include:

  • PDF Format Exclusivity: The manuscript, inclusive of appendices, must abide strictly to a PDF format.
  • Page Limitations: An 8-page limit is enforced on the main body of the paper, with appendices and references permitted additional space. Authors of accepted papers will have the leverage to expand the main body by an extra page in their final submission.
  • Author Anonymity: Initial submissions must obscure author identities, a decree supporting the double-blind review ethos of ICML.

Style and Formatting Nuances

The document’s stylistic and typographic elements bear significant weight. The adherence to a 10 point Times font throughout the textual content is mandatory, punctuated by exacting specifications regarding figure captions, table placements, and the encapsulation of references. Critical notations entail:

  • Font Integrity: The mandatory use of Type-1 fonts to avert complications in the transcoding of the document.
  • Element Positioning: Specific directives on the placement and formatting of figures, tables, and references to maintain consistency and readability.
  • Reference Formatting: A chronological ordering in citations, with a comprehensive detailing including page numbers where feasible, ensuring a uniform presentation of the bibliography.

Ethical Compliance and Anonymity

ICML’s staunch commitment to ethical scholarliness is evident through its stringent policies on simultaneous submissions, which are summarily rejected if found to be under consideration elsewhere. The anonymity clause extends to censoring any form of author identification within the submission text, fostering an unbiased review process. Additionally, any form of prior work by the authors should be cited in a manner that preserves the review's blind nature.

Evaluation Matrices and Ablation Studies

The guidelines underscore a distinctive emphasis on rigorous empirical evaluation, with a performance comparison table delineated in the paper serving as a template. The inclusion of an ablation paper serves not only to benchmark the proposed DiMA model against prevailing architectures but also highlights the incremental enhancements afforded by various model iterations, evidencing a methodical approach to model refinement.

Implications and Theoretical Contributions

While maintaining a detached narrative, it’s paramount to underscore the paper’s theoretical and practical implications within the machine learning community. The quantitative leaps in performance metrics postulated by the DiMA model accentuate its potential for improving predictive accuracies in protein sequence modeling. Moreover, the theoretical underpinnings detailed in the model's architecture propose a novel paradigm that may spur further research within the domain.

Conclusion and Future Directions

In sum, the ICML 2024 submission and formatting instructions provide a detailed blueprint for authors to follow, ensuring their work is presented in a coherent and standardized manner. The guidelines are designed to facilitate a fair and rigorous review process, encouraging the submission of high-quality papers that advance the field of machine learning. Through adherence to these guidelines, authors can contribute to the rich tapestry of knowledge that ICML represents, pushing the boundaries of what is possible in machine learning research.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets