Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Task (2302.06120v3)

Published 13 Feb 2023 in q-bio.QM and cs.LG

Abstract: RNA, whose functionality is largely determined by its structure, plays an important role in many biological activities. The prediction of pairwise structural proximity between each nucleotide of an RNA sequence can characterize the structural information of the RNA. Historically, this problem has been tackled by machine learning models using expert-engineered features and trained on scarce labeled datasets. Here, we find that the knowledge learned by a protein-coevolution Transformer-based deep neural network can be transferred to the RNA contact prediction task. As protein datasets are orders of magnitude larger than those for RNA contact prediction, our findings and the subsequent framework greatly reduce the data scarcity bottleneck. Experiments confirm that RNA contact prediction through transfer learning using a publicly available protein model is greatly improved. Our findings indicate that the learned structural patterns of proteins can be transferred to RNAs, opening up potential new avenues for research.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557):871–876, 2021.
  2. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35:32897–32912, 2022.
  3. The protein data bank. Nucleic acids research, 28(1):235–242, 2000.
  4. Simrna: a coarse-grained method for rna folding simulations and 3d structure prediction. Nucleic acids research, 44(7):e63–e63, 2016.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Protein structure determination from nmr chemical shifts. Proceedings of the National Academy of Sciences, 104(23):9615–9620, 2007.
  7. Atomic accuracy in predicting and designing noncanonical rna structure. Nature methods, 7(4):291–294, 2010.
  8. Robust deep learning–based protein sequence design using proteinmpnn. Science, 378(6615):49–56, 2022.
  9. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pp.  647–655. PMLR, 2014.
  10. Improved contact prediction in proteins: using pseudolikelihoods to infer potts models. Physical Review E, 87(1):012707, 2013.
  11. Esteller, M. Non-coding rnas in human disease. Nature reviews genetics, 12(12):861–874, 2011.
  12. Potent and specific genetic interference by double-stranded rna in caenorhabditis elegans. Nature, 391(6669):806–811, 1998.
  13. Glaeser, R. M. How good can cryo-em become? Nature methods, 13(1):28–32, 2016.
  14. Cross-lingual transfer of monolingual models. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp.  948–955, Marseille, France, June 2022. European Language Resources Association.
  15. Endogenous trna-derived fragments suppress breast cancer progression via ybx1 displacement. Cell, 161(4):790–802, 2015.
  16. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  17. Mutation effects predicted from sequence co-variation. Nature biotechnology, 35(2):128–135, 2017.
  18. Direct: Rna contact predictions by integrating structural patterns. BMC bioinformatics, 20(1):1–12, 2019.
  19. Non-linguistic supervision for contrastive learning of sentence embeddings. In Advances in Neural Information Processing Systems, 2022.
  20. Psicov: precise structural contact prediction using sparse inverse covariance estimation on large multiple sequence alignments. Bioinformatics, 28(2):184–190, 2012.
  21. Coarse-grained modeling of large rna molecules with knowledge-based potentials and structural filters. Rna, 15(2):189–199, 2009.
  22. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583–589, 2021.
  23. Template-based protein structure modeling using the raptorx web server. Nature protocols, 7(8):1511–1522, 2012.
  24. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
  25. ifoldrna v2: folding rna with constraints. Bioinformatics, 31(17):2891–2893, 2015.
  26. Rosetta3: an object-oriented software suite for the simulation and design of macromolecules. In Methods in enzymology, volume 487, pp.  545–574. Elsevier, 2011.
  27. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021.
  28. Evolutionary-scale prediction of atomic level protein structure with a language model. bioRxiv, 2022.
  29. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3431–3440, 2015.
  30. Pretrained transformers as universal computation engines. arXiv preprint arXiv:2103.05247, 2021.
  31. Cryo-em advances in rna structure determination. Signal Transduction and Targeted Therapy, 7(1):1–6, 2022.
  32. The traveling observer model: Multi-task learning through spatial variable embeddings. In ICLR, 2021.
  33. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021.
  34. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proceedings of the National Academy of Sciences, 108(49):E1293–E1301, 2011.
  35. adabmdca: adaptive boltzmann machine learning for biological sequences. BMC bioinformatics, 22(1):1–19, 2021.
  36. Infernal 1.1: 100-fold faster rna homology searches. Bioinformatics, 29(22):2933–2935, 2013.
  37. Learning music helps you read: Using transfer to study linguistic structure in language models. In EMNLP, pp.  6829–6839, 01 2020. doi: 10.18653/v1/2020.emnlp-main.554.
  38. Automated 3d structure composition for large rnas. Nucleic acids research, 40(14):e112–e112, 2012.
  39. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  40. Transformer protein language models are unsupervised structure learners. In International Conference on Learning Representations, 2021a.
  41. Msa transformer. In International Conference on Machine Learning, pp.  8844–8856. PMLR, 2021b.
  42. Moderna server: an online tool for modeling rna 3d structures. Bioinformatics, 27(17):2441–2442, 2011.
  43. Rna secondary structure prediction using deep learning with thermodynamic integration. Nature communications, 12(1):941, 2021.
  44. The petfold and petcofold web servers for intra-and intermolecular structures of multiple rna sequences. Nucleic acids research, 39(suppl_2):W107–W111, 2011.
  45. Biogenesis and function of trna fragments during sperm maturation and fertilization in mammals. Science, 351(6271):391–396, 2016.
  46. Redefining the invertebrate rna virosphere. Nature, 540(7634):539–543, 2016.
  47. Predicting 3d structure, flexibility, and stability of rna hairpins in monovalent and divalent ion solutions. Biophysical journal, 109(12):2654–2665, 2015.
  48. Predicting rna distance-based contact maps by integrated deep learning on physics-inferred secondary structure and evolutionary-derived mutational coupling. Bioinformatics, 38(16):3900–3910, 2022.
  49. Structure of rna and rna binding site in tobacco mosaic virus from 4-å map calculated from x-ray fibre diagrams. Nature, 267(5608):216–221, 1977.
  50. Enhanced prediction of rna solvent accessibility with long short-term memory neural networks and improved sequence profiles. Bioinformatics, 35(10):1686–1691, 2019.
  51. Rna inter-nucleotide 3d closeness prediction by deep residual neural networks. Bioinformatics, 37(8):1093–1098, 2021.
  52. Deep learning models for rna secondary structure prediction (probably) do not generalize across families. Bioinformatics, 38(16):3892–3899, 2022.
  53. Rna contact prediction by data efficient deep learning. Communications Biology, 6(1):913, 2023.
  54. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  55. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
  56. 3drna v2. 0: An updated web server for rna 3d structure prediction. International Journal of Molecular Sciences, 20(17):4116, 2019.
  57. GIT: A generative image-to-text transformer for vision and language. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https://openreview.net/forum?id=b4tMhpN0JC.
  58. Identification of direct residue contacts in protein–protein interaction by message passing. Proceedings of the National Academy of Sciences, 106(1):67–72, 2009.
  59. Vfold: a web server for rna structure and folding thermodynamics prediction. PloS one, 9(9):e107504, 2014.
  60. The i-tasser suite: protein structure and function prediction. Nature methods, 12(1):7–8, 2015.
  61. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3):1496–1503, 2020.
  62. pydca v1. 0: a comprehensive software for direct coupling analysis of rna and protein sequences. Bioinformatics, 36(7):2264–2265, 2020.
  63. Coconet—boosting rna contact prediction by convolutional neural networks. Nucleic acids research, 49(22):12661–12672, 2021.
  64. Co-evolution transformer for protein contact prediction. Advances in Neural Information Processing Systems, 34:14252–14263, 2021a.
  65. Rnacmap: a fully automatic pipeline for predicting contact maps of rnas by evolutionary coupling analysis. Bioinformatics, 37(20):3494–3500, 2021b.
  66. Automated and fast building of three-dimensional rna structures. Scientific reports, 2(1):1–6, 2012.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now