Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Molecule Joint Auto-Encoding: Trajectory Pretraining with 2D and 3D Diffusion (2312.03475v1)

Published 6 Dec 2023 in cs.LG, cs.AI, and q-bio.BM

Abstract: Recently, artificial intelligence for drug discovery has raised increasing interest in both machine learning and chemistry domains. The fundamental building block for drug discovery is molecule geometry and thus, the molecule's geometrical representation is the main bottleneck to better utilize machine learning techniques for drug discovery. In this work, we propose a pretraining method for molecule joint auto-encoding (MoleculeJAE). MoleculeJAE can learn both the 2D bond (topology) and 3D conformation (geometry) information, and a diffusion process model is applied to mimic the augmented trajectories of such two modalities, based on which, MoleculeJAE will learn the inherent chemical structure in a self-supervised manner. Thus, the pretrained geometrical representation in MoleculeJAE is expected to benefit downstream geometry-related tasks. Empirically, MoleculeJAE proves its effectiveness by reaching state-of-the-art performance on 15 out of 20 tasks by comparing it with 12 competitive baselines.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (76)
  1. Strategies for pre-training graph neural networks. arXiv preprint arXiv:1905.12265, 2019.
  2. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019.
  3. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1857–1867, 2020a.
  4. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, ICLR, 2020.
  5. Molecular geometry pretraining with SE(3)-invariant denoising distance matching. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openreview.net/forum?id=CjTHVo1dvR.
  6. Pre-training via denoising for molecular property prediction. arXiv preprint arXiv:2206.00133, 2022.
  7. Pre-training molecular graph representation with 3d geometry. arXiv preprint arXiv:2110.07728, 2021a.
  8. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. In International Conference on Machine Learning, pages 21497–21526. PMLR, 2023b.
  9. Symmetry-informed geometric representation for molecules, proteins, and crystalline materials. arXiv preprint arXiv:2306.09375, 2023c.
  10. Graph self-supervised learning: A survey. arXiv preprint arXiv:2103.00111, 2021b.
  11. Self-supervised learning of graph neural networks: A unified review. arXiv preprint arXiv:2102.10757, 2021.
  12. Self-supervised on graphs: Contrastive, generative, or predictive. arXiv preprint arXiv:2105.07342, 2021.
  13. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021c.
  14. Language models as few-shot learner for task-oriented dialogue systems. arXiv preprint arXiv:2008.06239, 2020.
  15. A survey of large language models. arXiv preprint arXiv:2303.18223, 2023.
  16. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34:11287–11302, 2021.
  17. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
  18. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.
  19. Molecular dynamics and monte carlo calculations in statistical mechanics. Annual Review of Physical Chemistry, 27(1):319–348, 1976.
  20. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  21. Score-based generative modeling with critically-damped langevin diffusion. arXiv preprint arXiv:2112.07068, 2021.
  22. Score-based generative modeling of graphs via the system of stochastic differential equations. In International Conference on Machine Learning, pages 10362–10383. PMLR, 2022.
  23. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734, 2022.
  24. Se (3) equivariant graph neural networks with complete local frames. In International Conference on Machine Learning, pages 5583–5608. PMLR, 2022a.
  25. A group symmetric stochastic differential equation model for molecule multi-modal pretraining. 2023d.
  26. Learning gradient fields for molecular conformation generation. In International Conference on Machine Learning, pages 9558–9568. PMLR, 2021.
  27. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867–8887. PMLR, 2022.
  28. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
  29. A flexible diffusion model. arXiv preprint arXiv:2206.10365, 2022b.
  30. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
  31. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343, 2022.
  32. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  33. Cold diffusion: Inverting arbitrary image transforms without noise. arXiv preprint arXiv:2208.09392, 2022.
  34. Generative adversarial networks. Communications of the ACM, 63(11):139–144, 2020.
  35. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  36. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827–6839, 2020.
  37. Negative data augmentation. arXiv preprint arXiv:2102.05113, 2021.
  38. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162–8171. PMLR, 2021.
  39. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022.
  40. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
  41. Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.08610, 2022.
  42. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944, 2023.
  43. gddim: Generalized denoising diffusion implicit models. arXiv preprint arXiv:2206.05564, 2022a.
  44. A variational perspective on diffusion-based generative models and score matching. Advances in Neural Information Processing Systems, 34:22863–22876, 2021.
  45. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323–9332. PMLR, 2021.
  46. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5):e1603015, 2017.
  47. Bayesian neural networks: An introduction and survey. Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018, pages 45–87, 2020.
  48. Your classifier is secretly an energy based model and you should treat it like one. arXiv preprint arXiv:1912.03263, 2019.
  49. Joint energy-based models for semi-supervised classification. In ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning, volume 1, 2020.
  50. Autoregressive generative modeling with noise conditional maximum likelihood estimation. arXiv preprint arXiv:2210.10715, 2022.
  51. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  52. Understanding and improving the role of projection head in self-supervised learning. arXiv preprint arXiv:2212.11491, 2022.
  53. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
  54. A new perspective on building efficient and expressive 3d equivariant graph neural networks. arXiv preprint arXiv:2304.04757, 2023.
  55. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687, 2020b.
  56. Pubchemqc project: a large-scale first-principles electronic structure database for data-driven chemistry. Journal of chemical information and modeling, 57(6):1300–1308, 2017.
  57. Schnet–a deep learning architecture for molecules and materials. The Journal of Chemical Physics, 148(24):241722, 2018.
  58. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning, pages 20479–20502. PMLR, 2022.
  59. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1–7, 2014.
  60. Torsional diffusion for molecular conformer generation. arXiv preprint arXiv:2206.01729, 2022.
  61. Accurate learning of graph representations with graph multiset pooling. arXiv preprint arXiv:2102.11533, 2021.
  62. Mdm: Molecular diffusion model for 3d molecule generation, 2022.
  63. Thomas A Halgren. Merck molecular force field. i. basis, form, scope, parameterization, and performance of mmff94. Journal of computational chemistry, 17(5-6):490–519, 1996.
  64. Contrastive and non-contrastive self-supervised learning recover global and local spectral embedding methods. arXiv preprint arXiv:2205.11508, 2022.
  65. Disentangling factors of variation in deep representation using adversarial training. Advances in neural information processing systems, 29, 2016.
  66. Consistency models. arXiv preprint arXiv:2303.01469, 2023.
  67. Rethinking adversarial transferability from a data distribution perspective. In International Conference on Learning Representations, 2021.
  68. How mask matters: Towards theoretical understandings of masked autoencoders. arXiv preprint arXiv:2210.08344, 2022b.
  69. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279–287, 2022.
  70. Molecular geometry pretraining with SE(3)-invariant denoising distance matching. In The Eleventh International Conference on Learning Representations, 2023e. URL https://openreview.net/forum?id=CjTHVo1dvR.
  71. Uni-mol: A universal 3d molecular representation learning framework. 2023.
  72. Soft diffusion: Score matching for general corruptions. arXiv preprint arXiv:2209.05442, 2022.
  73. Smart augmentation learning an optimal data augmentation strategy. Ieee Access, 5:5858–5869, 2017.
  74. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123, 2019.
  75. Midi: Mixed graph and 3d denoising diffusion for molecule generation, 2023.
  76. Geom, energy-annotated molecular conformations for property prediction and molecular generation. Scientific Data, 9(1):185, 2022. doi: 10.1038/s41597-022-01288-4. URL https://doi.org/10.1038/s41597-022-01288-4.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now