AlloyBERT: Alloy Property Prediction with Large Language Models (2403.19783v1)
Abstract: The pursuit of novel alloys tailored to specific requirements poses significant challenges for researchers in the field. This underscores the importance of developing predictive techniques for essential physical properties of alloys based on their chemical composition and processing parameters. This study introduces AlloyBERT, a transformer encoder-based model designed to predict properties such as elastic modulus and yield strength of alloys using textual inputs. Leveraging the pre-trained RoBERTa encoder model as its foundation, AlloyBERT employs self-attention mechanisms to establish meaningful relationships between words, enabling it to interpret human-readable input and predict target alloy properties. By combining a tokenizer trained on our textual data and a RoBERTa encoder pre-trained and fine-tuned for this specific task, we achieved a mean squared error (MSE) of 0.00015 on the Multi Principal Elemental Alloys (MPEA) data set and 0.00611 on the Refractory Alloy Yield Strength (RAYS) dataset. This surpasses the performance of shallow models, which achieved a best-case MSE of 0.00025 and 0.0076 on the MPEA and RAYS datasets respectively. Our results highlight the potential of LLMs in material science and establish a foundational framework for text-based prediction of alloy properties that does not rely on complex underlying representations, calculations, or simulations.
- Giustino, F. Materials modelling using density functional theory: properties and predictions; Oxford University Press, 2014
- Kitchin, J. R.; Miller, S. D.; Sholl, D. S. Density functional theory studies of alloys in heterogeneous catalysis. Specialist Periodical Reports: Chemical Modeling: Applications and Theories 2008,
- Vasudevan, R.; Pilania, G.; Balachandran, P. V. Machine learning for materials design and discovery. Journal of Applied Physics 2021, 129
- Wang, Y.; Li, Z.; Barati Farimani, A. Machine Learning in Molecular Sciences; Springer, 2023; pp 21–66
- Christofidellis, D.; Giannone, G.; Born, J.; Winther, O.; Laino, T.; Manica, M. Unifying molecular and textual representations via multi-task language modelling. arXiv preprint arXiv:2301.12586 2023,
- Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018,
- Peng, Y.; Yan, S.; Lu, Z. Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474 2019,
- Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019,
- Gillioz, A.; Casas, J.; Mugellini, E.; Abou Khaled, O. Overview of the Transformer-based Models for NLP Tasks. 2020 15th Conference on Computer Science and Information Systems (FedCSIS). 2020; pp 179–183
- Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; others Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 2023,
- Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. Advances in neural information processing systems 2017, 30
- Ock, J.; Magar, R.; Antony, A.; Farimani, A. B. Multimodal Language and Graph Learning of Adsorption Configuration in Catalysis. arXiv preprint arXiv:2401.07408 2024,
- Huang, H.; Magar, R.; Xu, C.; Farimani, A. B. Materials Informatics Transformer: A Language Model for Interpretable Materials Properties Prediction. arXiv preprint arXiv:2308.16259 2023,
- Shibata, Y.; Kida, T.; Fukamachi, S.; Takeda, M.; Shinohara, A.; Shinohara, T.; Arikawa, S. Byte Pair encoding: A text compression scheme that accelerates pattern matching. 1999,
- Huang, H.; Magar, R.; Barati Farimani, A. Pretraining Strategies for Structure Agnostic Material Property Prediction. Journal of Chemical Information and Modeling 2024,
- Kudo, T. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959 2018,
- Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017,
- Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014,
- Montgomery, D. C.; Peck, E. A.; Vining, G. G. Introduction to linear regression analysis; John Wiley & Sons, 2021
- Friedman, J. H. Greedy function approximation: a gradient boosting machine. Annals of statistics 2001, 1189–1232
- Falaki, A. A.; Gras, R. Attention Visualizer Package: Revealing Word Importance for Deeper Insight into Encoder-Only Transformer Models. arXiv preprint arXiv:2308.14850 2023,
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
