Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

XTSFormer: Cross-Temporal-Scale Transformer for Irregular-Time Event Prediction in Clinical Applications (2402.02258v2)

Published 3 Feb 2024 in cs.LG and cs.AI

Abstract: Adverse clinical events related to unsafe care are among the top ten causes of death in the U.S. Accurate modeling and prediction of clinical events from electronic health records (EHRs) play a crucial role in patient safety enhancement. An example is modeling de facto care pathways that characterize common step-by-step plans for treatment or care. However, clinical event data pose several unique challenges, including the irregularity of time intervals between consecutive events, the existence of cycles, periodicity, multi-scale event interactions, and the high computational costs associated with long event sequences. Existing neural temporal point processes (TPPs) methods do not effectively capture the multi-scale nature of event interactions, which is common in many real-world clinical applications. To address these issues, we propose the cross-temporal-scale transformer (XTSFormer), specifically designed for irregularly timed event data. Our model consists of two vital components: a novel Feature-based Cycle-aware Time Positional Encoding (FCPE) that adeptly captures the cyclical nature of time, and a hierarchical multi-scale temporal attention mechanism, where different temporal scales are determined by a bottom-up clustering approach. Extensive experiments on several real-world EHR datasets show that our XTSFormer outperforms multiple baseline methods. The code is available at https://github.com/spatialdatasciencegroup/XTSFormer.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Hawkes processes in finance. Market Microstructure and Liquidity, 1(01):1550005, 2015.
  2. Meta temporal point processes. In The Eleventh International Conference on Learning Representations, 2022.
  3. Neural spatio-temporal point processes. In International Conference on Learning Representations, 2020.
  4. Point processes, volume 12. CRC Press, 1980.
  5. Ms-tct: multi-scale temporal convtransformer for action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20041–20051, 2022.
  6. An introduction to the theory of point processes: volume II: general theory and structure. Springer, 2008.
  7. Efficient algorithms for agglomerative hierarchical clustering methods. Journal of classification, 1:7–24, 1984.
  8. Temporal knowledge graph reasoning with low-rank and model-agnostic representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pages 111–120, 2022.
  9. Improving patient safety through provider communication strategy enhancements. 2011.
  10. Recurrent marked temporal point processes: Embedding event history to vector. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1555–1564, 2016.
  11. Variational neural temporal point process. arXiv preprint arXiv:2202.10585, 2022.
  12. Alan G Hawkes. Point spectra of some mutually exciting point processes. Journal of the Royal Statistical Society Series B: Statistical Methodology, 33(3):438–443, 1971.
  13. Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19013–19022, 2022.
  14. Neural jump stochastic differential equations. Advances in Neural Information Processing Systems, 32, 2019.
  15. Time2vec: Learning a vector representation of time. arXiv preprint arXiv:1907.05321, 2019.
  16. Rethinking positional encoding in language pre-training. In International Conference on Learning Representations, 2021.
  17. Snap datasets: Stanford large network dataset collection, 2014.
  18. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32, 2019.
  19. Temporal logic point processes. In International Conference on Machine Learning, pages 5990–6000. PMLR, 2020.
  20. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34:15816–15829, 2021.
  21. Exploring generative neural temporal point process. Transactions on Machine Learning Research, 2022.
  22. Learning to encode position for transformer with continuous dynamical model. In International conference on machine learning, pages 6327–6335. PMLR, 2020.
  23. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022.
  24. Multi-channel fusion lstm for medical event prediction using ehrs. Journal of Biomedical Informatics, 127:104011, 2022.
  25. The neural hawkes process: A neurally self-modulating multivariate point process. Advances in neural information processing systems, 30, 2017.
  26. Fully neural network based model for general temporal point processes. Advances in neural information processing systems, 32, 2019.
  27. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020.
  28. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
  29. Horst Rinne. The Weibull distribution: a handbook. CRC press, 2008.
  30. Point processes, spatial-temporal. Encyclopedia of environmetrics, 3:1573–1577, 2002.
  31. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In International Conference on Learning Representations, 2023.
  32. Self-attention with relative position representations. In Proceedings of NAACL-HLT, pages 464–468, 2018.
  33. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  34. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665–21674, 2020.
  35. Self-attention with structural position representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, 2019.
  36. Modeling sequential listening behaviors with attentive temporal point process for next and next new music recommendation. IEEE Transactions on Multimedia, 24:4170–4182, 2021.
  37. Clusterformer: Neural clustering attention for efficient and effective transformer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2390–2402, 2022.
  38. Hierarchical contrastive learning for temporal point processes. Proceedings of the AAAI Conference on Artificial Intelligence, 37(8):10166–10174, Jun. 2023.
  39. Modeling the intensity function of point process via recurrent neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
  40. Self-attention with functional time representation learning. Advances in neural information processing systems, 32, 2019.
  41. Inductive representation learning on temporal graphs. In International Conference on Learning Representations (ICLR), 2020.
  42. Transformer embeddings of irregularly spaced events and their participants. In Proceedings of the Tenth International Conference on Learning Representations (ICLR), 2022.
  43. Semantic periodic pattern mining from spatio-temporal trajectories. Information Sciences, 502:164–189, 2019.
  44. Self-attentive hawkes process. In International conference on machine learning, pages 11183–11193. PMLR, 2020.
  45. Learning neural point processes with latent graphs. In Proceedings of the Web Conference 2021, pages 1495–1505, 2021.
  46. Learning mixture of neural temporal point processes for multi-dimensional event sequence clustering. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, Vienna, Austria, pages 23–29, 2022.
  47. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.
  48. Intensity-free convolutional temporal point process: Incorporating local and global event contexts. Information Sciences, page 119318, 2023.
  49. Transformer hawkes process. In International conference on machine learning, pages 11692–11702. PMLR, 2020.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now