Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Causal Ordering Prior for Unsupervised Representation Learning (2307.05704v1)

Published 11 Jul 2023 in cs.LG, cs.AI, and cs.CV

Abstract: Unsupervised representation learning with variational inference relies heavily on independence assumptions over latent variables. Causal representation learning (CRL), however, argues that factors of variation in a dataset are, in fact, causally related. Allowing latent variables to be correlated, as a consequence of causal relationships, is more realistic and generalisable. So far, provably identifiable methods rely on: auxiliary information, weak labels, and interventional or even counterfactual data. Inspired by causal discovery with functional causal models, we propose a fully unsupervised representation learning method that considers a data generation process with a latent additive noise model (ANM). We encourage the latent space to follow a causal ordering via loss function based on the Hessian of the latent distribution.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022.
  2. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  3. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319–38331, 2022.
  4. Morpho-mnist: Quantitative assessment and diagnostics for representation learning. Journal of Machine Learning Research, 20(178):1–29, 2019.
  5. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
  6. Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
  7. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems, 34:8676–8690, 2021.
  8. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10, 2019.
  9. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
  10. Symmetry-based representations for artificial and biological general intelligence. Frontiers in Computational Neuroscience, 16, 2022.
  11. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems, volume 21, 2008.
  12. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  13. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429–439, 1999.
  14. Identifiability of latent-variable and structural-equation models: from linear to nonlinear, 2023.
  15. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148, 2016.
  16. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017.
  17. Composing graphical models with neural networks for structured representations and fast inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  18. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal of Machine Learning Research, 8(3), 2007.
  19. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.
  20. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems, volume 33, 2020.
  21. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687–15701, 2022.
  24. Glance: Global to local architecture-neutral concept-based explanations. arXiv preprint arXiv:2207.01917, 2022.
  25. Gradient estimators for implicit models. arXiv preprint arXiv:1705.07107, 2017.
  26. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557–13603. PMLR, 2022.
  27. Learning disentangled representations in the imaging domain. Medical Image Analysis, 80, 2022.
  28. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019.
  29. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348–6359. PMLR, 2020.
  30. Disentangling disentanglement in variational autoencoders. In Proceedings of the 36th International Conference on Machine Learning, 2019.
  31. Distinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1):1103–1204, 2016.
  32. On approximations via convolution-defined mixture models. Communications in Statistics-Theory and Methods, 48(16):3945–3955, 2019.
  33. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
  34. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15, 2014.
  35. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009–2053, 2014.
  36. Hans Reichenbach. The direction time. Univ. of California Press, 1956.
  37. Score matching enables causal discovery of nonlinear additive noise models. In International Conference on Machine Learning, pages 18741–18753. PMLR, 2022.
  38. Diffusion models for causal discovery via topological ordering. In The Eleventh International Conference on Learning Representations, 2023.
  39. Toward causal representation learning. Proceedings of the IEEE, 109, 2021.
  40. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23:1–55, 2022.
  41. On disentangled representations learned from correlated data. In Proceedings of the 38th International Conference on Machine Learning, 2021.
  42. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.
  43. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.
  44. I don’t need u: Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05238, 2021.
  45. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9593–9602, 2021.
  46. gcastle: A python toolbox for causal discovery, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Avinash Kori (29 papers)
  2. Pedro Sanchez (20 papers)
  3. Konstantinos Vilouras (4 papers)
  4. Ben Glocker (143 papers)
  5. Sotirios A. Tsaftaris (100 papers)

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now