Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 60 tok/s
Gemini 2.5 Pro 51 tok/s Pro
GPT-5 Medium 18 tok/s Pro
GPT-5 High 14 tok/s Pro
GPT-4o 77 tok/s Pro
Kimi K2 159 tok/s Pro
GPT OSS 120B 456 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

A Causal Ordering Prior for Unsupervised Representation Learning (2307.05704v1)

Published 11 Jul 2023 in cs.LG, cs.AI, and cs.CV

Abstract: Unsupervised representation learning with variational inference relies heavily on independence assumptions over latent variables. Causal representation learning (CRL), however, argues that factors of variation in a dataset are, in fact, causally related. Allowing latent variables to be correlated, as a consequence of causal relationships, is more realistic and generalisable. So far, provably identifiable methods rely on: auxiliary information, weak labels, and interventional or even counterfactual data. Inspired by causal discovery with functional causal models, we propose a fully unsupervised representation learning method that considers a data generation process with a latent additive noise model (ANM). We encourage the latent space to follow a causal ordering via loss function based on the Hessian of the latent distribution.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Interventional causal representation learning. arXiv preprint arXiv:2209.11924, 2022.
  2. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798–1828, 2013.
  3. Weakly supervised causal representation learning. Advances in Neural Information Processing Systems, 35:38319–38331, 2022.
  4. Morpho-mnist: Quantitative assessment and diagnostics for representation learning. Journal of Machine Learning Research, 20(178):1–29, 2019.
  5. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems, 2016.
  6. Cian Eastwood and Christopher K. I. Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference on Learning Representations, 2018.
  7. Multi-facet clustering variational autoencoders. Advances in Neural Information Processing Systems, 34:8676–8690, 2021.
  8. Review of causal discovery methods based on graphical models. Frontiers in Genetics, 10, 2019.
  9. beta-VAE: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
  10. Symmetry-based representations for artificial and biological general intelligence. Frontiers in Computational Neuroscience, 16, 2022.
  11. Nonlinear causal discovery with additive noise models. In Advances in Neural Information Processing Systems, volume 21, 2008.
  12. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  13. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429–439, 1999.
  14. Identifiability of latent-variable and structural-equation models: from linear to nonlinear, 2023.
  15. Variational deep embedding: An unsupervised and generative approach to clustering. arXiv preprint arXiv:1611.05148, 2016.
  16. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, 2017.
  17. Composing graphical models with neural networks for structured representations and fast inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
  18. Estimating high-dimensional directed acyclic graphs with the pc-algorithm. Journal of Machine Learning Research, 8(3), 2007.
  19. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207–2217. PMLR, 2020.
  20. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. In Advances in Neural Information Processing Systems, volume 33, 2020.
  21. Disentangling by factorising. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2018.
  22. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  23. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687–15701, 2022.
  24. Glance: Global to local architecture-neutral concept-based explanations. arXiv preprint arXiv:2207.01917, 2022.
  25. Gradient estimators for implicit models. arXiv preprint arXiv:1705.07107, 2017.
  26. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pages 13557–13603. PMLR, 2022.
  27. Learning disentangled representations in the imaging domain. Medical Image Analysis, 80, 2022.
  28. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pages 4114–4124. PMLR, 2019.
  29. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348–6359. PMLR, 2020.
  30. Disentangling disentanglement in variational autoencoders. In Proceedings of the 36th International Conference on Machine Learning, 2019.
  31. Distinguishing cause from effect using observational data: methods and benchmarks. The Journal of Machine Learning Research, 17(1):1103–1204, 2016.
  32. On approximations via convolution-defined mixture models. Communications in Statistics-Theory and Methods, 48(16):3945–3955, 2019.
  33. Elements of Causal Inference: Foundations and Learning Algorithms. The MIT Press, 2017.
  34. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15, 2014.
  35. Causal discovery with continuous additive noise models. Journal of Machine Learning Research, 15:2009–2053, 2014.
  36. Hans Reichenbach. The direction time. Univ. of California Press, 1956.
  37. Score matching enables causal discovery of nonlinear additive noise models. In International Conference on Machine Learning, pages 18741–18753. PMLR, 2022.
  38. Diffusion models for causal discovery via topological ordering. In The Eleventh International Conference on Learning Representations, 2023.
  39. Toward causal representation learning. Proceedings of the IEEE, 109, 2021.
  40. Weakly supervised disentangled generative causal representation learning. Journal of Machine Learning Research, 23:1–55, 2022.
  41. On disentangled representations learned from correlated data. In Proceedings of the 38th International Conference on Machine Learning, 2021.
  42. Score-based causal representation learning with interventions. arXiv preprint arXiv:2301.08230, 2023.
  43. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.
  44. I don’t need u: Identifiable non-linear ica without side information. arXiv preprint arXiv:2106.05238, 2021.
  45. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9593–9602, 2021.
  46. gcastle: A python toolbox for causal discovery, 2021.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Lightbulb On Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now