Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Partial Identifiability for Domain Adaptation (2306.06510v1)

Published 10 Jun 2023 in cs.LG and stat.ML

Abstract: Unsupervised domain adaptation is critical to many real-world applications where label information is unavailable in the target domain. In general, without further assumptions, the joint distribution of the features and the label is not identifiable in the target domain. To address this issue, we rely on the property of minimal changes of causal mechanisms across domains to minimize unnecessary influences of distribution shifts. To encode this property, we first formulate the data-generating process using a latent variable model with two partitioned latent subspaces: invariant components whose distributions stay the same across domains and sparse changing components that vary across domains. We further constrain the domain shift to have a restrictive influence on the changing components. Under mild conditions, we show that the latent variables are partially identifiable, from which it follows that the joint distribution of data and labels in the target domain is also identifiable. Given the theoretical insights, we propose a practical domain adaptation framework called iMSDA. Extensive experimental results reveal that iMSDA outperforms state-of-the-art domain adaptation algorithms on benchmark datasets, demonstrating the effectiveness of our framework.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (79)
  1. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129–1159, 1995.
  2. A theory of learning from different domains. Machine learning, 79(1-2):151–175, 2010.
  3. Adamatch: A unified approach to semi-supervised learning and domain adaptation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Q5uh1Nvv5dm.
  4. Domain separation networks. In Advances in Neural Information Processing Systems, pp. 343–351, 2016.
  5. Learning disentangled semantic representation for domain adaptation. In IJCAI: proceedings of the conference, volume 2019, pp. 2060. NIH Public Access, 2019.
  6. Separation of mixed audio sources by independent subspace analysis. In ICMC, pp.  154–161, 2000.
  7. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learning, pp. 1081–1090. PMLR, 2019.
  8. Comon, P. Independent component analysis, a new concept? Signal processing, 36(3):287–314, 1994.
  9. Learning bounds for importance weighting. In NIPS 23, 2010.
  10. Joint distribution optimal transportation for domain adaptation. In NIPS, 2017.
  11. Cluster alignment with a teacher for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  9944–9953, 2019.
  12. Density estimation using real nvp, 2017.
  13. Neural spline flows, 2019.
  14. Source-free adaptation to measurement shift via bottom-up feature restoration. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1JDiK_TbV4S.
  15. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495, 2014.
  16. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(1):2096–2030, 2016.
  17. Domain adaptation with conditional transferable components. In International conference on machine learning, pp. 2839–2848. PMLR, 2016.
  18. Ltf: A label transformation framework for correcting label shift. In International Conference on Machine Learning, pp. 3843–3853. PMLR, 2020.
  19. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. In Conference on Uncertainty in Artificial Intelligence, pp. 939–948. PMLR, 2020.
  20. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  21. beta-vae: Learning basic visual concepts with a constrained variational framework. 2016.
  22. Causal discovery from heterogeneous/nonstationary data. Journal of Machine Learning Research, 2020.
  23. Neural autoregressive flows, 2018.
  24. Correcting sample selection bias by unlabeled data. In NIPS 19, pp.  601–608, 2007.
  25. Emergence of phase-and shift-invariant features by decomposition of natural images into independent feature subspaces. Neural computation, 12(7):1705–1720, 2000.
  26. Unsupervised feature extraction by time-contrastive learning and nonlinear ica, 2016.
  27. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429–439, 1999.
  28. Independent Component Analysis. John Wiley & Sons, Inc, 2001.
  29. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.  859–868. PMLR, 2019.
  30. Variational autoencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pp.  2207–2217. PMLR, 2020a.
  31. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33:12768–12778, 2020b.
  32. Auto-encoding variational bayes, 2014.
  33. Mapping conditional distributions for domain adaptation under generalized target shift. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=sPfB2PI87BZ.
  34. Towards nonlinear disentanglement in natural data with temporal sparse coding. arXiv preprint arXiv:2007.10930, 2020.
  35. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. arXiv preprint arXiv:2107.10098, 2021.
  36. Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis. In CVPR 2011, pp.  3361–3368. IEEE, 2011.
  37. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp.  5542–5550, 2017.
  38. T-svdnet: Exploring high-order prototypical correlations for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  9991–10000, 2021.
  39. Learning transferable features with deep adaptation networks. In International conference on machine learning, pp. 97–105. PMLR, 2015.
  40. Conditional adversarial domain adaptation. arXiv preprint arXiv:1705.10667, 2017a.
  41. Deep transfer learning with joint adaptation networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp.  2208–2217. JMLR. org, 2017b.
  42. Invariant causal representation learning. 2020.
  43. Boosting domain adaptation by discovering latent domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  3771–3780, 2018.
  44. Source free unsupervised graph domain adaptation. arXiv preprint arXiv:2112.00955, 2021.
  45. Information-theoretic regularization for multi-source domain adaptation. arXiv preprint arXiv:2104.01568, 2021.
  46. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1406–1415, 2019.
  47. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3723–3732, 2018.
  48. Universal domain adaptation through self supervision. arXiv preprint arXiv:2002.07953, 2020.
  49. On causal and anticausal learning. In ICML-12, Edinburgh, Scotland, 2012.
  50. Shimodaira, H. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90:227–244, 2000.
  51. A dirt-t approach to unsupervised domain adaptation. In Proc. 6th International Conference on Learning Representations, 2018.
  52. Disentanglement by nonlinear ica with general incompressible-flow networks (gin). arXiv preprint arXiv:2001.04872, 2020.
  53. Data-driven approach to multiple-source domain adaptation. Proceedings of machine learning research, 89:3487, 2019.
  54. Domain adaptation with invariant representation learning: What transformations to learn? Advances in Neural Information Processing Systems, 34, 2021.
  55. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics, 60:699–746, 2008.
  56. Theis, F. Towards a general independent subspace analysis. Advances in Neural Information Processing Systems, 19:1361–1368, 2006.
  57. Adversarial support alignment. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=26gKg6x-ie.
  58. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  7167–7176, 2017.
  59. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5018–5027, 2017.
  60. Self-supervised learning with data augmentations provably isolates content from style. arXiv preprint arXiv:2106.04619, 2021.
  61. Learning to combine: Knowledge aggregation for multi-source domain adaptation. In European Conference on Computer Vision, pp.  727–744. Springer, 2020.
  62. Easy transfer learning by exploiting intra-domain structures. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp.  1210–1215. IEEE, 2019.
  63. Domain adaptation with asymmetrically-relaxed distribution alignment. In International Conference on Machine Learning, pp. 6872–6881. PMLR, 2019.
  64. Stylealign: Analysis and applications of aligned styleGAN models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Qg2vi4ZbHM9.
  65. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning, pp. 5423–5432. PMLR, 2018.
  66. Deep cocktail network: Multi-source unsupervised domain adaptation with category shift. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  3964–3973, 2018.
  67. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  1426–1435, 2019.
  68. CDTrans: Cross-domain transformer for unsupervised domain adaptation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=XGzk5OKWFFc.
  69. Curriculum manager for source selection in multi-source domain adaptation. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp.  608–624. Springer, 2020.
  70. Zadrozny, B. Learning and evaluating classifiers under sample selection bias. In ICML-04, pp.  114–121, Banff, Canada, 2004.
  71. Joint geometrical and statistical alignment for visual domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.  1859–1867, 2017a.
  72. Domain adaptation under target and conditional shift. In International Conference on Machine Learning, pp. 819–827, 2013.
  73. Multi-source domain adaptation: A causal view. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  74. Causal discovery from nonstationary/heterogeneous data: Skeleton estimation and orientation determination. In IJCAI, volume 2017, pp.  1347, 2017b.
  75. Domain adaptation as a problem of inference on graphical models. Advances in Neural Information Processing Systems, 33, 2020.
  76. Collaborative and adversarial network for unsupervised domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  3801–3809, 2018.
  77. Domain-symmetric networks for adversarial domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5031–5040, 2019.
  78. Adversarial multiple source domain adaptation. In Advances in neural information processing systems, pp. 8559–8570, 2018.
  79. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=vqGi8Kp0wM.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now