An Improved Algorithm for Learning Drifting Discrete Distributions (2403.05446v1)
Abstract: We present a new adaptive algorithm for learning discrete distributions under distribution drift. In this setting, we observe a sequence of independent samples from a discrete distribution that is changing over time, and the goal is to estimate the current distribution. Since we have access to only a single sample for each time step, a good estimation requires a careful choice of the number of past samples to use. To use more samples, we must resort to samples further in the past, and we incur a drift error due to the bias introduced by the change in distribution. On the other hand, if we use a small number of past samples, we incur a large statistical error as the estimation has a high variance. We present a novel adaptive algorithm that can solve this trade-off without any prior knowledge of the drift. Unlike previous adaptive results, our algorithm characterizes the statistical error using data-dependent bounds. This technicality enables us to overcome the limitations of the previous work that require a fixed finite support whose size is known in advance and that cannot change over time. Additionally, we can obtain tighter bounds depending on the complexity of the drifting distribution, and also consider distributions with infinite support.
- Neural network learning: Theoretical foundations, volume 9. Cambridge University Press.
- Theory and algorithm for batch distribution drift problems. In Proc. AISTATS, pages 9826–9851.
- Bartlett, P. L. (1992). Learning with a slowly changing distribution. In Proc. COLT, pages 243–252.
- On the complexity of learning from drifting distributions. In Proc. COLT, pages 122–130.
- A theory of learning from different domains. Machine Learning, 79:151–175.
- A sharp estimate of the binomial mean absolute deviation with applications. Statistics & Probability Letters, 83(4):1254–1259.
- Learning discrete distributions with infinite support. In Proc. NeurIPS, pages 3942–3951.
- Nonparametric density estimation : the l[1] view. Journal of the American Statistical Association, 82:344.
- Combinatorial methods in density estimation. Springer Science & Business Media.
- Fast and three-rious: Speeding up weak supervision with triplet methods. In Proc. ICML, pages 3280–3291.
- Online wavelet-based density estimation for non-stationary streaming data. Computational statistics & data analysis, 56(2):327–344.
- Online density estimation of nonstationary sources using exponential family of distributions. IEEE Transactions on Neural Networks and Learning Systems, 29(9):4473–4478.
- Minimax estimation of discrete distributions. In 2015 IEEE International Symposium on Information Theory (ISIT), pages 2291–2295. IEEE.
- Learning with a drifting target concept. In Proc. ALT, pages 149–164.
- Statistical learning under nonstationary mixing processes. In Proc. AISTATS, pages 1678–1686.
- Tracking drifting concepts using random examples. In Proc. COLT, pages 13–23.
- Tracking drifting concepts by minimizing disagreements. Machine Learning, 14:27–45.
- Minimax estimation of functionals of discrete distributions. IEEE Transactions on Information Theory, 61(5):2835–2885.
- On learning distributions from their samples. In Proc. COLT, pages 1066–1100.
- Multivariate online kernel density estimation with gaussian kernels. Pattern recognition, 44(10-11):2630–2642.
- Long, P. M. (1998). The complexity of learning according to two models of a drifting environment. In Proc. COLT, pages 116–125.
- Domain adaptation: Learning bounds and algorithms. In Proc. COLT.
- An adaptive algorithm for learning with unknown distribution drift. In Proc. NeurIPS.
- Nonparametric density estimation under distribution drift. In Proc. ICML, pages 24251–24270.
- New analysis and algorithm for learning with drifting distributions. In Proc. ALT, pages 124–138.
- Competitive distribution estimation: Why is good-turing good. In Proc. NeurIPS.
- Silverman, B. W. (1986). Density estimation for statistics and data analysis, volume 26. CRC press.
Collections
Sign up for free to add this paper to one or more collections.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.