Siu-On Chan
ABSTRACT
We give a computationally efficient semi-agnostic algorithm for learning univariate probability distributions that are well approximated by piecewise polynomial density functions. Let p be an arbitrary distribution over an interval I, and suppose that p is τ-close (in total variation distance) to an unknown probability distribution q that is defined by an unknown partition of I into t intervals and t unknown degree d polynomials specifying q over each of the intervals. We give an algorithm that draws Õ(t(d + 1)/ε2) samples from p, runs in time poly(t, d + 1, 1/ε), and with high probability outputs a piecewise polynomial hypothesis distribution h that is (14τ + ε)-close to p in total variation distance. Our algorithm combines tools from real approximation theory, uniform convergence, linear programming, and dynamic programming. Its sample complexity is simultaneously near optimal in all three parameters t, d and ε; we show that even for τ = 0, any algorithm that learns an unknown t-piecewise degree-d probability distribution over I to accuracy ε must use [EQUATION] samples from the distribution, regardless of its running time.
We apply this general algorithm to obtain a wide range of results for many natural density estimation problems over both continuous and discrete domains. These include state-of-the-art results for learning mixtures of log-concave distributions; mixtures of t-modal distributions; mixtures of Monotone Hazard Rate distributions; mixtures of Poisson Binomial Distributions; mixtures of Gaussians; and mixtures of k-monotone densities. Our general technique gives improved results, with provably optimal sample complexities (up to logarithmic factors) in all parameters in most cases, for all these problems via a single unified algorithm.
Supplemental Material
- {AK03} S. Arora and S. Khot. Fitting algebraic curves to noisy data. J. Comput. Syst. Sci., 67(2):325--340, 2003. Google Scholar
Digital Library
- {Ass83} P. Assouad. Deux remarques sur l'estimation. C. R. Acad. Sci. Paris Sér. I, 296:1021--1024, 1983.Google Scholar
- {BBBB72} R. E. Barlow, D. J. Bartholomew, J. M. Bremner, and H. D. Brunk. Statistical Inference under Order Restrictions. Wiley, New York, 1972.Google Scholar
- {Bir87a} L. Birgé. Estimating a density under order restrictions: Nonasymptotic minimax risk. Annals of Statistics, 15(3):995--1012, 1987.Google ScholarCross Ref
- {Bir87b} L. Birgé. On the risk of histograms for estimating decreasing densities. Annals of Statistics, 15(3):1013--1022, 1987.Google ScholarCross Ref
- {Bru58} H. D. Brunk. On the estimation of parameters restricted by inequalities. Ann. Math. Statist., 29(2):pp. 437--454, 1958.Google ScholarCross Ref
- {BRW09} F. Balabdaoui, K. Rufibach, and J. A. Wellner. Limit distribution theory for maximum likelihood estimation of a log-concave density. Ann. Statist., 37(3):pp. 1299--1331, 2009.Google ScholarCross Ref
- {BS10} M. Belkin and K. Sinha. Polynomial learning of distribution families. In FOCS, pages 103--112, 2010. Google ScholarDigital Library
- {BW07} F. Balabdaoui and J. A. Wellner. Estimation of a k-monotone density: Limit distribution theory and the spline connection. Ann. Statist., 35(6):pp. 2536--2564, 2007.Google ScholarCross Ref
- {BW10} F. Balabdaoui and J. A. Wellner. Estimation of a k-monotone density: characterizations, consistency and minimax lower bounds. Statistica Neerlandica, 64(1):45--70, 2010.Google ScholarCross Ref
- {CDSS13} S. Chan, I. Diakonikolas, R. Servedio, and X. Sun. Learning mixtures of structured distributions over discrete domains. In SODA, 2013. Google ScholarDigital Library
- {DDO+13} C. Daskalakis, I. Diakonikolas, R. O'Donnell, R. A. Servedio, and L. Tan. Learning Sums of Independent Integer Random Variables. In FOCS, pages 217--226, 2013. Google ScholarDigital Library
- {DDS12a} C. Daskalakis, I. Diakonikolas, and R. A. Servedio. Learning k-modal distributions via testing. In SODA, 2012. Google ScholarDigital Library
- {DDS12b} C. Daskalakis, I. Diakonikolas, and R. A. Servedio. Learning Poisson Binomial Distributions. In STOC, pages 709--728, 2012. Google ScholarDigital Library
- {DG85} L. Devroye and L. Györfi. Nonparametric Density Estimation: The L1 View. John Wiley & Sons, 1985.Google Scholar
- {DGJ+10} I. Diakoniokolas, P. Gopalan, R. Jaiswal, R. Servedio, and E. Viola. Bounded independence fools halfspaces. SIAM Journal on Computing, 39(8):3441--3462, 2010. Google ScholarDigital Library
- {DL01} L. Devroye and G. Lugosi. Combinatorial methods in density estimation. Springer Series in Statistics, Springer, 2001.Google ScholarCross Ref
- {DR09} L. D umbgen and K. Rufibach. Maximum likelihood estimation of a log-concave density and its distribution function: Basic properties and uniform consistency. Bernoulli, 15(1):40--68, 2009.Google ScholarCross Ref
- {Dud74} R. M Dudley. Metric entropy of some classes of sets with differentiable boundaries. Journal of Approximation Theory, 10(3):227--236, 1974.Google ScholarCross Ref
- {FM99} Y. Freund and Y. Mansour. Estimating a mixture of two product distributions. In Proc. 12th COLT, pages 183--192, 1999. Google ScholarDigital Library
- {FOS05} J. Feldman, R. O'Donnell, and R. Servedio. Learning mixtures of product distributions over discrete domains. In Proc. 46th IEEE FOCS, pages 501--510, 2005. Google ScholarDigital Library
- {Gre56} U. Grenander. On the theory of mortality measurement. Skand. Aktuarietidskr., 39:125--153, 1956.Google Scholar
- {Gro85} P. Groeneboom. Estimating a monotone density. In Proc. Berkeley Conf. in Honor of J. Neyman and J. Kiefer, pages 539--555, 1985.Google Scholar
- {GW09} F. Gao and J. A. Wellner. On the rate of convergence of the maximum likelihood estimator of a k-monotone density. Science in China Series A: Math., 52:1525--1538, 2009.Google ScholarCross Ref
- {HP76} D. L. Hanson and G. Pledger. Consistency in concave regression. The Annals of Statistics, 4(6):pp. 1038--1050, 1976.Google ScholarCross Ref
- {KL04} V. N. Konovalov and D. Leviatan. Free-knot splines approximation of s-monotone functions. Adv. Comput. Math., 20(4):347--366, 2004.Google ScholarCross Ref
- {KL07} V. N. Konovalov and D. Leviatan. Freeknot splines approximation of sobolev-type classes of s-monotone functions. Adv. Comput. Math., 27(2):211--236, 2007.Google ScholarCross Ref
- {KM10} R. Koenker and I. Mizera. Quasi-concave density estimation. Ann. Statist., 38(5):2998--3027, 2010.Google ScholarCross Ref
- {KMR+94} M. Kearns, Y. Mansour, D. Ron, R. Rubinfeld, R. Schapire, and L. Sellie. On the learnability of discrete distributions. In Proc. 26th STOC, pages 273--282, 1994. Google ScholarDigital Library
- {KMV10} A. T. Kalai, A. Moitra, and G. Valiant. Efficiently learning mixtures of two Gaussians. In STOC, pages 553--562, 2010. Google ScholarDigital Library
- {MV10} A. Moitra and G. Valiant. Settling the polynomial learnability of mixtures of Gaussians. In FOCS, pages 93--102, 2010. Google ScholarDigital Library
- {Nov88} E. Novak. Deterministic and Stochastic Error Bounds In Numerical Analysis. Springer-Verlag, 1988.Google ScholarCross Ref
- {PA13} D. Papp and F. Alizadeh. Shape constrained estimation using nonnegative splines. J. Comput. & Graph. Statist., 0, 2013.Google Scholar
- {Rao69} B. L. S. Prakasa Rao. Estimation of a unimodal density. Sankhya Ser. A, 31:23--36, 1969.Google Scholar
- {Reb05} L. Reboul. Estimation of a function under shape restrictions. Applications to reliability. Ann. Statist., 33(3):1330--1356, 2005.Google ScholarCross Ref
- {Sco92} D. W. Scott. Multivariate Density Estimation: Theory, Practice and Visualization. Wiley, New York, 1992.Google ScholarCross Ref
- {Sil86} B. W. Silverman. Density Estimation. Chapman and Hall, London, 1986.Google ScholarCross Ref
- {Wal09} G. Walther. Inference and modeling with log-concave distributions. Statistical Science, 24(3):319--327, 2009.Google ScholarCross Ref
- {Weg70} E. J. Wegman. Maximum likelihood estimation of a unimodal density. I. and II. Ann. Math. Statist., 41:457--471, 2169--2174, 1970.Google ScholarCross Ref
Index Terms
- Efficient density estimation via piecewise polynomial approximation
Recommendations
Learning poisson binomial distributions
STOC '12: Proceedings of the forty-fourth annual ACM symposium on Theory of computingWe consider a basic problem in unsupervised learning: learning an unknown Poisson Binomial Distribution. A Poisson Binomial Distribution (PBD) over {0,1,...,n} is the distribution of a sum of n independent Bernoulli random variables which may have ...
Estimation of quantile mixtures via L-moments and trimmed L-moments
Moments or cumulants have been traditionally used to characterize a probability distribution or an observed data set. Recently, L-moments and trimmed L-moments have been noticed as appealing alternatives to the conventional moments. This paper promotes ...
Noncentral elliptical configuration density
The noncentral configuration density, derived under an elliptical model, generalizes and corrects the Gaussian configuration and some Pearson results. Partition theory is then used to obtain explicit configuration densities associated with matrix ...
Comments