×

Optimal model selection in density estimation. (English. French summary) Zbl 1244.62052

Summary: In order to calibrate a penalization procedure for model selection, the statistician has to choose a shape for the penalty and a leading constant. We study, for the marginal density estimation problem, the resampling penalties as general estimators of the shape of an ideal penalty. We prove that the selected estimator satisfies sharp oracle inequalities without remainder terms under a few assumptions on the marginal density \(s\) and the collection of models. We also study the slope heuristic, which yields a data-driven choice of the leading constant in front of the penalty when the complexity of the models is well-chosen.

MSC:

62G07 Density estimation
62G09 Nonparametric statistical resampling methods
62G20 Asymptotic properties of nonparametric inference

Software:

CAPUSHE
PDFBibTeX XMLCite
Full Text: DOI arXiv Euclid

References:

[1] H. Akaike. Statistical predictor identification. Ann. Inst. Statist. Math. 22 (1970) 203-217. · Zbl 0259.62076 · doi:10.1007/BF02506337
[2] H. Akaike. Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Tsahkadsor, 1971) 267-281. Akadémiai Kiadó, Budapest, 1973. · Zbl 0283.62006
[3] S. Arlot. Resampling and model selection. Ph.D. thesis, Université Paris-Sud 11, 2007. · Zbl 1326.62097
[4] S. Arlot. Model selection by resampling penalization. Electron. J. Stat. 3 (2009) 557-624. · Zbl 1326.62097 · doi:10.1214/08-EJS196
[5] S. Arlot and P. Massart. Data-driven calibration of penalties for least-squares regression. J. Mach. Learn. Res. 10 (2009) 245-279.
[6] A. Barron, L. Birgé and P. Massart. Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 (3) (1999) 301-413. · Zbl 0946.62036 · doi:10.1007/s004400050210
[7] J.-P. Baudry, K. Maugis and B. Michel. Slope heuristics: Overview and implementation. Report INRIA , 2010. Available at . · Zbl 1322.62007
[8] L. Birgé. Model selection for density estimation with \(l^{2}\)-loss. Preprint, 2008.
[9] L. Birgé and P. Massart. From model selection to adaptive estimation. In Festschrift for Lucien Le Cam 55-87. Springer, New York, 1997. · Zbl 0920.62042
[10] L. Birgé and P. Massart. Minimal penalties for Gaussian model selection. Probab. Theory Related Fields 138 (1-2) (2007) 33-73. · Zbl 1112.62082 · doi:10.1007/s00440-006-0011-8
[11] O. Bousquet. A Bennett concentration inequality and its application to suprema of empirical processes. C. R. Math. Acad. Sci. Paris 334 (6) (2002) 495-500. · Zbl 1001.60021 · doi:10.1016/S1631-073X(02)02292-6
[12] F. Bunea, A. B. Tsybakov and M. H. Wegkamp. Sparse density estimation with \(\ell_{1}\) penalties. In Learning Theory 530-543. Lecture Notes in Comput. Sci. 4539 . Springer, Berlin, 2007. · Zbl 1203.62053 · doi:10.1007/978-3-540-72927-3_38
[13] A. Célisse. Density estimation via cross validation: Model selection point of view. Preprint, 2008. Available at .
[14] D. L. Donoho, I. M. Johnstone, G. Kerkyacharian and D. Picard. Density estimation by wavelet thresholding. Ann. Statist. 24 (2) (1996) 508-539. · Zbl 0860.62032 · doi:10.1214/aos/1032894451
[15] B. Efron. Bootstrap methods: Another look at the jackknife. Ann. Statist. 7 (1) (1979) 1-26. · Zbl 0406.62024 · doi:10.1214/aos/1176344552
[16] I. Gannaz and O. Wintenberger. Adaptive density estimation under dependence. ESAIM Probab. Stat. 14 (2010) 151-172. · Zbl 1209.62056 · doi:10.1051/ps:2008025
[17] C. Houdré and P. Reynaud-Bouret. Exponential inequalities, with constants, for U-statistics of order two. In Stochastic Inequalities and Applications 55-69. Progr. Probab. 56 . Birkhäuser, Basel, 2003. · Zbl 1036.60015
[18] T. Klein and E. Rio. Concentration around the mean for maxima of empirical processes. Ann. Probab. 33 (3) (2005) 1060-1077. · Zbl 1066.60023 · doi:10.1214/009117905000000044
[19] C. L. Mallows. Some comments on \(c_{p}\). Technometrics 15 (1973) 661-675. · Zbl 0269.62061 · doi:10.2307/1267380
[20] P. Massart. Concentration Inequalities and Model Selection. Lecture Notes in Mathematics 1896 . Springer, Berlin, 2007. Lectures from the 33rd Summer School on Probability Theory held in Saint-Flour, July 6-23, 2003, With a foreword by Jean Picard. · Zbl 1170.60006 · doi:10.1007/978-3-540-48503-2
[21] P. Rigollet. Adaptive density estimation using the blockwise Stein method. Bernoulli 12 (2) (2006) 351-370. · Zbl 1098.62040 · doi:10.3150/bj/1145993978
[22] P. Rigollet and A. B. Tsybakov. Linear and convex aggregation of density estimators. Math. Methods Statist. 16 (3) (2007) 260-280. · Zbl 1231.62057 · doi:10.3103/S1066530707030052
[23] M. Rudemo. Empirical choice of histograms and kernel density estimators. Scand. J. Stat. 9 (2) (1982) 65-78. · Zbl 0501.62028
[24] G. Schwarz. Estimating the dimension of a model. Ann. Statist. 6 (1978) 461-464. · Zbl 0379.62005 · doi:10.1214/aos/1176344136
[25] M. Stone. Cross-validatory choice and assessment of statistical predictions. J. R. Stat. Soc. Ser. B Stat. Methodol. 36 (1974) 111-147. With discussion by G. A. Barnard, A. C. Atkinson, L. K. Chan, A. P. Dawid, F. Downton, J. Dickey, A. G. Baker, O. Barndorff-Nielsen, D. R. Cox, S. Giesser, D. Hinkley, R. R. Hocking and A. S. Young and with a reply by the authors. · Zbl 0308.62063
[26] M. Talagrand. New concentration inequalities in product spaces. Invent. Math. 126 (3) (1996) 505-563. · Zbl 0893.60001 · doi:10.1007/s002220050108
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.