×

On regularization algorithms in learning theory. (English) Zbl 1109.68088

Summary: In this paper we discuss a relation between Learning Theory and Regularization of linear ill-posed inverse problems. It is well known that Tikhonov regularization can be profitably used in the context of supervised learning, where it usually goes under the name of regularized least-squares algorithm. Moreover, the gradient descent algorithm was studied recently, which is an analog of Landweber regularization scheme. In this paper we show that a notion of regularization defined according to what is usually done for ill-posed inverse problems allows to derive learning algorithms which are consistent and provide a fast convergence rate. It turns out that for priors expressed in term of variable Hilbert scales in reproducing kernel Hilbert spaces our results for Tikhonov regularization match those of S. Smale and D. Zhou [Learning theory estimates via integral operators and their approximations, submitted for publication, retrievable at \(\langle\)http://www.tti-c.org/smale.html\(\rangle\) (2005)] and improve the results for Landweber iterations obtained by Y. Yao, L. Rosasco and A. Caponnetto [On early stopping in gradient descent learning, Constructive Approximation (2005), submitted for publication]. The remarkable fact is that our analysis shows that the same properties are shared by a large class of learning algorithms which are essentially all the linear regularization schemes. The concept of operator monotone functions turns out to be an important tool for the analysis.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Aronszajn, N., Theory of reproducing kernels, Trans. Amer. Math. Soc., 68, 337-404 (1950) · Zbl 0037.20701
[2] P.L. Bartlett, M.J. Jordan, J.D. McAuliffe, Convexity, classification, and risk bounds, Technical Report 638, Department of Statistics, U.C. Berkeley, 2003.; P.L. Bartlett, M.J. Jordan, J.D. McAuliffe, Convexity, classification, and risk bounds, Technical Report 638, Department of Statistics, U.C. Berkeley, 2003. · Zbl 1118.62330
[3] Birman, M. S.; Solomyak, M., Double operators integrals in Hilbert spaces, Integral Equations Oper. Theory, 131-168 (2003) · Zbl 1054.47030
[4] N. Bissantz, T. Hohage, A, Munk, F. Ruymgaart, Convergence rates of general regularization methods for statistical inverse problems and applications, preprint, 2006.; N. Bissantz, T. Hohage, A, Munk, F. Ruymgaart, Convergence rates of general regularization methods for statistical inverse problems and applications, preprint, 2006. · Zbl 1234.62062
[5] Bousquet, O.; Boucheron, S.; Lugosi, G., (Introduction to Statistical Learning Theory, Lectures Notes in Artificial Intelligence, vol. 3176 (2004), Springer: Springer Heidelberg, Germany, Wiley-Interscience Publication), 169-207 · Zbl 1120.68428
[6] O. Bousquet, S. Boucheron, G. Lugosi, Theory of classification: a survey of recent advances, ESAIM Probab. Statist. (2004), to appear.; O. Bousquet, S. Boucheron, G. Lugosi, Theory of classification: a survey of recent advances, ESAIM Probab. Statist. (2004), to appear. · Zbl 1136.62355
[7] A. Caponnetto, E. De Vito, Fast rates for regularized least-squares algorithm, Technical Report CBCL Paper 248/AI Memo 2005-033, Massachusetts Institute of Technology, Cambridge, MA, 2005.; A. Caponnetto, E. De Vito, Fast rates for regularized least-squares algorithm, Technical Report CBCL Paper 248/AI Memo 2005-033, Massachusetts Institute of Technology, Cambridge, MA, 2005. · Zbl 1129.68058
[8] A. Caponnetto, E. De Vito, Optimal rates for regularized least-squares algorithm, Foundation Comput. Math. (2005), accepted for publication.; A. Caponnetto, E. De Vito, Optimal rates for regularized least-squares algorithm, Foundation Comput. Math. (2005), accepted for publication. · Zbl 1129.68058
[9] A. Caponnetto, L. Rosasco, E. De Vito, A. Verri, Empirical effective dimensions and fast rates for regularized least-squares algorithm, Technical Report CBCL Paper 252/AI Memo 2005-019, Massachusetts Institute of Technology, Cambridge, MA, 2005.; A. Caponnetto, L. Rosasco, E. De Vito, A. Verri, Empirical effective dimensions and fast rates for regularized least-squares algorithm, Technical Report CBCL Paper 252/AI Memo 2005-019, Massachusetts Institute of Technology, Cambridge, MA, 2005.
[10] C. Carmeli, E. De Vito, A. Toigo, Reproducing kernel Hilbert spaces and mercer theorem, eprint arXiv: math/0504071. Available at \(\langle;\) http://arxiv.org \(\rangle;\); C. Carmeli, E. De Vito, A. Toigo, Reproducing kernel Hilbert spaces and mercer theorem, eprint arXiv: math/0504071. Available at \(\langle;\) http://arxiv.org \(\rangle;\)
[11] Cucker, F.; Smale, S., On the mathematical foundations of learning, Bull. Amer. Math. Soc. (N.S.), 39, 1, 1-49 (2002), (electronic) · Zbl 0983.68162
[12] E. De Vito, L. Rosasco, A. Caponnetto, Discretization error analysis for tikhonov regularization, Anal. Appl. (2005), to appear.; E. De Vito, L. Rosasco, A. Caponnetto, Discretization error analysis for tikhonov regularization, Anal. Appl. (2005), to appear. · Zbl 1088.65056
[13] De Vito, E.; Rosasco, L.; Caponnetto, A.; De Giovannini, U.; Odone, F., Learning from examples as an inverse problem, J. Mach. Learning Res., 6, 883-904 (2005) · Zbl 1222.68180
[14] R. DeVore, G. Kerkyacharian, D. Picard, V. Temlyakov, On mathematical methods of learning, Technical Report 2004:10, Industrial Mathematics Institute, Department of Mathematics University of South Carolina, retrievable at \(\langle;\) http://www.math/sc/edu/imip/04papers/0410.ps \(\rangle;\); R. DeVore, G. Kerkyacharian, D. Picard, V. Temlyakov, On mathematical methods of learning, Technical Report 2004:10, Industrial Mathematics Institute, Department of Mathematics University of South Carolina, retrievable at \(\langle;\) http://www.math/sc/edu/imip/04papers/0410.ps \(\rangle;\) · Zbl 1146.62322
[15] Devroye, L.; Györfi, L.; Lugosi, G., A Probabilistic Theory of Pattern Recognition, Applications of Mathematics, vol. 31 (1996), Springer: Springer New York · Zbl 0853.68150
[16] Engl, H. W.; Hanke, M.; Neubauer, A., Regularization of Inverse Problems, Mathematics and its Applications, vol. 375 (1996), Kluwer Academic Publishers Group: Kluwer Academic Publishers Group Dordrecht
[17] Evgeniou, T.; Pontil, M.; Poggio, T., Regularization networks and support vector machines, Adv. Comp. Math., 13, 1-50 (2000) · Zbl 0939.68098
[18] Györfi, L.; Kohler, M.; Krzyzak, A.; Walk, H., A Distribution-free Theory of Non-parametric Regression (1996), Springer Series in Statistics: Springer Series in Statistics New York
[19] F. Hansen, Operator inequalities associated to Jensen’s inequality, survey of “Classical Inequalities”, pp. 67-i98.; F. Hansen, Operator inequalities associated to Jensen’s inequality, survey of “Classical Inequalities”, pp. 67-i98. · Zbl 1040.47012
[20] Mathé, P.; Pereverzev, S., Moduli of continuity for operator monotone functions, Numer. Funct. Anal. Optim., 23, 623-631 (2002) · Zbl 1027.47006
[21] Mathé, P.; Pereverzev, S., Geometry of linear ill-posed problems in variable Hilbert scale, Inverse Problems, 19, 789-803 (2003) · Zbl 1026.65040
[22] P. Mathé, S. Pereverzev, Regularization of some linear ill-posed problems with discretized random noisy data, Math. Comput. (2005), accepted for publication.; P. Mathé, S. Pereverzev, Regularization of some linear ill-posed problems with discretized random noisy data, Math. Comput. (2005), accepted for publication.
[23] Pinelis, I. F.; Sakhanenko, A. I., Remarks on inequalities for probabilities of large deviations, Theory Probab. Appl., 30, 1, 143-148 (1985) · Zbl 0583.60023
[24] L. Rosasco, E. De Vito, A. Verri, Spectral methods for regularization in learning theory, Technical Report DISI-TR-05-18, DISI, Universitá degli Studi di Genova, Italy, retrievable at \(\langle;\) http://www.disi.unige.it/person/RosascoL \(\rangle;\); L. Rosasco, E. De Vito, A. Verri, Spectral methods for regularization in learning theory, Technical Report DISI-TR-05-18, DISI, Universitá degli Studi di Genova, Italy, retrievable at \(\langle;\) http://www.disi.unige.it/person/RosascoL \(\rangle;\)
[25] Rudin, W., Functional Analysis, International Series in Pure and Applied Mathematics (1991), McGraw-Hill: McGraw-Hill Princeton
[26] Schwartz, L., Sous-espaces hilbertiens d’espaces vectoriels topologiques et noyaux associés (noyaux reproduisants), J. Analyse Math., 13, 115-256 (1964) · Zbl 0124.06504
[27] S. Smale, D. Zhou, Learning theory estimates via integral operators and their approximations, submitted for publication, retrievable at \(\langle;\) http://www.tti-c.org/smale.html \(\rangle;\); S. Smale, D. Zhou, Learning theory estimates via integral operators and their approximations, submitted for publication, retrievable at \(\langle;\) http://www.tti-c.org/smale.html \(\rangle;\)
[28] S. Smale, D. Zhou, Shannon sampling ii: connections to learning theory, retrievable at \(\langle;\) http://www.tti-c.org/smale.html \(\rangle;\); S. Smale, D. Zhou, Shannon sampling ii: connections to learning theory, retrievable at \(\langle;\) http://www.tti-c.org/smale.html \(\rangle;\) · Zbl 1107.94008
[29] Tsybakov, A. B., Optimal aggregation of classifiers in statistical learning, Ann. Statist., 32, 135-166 (2004) · Zbl 1105.62353
[30] van de Geer, S. A., Empirical Process in M-Estimation, Cambridge Series in Statistical and Probabilistic Mathematics (2000), Cambridge University Press: Cambridge University Press Cambridge
[31] van de Vaart, S. W.; Wellner, J. A., Weak Convergence and Empirical Process Theory, Springer Series in Statistics, New York, 1996 (1996), Springer: Springer New York
[32] Vapnik, V. N., Statistical learning theory, Adaptive and Learning Systems for Signal Processing, Communications, and Control, Wiley, New York (1998), a Wiley-Interscience Publication
[33] Y. Yao, L. Rosasco, A. Caponnetto, On early stopping in gradient descent learning, Constructive Approximation (2005), submitted for publication.; Y. Yao, L. Rosasco, A. Caponnetto, On early stopping in gradient descent learning, Constructive Approximation (2005), submitted for publication. · Zbl 1125.62035
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.