×

PAC learning under helpful distributions. (English) Zbl 0992.68118

Summary: A PAC teaching model – under helpful distributions – is proposed which introduces the classical ideas of teaching models within the PAC setting: a polynomial-sized teaching set is associated with each target concept; the criterion of success is PAC identification; an additional parameter, namely the inverse of the minimum probability assigned to any example in the teaching set, is associated with each distribution; the learning algorithm running time takes this new parameter into account. An Occam razor theorem and its converse are proved. Some classical classes of Boolean functions, such as decision lists, DNF and CNF formulas are proved learnable in this model. Comparisons with other teaching models are made: learnability in the Goldman and Mathias model implies PAC learnability under helpful distributions. Note that decision lists and DNF are not known to be learnable in the Goldman and Mathias model. A new simple PAC model, where “simple” refers to Kolmogorov complexity, is introduced. We show that most learnability results obtained within previously defined simple PAC models can be simply derived from more general results in our model.

MSC:

68Q32 Computational learning theory
68Q30 Algorithmic information theory (Kolmogorov complexity, etc.)
PDFBibTeX XMLCite
Full Text: DOI Numdam EuDML Link

References:

[1] D. Angluin , Learning Regular Sets from Queries and Counterexamples . Inform. and Comput. 75 ( 1987 ) 87 - 106 . MR 916360 | Zbl 0636.68112 · Zbl 0636.68112 · doi:10.1016/0890-5401(87)90052-6
[2] D. Angluin , Queries and Concept Learning . Machine Learning 2 ( 1988 ) 319 - 342 .
[3] G.M. Benedek and A. Itai , Nonuniform Learnability , in ICALP ( 1988 ) 82 - 92 . MR 1023628 | Zbl 0649.68080 · Zbl 0649.68080
[4] A. Blumer , A. Ehrenfeucht , D. Haussler and M.K. Warmuth , Occam’s Razor . Inform. Process. Lett. 24 ( 1987 ) 377 - 380 . Zbl 0653.68084 · Zbl 0653.68084 · doi:10.1016/0020-0190(87)90114-1
[5] R. Board and L. Pitt , On the Necessity of Occam Algorithms . Theoret. Comput. Sci. 100 ( 1992 ) 157 - 184 . MR 1171438 | Zbl 0825.68544 · Zbl 0825.68544 · doi:10.1016/0304-3975(92)90367-O
[6] N.H. Bshouty , Exact Learning Boolean Function via the Monotone Theory . Inform. and Comput. 123 ( 1995 ) 146 - 153 . MR 1358974 | Zbl 1096.68634 · Zbl 1096.68634 · doi:10.1006/inco.1995.1164
[7] J. Castro and J.L. Balcázar , Simple PAC learning of simple decision lists , in ALT 95, 6th International Workshop on Algorithmic Learning Theory. Springer, Lecture Notes in Comput. Sci. 997 ( 1995 ) 239 - 250 .
[8] J. Castro and D. Guijarro , PACS, simple-PAC and query learning . Inform. Process. Lett. 73 ( 2000 ) 11 - 16 . MR 1741500 · Zbl 1339.68130
[9] F. Denis , Learning regular languages from simple positive examples , Machine Learning. Technical Report LIFL 321 - 1998; http://www.lifl.fr/denis (to appear). Zbl 0983.68104 · Zbl 0983.68104 · doi:10.1023/A:1010826628977
[10] F. Denis , C. D’Halluin and R. Gilleron , PAC Learning with Simple Examples , in 13th Annual Symposium on Theoretical Aspects of Computer Science. Springer-Verlag, Lecture Notes in Comput. Sci. 1046 ( 1996 ) 231 - 242 . · Zbl 1379.68190
[11] F. Denis and R. Gilleron , PAC learning under helpful distributions , in Proc. of the 8th International Workshop on Algorithmic Learning Theory (ALT-97), edited by M. Li and A. Maruoka. Springer-Verlag, Berlin, Lecture Notes in Comput. Sci. 1316 ( 1997 ) 132 - 145 . MR 1707552 | Zbl 0887.68084 · Zbl 0887.68084
[12] E.M. Gold , Complexity of Automaton Identification from Given Data . Inform. and Control 37 ( 1978 ) 302 - 320 . MR 495194 | Zbl 0376.68041 · Zbl 0376.68041 · doi:10.1016/S0019-9958(78)90562-4
[13] S.A. Goldman and M.J. Kearns , On the Complexity of Teaching . J. Comput. System Sci. 50 ( 1995 ) 20 - 31 . MR 1322630 | Zbl 0939.68770 · Zbl 0939.68770 · doi:10.1006/jcss.1995.1003
[14] S.A. Goldman and H.D. Mathias , Teaching a Smarter Learner . J. Comput. System Sci. 52 ( 1996 ) 255 - 267 . MR 1393993 · Zbl 1152.68451 · doi:10.1006/jcss.1996.0020
[15] T. Hancock , T. Jiang , M. Li and J. Tromp , Lower Bounds on Learning Decision Lists and Trees . Inform. and Comput. 126 ( 1996 ) 114 - 122 . MR 1391107 | Zbl 0856.68121 · Zbl 0856.68121 · doi:10.1006/inco.1996.0040
[16] D. Haussler , M. Kearns , N. Littlestone and M.K. Warmuth , Equivalence of Models for Polynomial Learnability . Inform. and Comput. 95 ( 1991 ) 129 - 161 . MR 1138115 | Zbl 0743.68115 · Zbl 0743.68115 · doi:10.1016/0890-5401(91)90042-Z
[17] C.D.L. Higuera , Characteristic Sets for Polynomial Grammatical Inference . Machine Learning 27 ( 1997 ) 125 - 137 . Zbl 0884.68107 · Zbl 0884.68107 · doi:10.1023/A:1007353007695
[18] M. Kearns , M. Li , L. Pitt and L.G. Valiant , Recent Results on Boolean Concept Learning , in Proc. of the Fourth International Workshop on Machine Learning ( 1987 ) 337 - 352 .
[19] M.J. Kearns and U.V. Vazirani , An Introduction to Computational Learning Theory . MIT Press ( 1994 ). MR 1331838
[20] M. Li and P.M.B. Vitányi , Learning simple concepts under simple distributions . SIAM J. Comput. 20 ( 1991 ) 911 - 935 . MR 1115658 | Zbl 0751.68055 · Zbl 0751.68055 · doi:10.1137/0220056
[21] M. Li and P. Vitányi , An introduction to Kolmogorov complexity and its applications , 2nd Edition. Springer-Verlag ( 1997 ). MR 1438307 | Zbl 0866.68051 · Zbl 0866.68051
[22] D.H. Mathias , DNF: If You Can’t Learn ’em , Teach ’em: An Interactive Model of Teaching, in Proc. of the 8th Annual Conference on Computational Learning Theory (COLT’95). ACM Press, New York ( 1995 ) 222 - 229 .
[23] B.K. Natarajan , Machine Learning: A Theoretical Approach . Morgan Kaufmann, San Mateo, CA ( 1991 ). MR 1137519 · Zbl 0722.68093
[24] B.K. Natarajan , On Learning Boolean Functions , in Proc. of the 19th Annual ACM Symposium on Theory of Computing. ACM Press ( 1987 ) 296 - 304 .
[25] J. Oncina and P. Garcia , Inferring regular languages in polynomial update time , in Pattern Recognition and Image Analysis ( 1992 ) 49 - 61 .
[26] R. Parekh and V. Honavar , On the Relationships between Models of Learning in Helpful Environments , in Proc. Fifth International Conference on Grammatical Inference ( 2000 ). Zbl 0974.68165 · Zbl 0974.68165
[27] R. Parekh and V. Honavar , Learning DFA from simple examples , in Proc. of the 8th International Workshop on Algorithmic Learning Theory (ALT-97), edited by M. Li and A. Maruoka. Springer, Berlin, Lecture Notes in Artificial Intelligence 1316 ( 1997 ) 116 - 131 . MR 1707551 | Zbl 0887.68083 · Zbl 0887.68083
[28] R. Parekh and V. Honavar , Simple DFA are polynomially probably exactly learnable from simple examples , in Proc. 16th International Conf. on Machine Learning ( 1999 ) 298 - 306 .
[29] R.L. Rivest , Learning Decision Lists . Machine Learning 2 ( 1987 ) 229 - 246 .
[30] K. Romanik , Approximate Testing and Learnability , in Proc. of the 5th Annual ACM Workshop on Computational Learning Theory, edited by D. Haussler. ACM Press, Pittsburgh, PA ( 1992 ) 327 - 332 .
[31] S. Salzberg , A. Delcher , D. Heath and S. Kasif , Learning with a Helpful Teacher , in Proc. of the 12th International Joint Conference on Artificial Intelligence, edited by R. Myopoulos and J. Reiter. Morgan Kaufmann, Sydney, Australia ( 1991 ) 705 - 711 . Zbl 0748.68065 · Zbl 0748.68065
[32] R.E. Schapire , The Strength of Weak Learnability . Machine Learning 5 ( 1990 ) 197 - 227 . Zbl 0747.68058 · Zbl 0747.68058
[33] A. Shinohara and S. Miyano , Teachability in Computational Learning . NEWGEN: New Generation Computing 8 ( 1991 ). Zbl 0712.68084 · Zbl 0712.68084 · doi:10.1007/BF03037091
[34] L.G. Valiant , A Theory of the Learnable . Commun. ACM 27 ( 1984 ) 1134 - 1142 . Zbl 0587.68077 · Zbl 0587.68077 · doi:10.1145/1968.1972
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.