×

Unsupervised learning in neural computation. (English) Zbl 1061.68129

Summary: In this article, we consider unsupervised learning from the point of view of applying neural computation on signal and data analysis problems. The article is an introductory survey, concentrating on the main principles and categories of unsupervised learning. In neural computation, there are two classical categories for unsupervised learning methods and models: first, extensions of principal component analysis and factor analysis, and second, learning vector coding or clustering methods that are based on competitive learning. These are covered in this article. The more recent trend in unsupervised learning is to consider this problem in the framework of probabilistic generative models. If it is possible to build and estimate a model that explains the data in terms of some latent variables, key insights may be obtained into the true nature and structure of the data. This approach is also briefly reviewed.

MSC:

68T05 Learning and adaptive systems in artificial intelligence

Software:

FastICA; SOM
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Amari, S.-I.; Cichocki, A.; Yang, H., A new learning algorithm for blind source separation, (Touretzky, D. S.; Moser, M. C.; Hasselmo, M. E., Advances in Neural Information Processing Systems, Vol. 8 (1996), MIT Press: MIT Press Cambridge, MA), 757-763
[2] Attias, H., Independent factor analysis, Neural Comput., 11, 4, 803-851 (1999)
[3] Baldi, P.; Hornik, K., Learning in linear neural networksa survey, IEEE Trans. Neural Networks, 6, 4, 837-858 (1995)
[4] Barlow, H., Unsupervised learning, Neural Comput., 1, 295-311 (1989)
[5] Bell, A.; Sejnowski, T., An information-maximization approach to blind separation and blind deconvolution, Neural Comput., 7, 1129-1159 (1995)
[6] Bishop, C.; Svensen, M.; Williams, C., GTMthe generative topographic mapping, Neural Comput., 10, 215-234 (1998)
[7] Bourlard, H.; Kamp, Y., Auto-association by multilayer perceptrons and singular value decomposition, Biol. Cybernet., 59, 291-294 (1988) · Zbl 0651.92006
[8] Cardoso, J.-F., Blind signal separationstatistical principles, Proc. IEEE, 9, 10, 2009-2025 (1998)
[9] Cichocki, A.; Unbehauen, R., Robust neural networks with on-line learning for blind identification and blind separation of sources, IEEE Trans. Circuits and Systems, 43, 11, 894-906 (1996)
[10] G. Cottrell, P. Munro, D. Zipser, Learning internal representations from gray-scale images: an example of extensional programming, Proc. 9th Ann. Conf. of the Cognitive Science Society, Seattle, WA, 1987, pp. 462-473.; G. Cottrell, P. Munro, D. Zipser, Learning internal representations from gray-scale images: an example of extensional programming, Proc. 9th Ann. Conf. of the Cognitive Science Society, Seattle, WA, 1987, pp. 462-473.
[11] Der, R.; Herrmann, M., Second-order learning in self-organizing maps, (Oja, E.; Kaski, S., Kohonen Maps (1999), Elsevier: Elsevier Amsterdam), 293-302
[12] Diamantaras, K.; Kung, S., Principal Component Neural NetworksTheory and Applications (1996), Wiley: Wiley New York · Zbl 0911.68157
[13] The FastICA package, Available from www.cis.hut.fi/projects/ica/; The FastICA package, Available from www.cis.hut.fi/projects/ica/
[14] P. Foldiak, Adaptive network for optimal linear feature extraction, Proc. Internat. J. Conf. on Neural Networks, Washington, DC, 1989, pp. 401-406.; P. Foldiak, Adaptive network for optimal linear feature extraction, Proc. Internat. J. Conf. on Neural Networks, Washington, DC, 1989, pp. 401-406.
[15] Funahashi, K., On the approximate realization of continuous mappings by neural networks, Neural Networks, 2, 183-192 (1989)
[16] Grossberg, S., Direct perception or adaptive resonance?, Behav. Brain Sci., 3, 385 (1980)
[17] Grossberg, S., Neural Networks and Natural Intelligence (1988), MIT Press: MIT Press Cambridge, MA
[18] Harman, H. H., Modern Factor Analysis (1967), University of Chicago Press: University of Chicago Press Chicago · Zbl 0161.39805
[19] Haykin, S., Neural Networks-Comprehensive Foundation (1998), MacMillan College Publ. Co.: MacMillan College Publ. Co. New York · Zbl 0934.68076
[20] Heskes, T., Energy functions for self-organizing maps, (Oja, E.; Kaski, S., Kohonen Maps (1999), Elsevier: Elsevier Amsterdam), 303-316
[21] Hinton, G.; Revow, M.; Dayan, P., Recognizing handwritten digits using mixtures of linear models, (Tesauro, G.; Touretzky, D.; Leen, T., Advances in Neural Information Processing Systems, Vol. 6 (1995), Kauffman: Kauffman San Mateo), 1015-1022
[22] Hinton, G.; Sejnowski, T. J., Unsupervised Learning—Foundations of Neural Computation (1999), MIT Press: MIT Press Cambridge, MA
[23] Hornik, M.; Stinchcombe, M.; White, H., Multilayer feedforward networks are universal approximators, Neural Networks, 2, 359-368 (1989) · Zbl 1383.92015
[24] Hyvärinen, A., Fast and robust fixed-point algorithms for independent component analysis, IEEE Trans. Neural Networks, 10, 3, 626-634 (1999)
[25] A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis, Wiley, New York, (2001).; A. Hyvärinen, J. Karhunen, E. Oja, Independent Component Analysis, Wiley, New York, (2001).
[26] Hyvärinen, A.; Oja, E., A fast fixed-point algorithm for independent component analysis, Neural Comput., 9, 7, 1483-1492 (1997)
[27] Japkowitz, N.; Hanson, S.; Gluck, A., Nonlinear autoassociation is not equivalent to PCA, Neural Comput., 12, 3, 531-545 (2000)
[28] Jutten, C.; Herault, J., Blind separation of sources, part Ian adaptive algorithm based on neuromimetic architecture, Signal Process., 24, 1-10 (1991) · Zbl 0729.73650
[29] Karhunen, J.; Oja, E.; Wang, L.; Vigario, R.; Joutsensalo, J., A class of neural networks for independent component analysis, IEEE Trans. Neural Networks, 8, 3, 486-504 (1997)
[30] M. Kendall, A. Stuart, The Advanced Theory of Statistics, Vols. 1-3, MacMillan, New York, 1976-1979.; M. Kendall, A. Stuart, The Advanced Theory of Statistics, Vols. 1-3, MacMillan, New York, 1976-1979.
[31] Kohonen, T., Self-Organizing Maps (1995), Springer: Springer Berlin, (3rd Edition, 2001)
[32] Kohonen, T.; Kaski, S.; Lagus, K.; Salojärvi, J.; Paatero, V.; Saarela, A., Self organization of massive document collection, IEEE Trans. Neural Networks, 11, 3, 574-585 (2000)
[33] Lappalainen, H.; Honkela, A., Bayesian nonlinear independent component analysis by multi-layer perceptrons, (Girolami, M., Advances in Independent Component Analysis (2000), Springer: Springer Berlin), 93-121
[34] von der Malsburg, C., Self-organization of orientation sensitive cells in the striate cortex, Kybernetik, 14, 85-100 (1973)
[35] Oja, E., A simplified neuron model as a principal components analyzer, J. Math. Biol., 15, 267-273 (1982) · Zbl 0488.92012
[36] Oja, E., Subspace Methods of Pattern Recognition (1983), RSP and Wiley: RSP and Wiley Letchworth
[37] E. Oja, Data compression, feature extraction, and autoassociation in feedforward neural networks, Proc. ICANN-91, Espoo, Finland, June 24-28, 1991, pp. 737-745.; E. Oja, Data compression, feature extraction, and autoassociation in feedforward neural networks, Proc. ICANN-91, Espoo, Finland, June 24-28, 1991, pp. 737-745.
[38] Oja, E., Principal components, minor components, and linear neural networks, Neural Networks, 5, 927-935 (1992)
[39] Oja, E., The nonlinear PCA learning rule in independent component analysis, Neurocomputing, 17, 1, 25-46 (1997)
[40] E. Oja, S. Kaski (Eds.), Kohonen Maps, Elsevier, Amsterdam, 1999.; E. Oja, S. Kaski (Eds.), Kohonen Maps, Elsevier, Amsterdam, 1999.
[41] Oja, E.; Wang, L., Neural fittingrobustness by anti-Hebbian learning, Neurocomputing, 12, 155-170 (1976)
[42] Ritter, H.; Martinetz, T.; Schulten, K., Neural Computation and Self-Organizing MapsAn Introduction (1992), Addison-Wesley: Addison-Wesley Reading, MA
[43] Roweis, S.; Ghahramani, Z., A unifying review of linear Gaussian models, Neural Comput., 11, 2, 305-346 (1999)
[44] Rubner, J.; Tavan, P., A self-organizing network for principal component analysis, Europhys. Lett., 10, 7, 693-698 (1989)
[45] Sanger, T., Optimal unsupervised learning in a single-layered linear feedforward network, Neural Networks, 2, 459-473 (1989)
[46] Schalkoff, R., Pattern Recognition—Statistical, Structural, and Neural Approaches (1992), Wiley: Wiley New York
[47] Bibliography of SOM papers: a reference list of over 4000 studies on the self-organizing map, Available at www.cis.hut.fi/research/som-bibl/; Bibliography of SOM papers: a reference list of over 4000 studies on the self-organizing map, Available at www.cis.hut.fi/research/som-bibl/
[48] The SOM Toolbox for Matlab, Available at www.cis.hut.fi/projects/somtoolbox/; The SOM Toolbox for Matlab, Available at www.cis.hut.fi/projects/somtoolbox/
[49] Tipping, M. E.; Bishop, C. M., Mixtures of probabilistic principal component analyzers, Neural Comput., 11, 2, 443-482 (1999)
[50] H. Valpola, Bayesian ensemble learning for nonlinear factor analysis, Acta Polytech. Scand. Math. 108 (2000); D.Sc. Thesis, Helsinki University of Technology.; H. Valpola, Bayesian ensemble learning for nonlinear factor analysis, Acta Polytech. Scand. Math. 108 (2000); D.Sc. Thesis, Helsinki University of Technology. · Zbl 0986.62046
[51] VanHulle, M., Faithful Representations and Topographic Maps (2000), Wiley: Wiley New York
[52] Villmann, T., Topology preservation in self-organizing maps, (Oja, E.; Kaski, S., Kohonen Maps (1999), Elsevier: Elsevier Amsterdam), 267-292
[53] Wang, L.; Karhunen, J., A unified neural bigradient algorithm for robust PCA and MCA, Internat. J. Neural Systems, 7, 1, 53-67 (1996)
[54] Webb, A., Statistical Pattern Recognition (1999), Arnold: Arnold Paris · Zbl 0968.68540
[55] Xu, L., Temporal BYY learning for state space approach, hidden Markov model, and blind source separation, IEEE Trans. Signal Process., 48, 2132-2144 (2000) · Zbl 0982.94004
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.