×

Minimax mutual information approach for independent component analysis. (English) Zbl 1089.68590

Summary: Minimum output mutual information is regarded as a natural criterion for independent component analysis (ICA) and is used as the performance measure in many ICA algorithms. Two common approaches in information-theoretic ICA algorithms are minimum mutual information and maximum output entropy approaches. In the former approach, we substitute some form of probability density function (pdf) estimate into the mutual information expression, and in the latter we incorporate the source pdf assumption in the algorithm through the use of nonlinearities matched to the corresponding cumulative density functions (cdf). Alternative solutions to ICA use higher-order cumulant-based optimization criteria, which are related to either one of these approaches through truncated series approximations for densities. In this article, we propose a new ICA algorithm motivated by the maximum entropy principle (for estimating signal distributions). The optimality criterion is the minimum output mutual information, where the estimated pdfs are from the exponential family and are approximate solutions to a constrained entropy maximization problem. This approach yields an upper bound for the actual mutual information of the output signals – hence, the name minimax mutual information ICA algorithm. In addition, we demonstrate that for a specific selection of the constraint functions in the maximum entropy density estimation procedure, the algorithm relates strongly to ICA methods using higher-order cumulants.

MSC:

68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] DOI: 10.1162/neco.1995.7.6.1129 · Zbl 05479838 · doi:10.1162/neco.1995.7.6.1129
[2] DOI: 10.1162/089976699300016863 · doi:10.1162/089976699300016863
[3] DOI: 10.1049/ip-f-2.1993.0054 · doi:10.1049/ip-f-2.1993.0054
[4] DOI: 10.1016/0165-1684(94)90029-9 · Zbl 0791.62004 · doi:10.1016/0165-1684(94)90029-9
[5] DOI: 10.1109/97.508167 · doi:10.1109/97.508167
[6] DOI: 10.1214/aos/1176342706 · Zbl 0283.62042 · doi:10.1214/aos/1176342706
[7] DOI: 10.1049/el:19970977 · doi:10.1049/el:19970977
[8] DOI: 10.1162/089976602317250942 · Zbl 0986.62021 · doi:10.1162/089976602317250942
[9] DOI: 10.1109/97.923043 · doi:10.1109/97.923043
[10] Hyvarinen A., Neural Computing Surveys 2 pp 94– (1999)
[11] DOI: 10.1109/72.761722 · doi:10.1109/72.761722
[12] DOI: 10.1103/PhysRev.106.620 · Zbl 0084.43701 · doi:10.1103/PhysRev.106.620
[13] DOI: 10.1109/89.841214 · doi:10.1109/89.841214
[14] DOI: 10.1214/aoms/1177704472 · Zbl 0116.11302 · doi:10.1214/aoms/1177704472
[15] DOI: 10.1109/78.542183 · doi:10.1109/78.542183
[16] DOI: 10.1016/S0165-1684(00)00260-7 · Zbl 1098.94576 · doi:10.1016/S0165-1684(00)00260-7
[17] DOI: 10.1109/TIT.1980.1056144 · Zbl 0429.94011 · doi:10.1109/TIT.1980.1056144
[18] DOI: 10.1109/89.242486 · doi:10.1109/89.242486
[19] DOI: 10.1162/neco.1997.9.7.1457 · doi:10.1162/neco.1997.9.7.1457
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.