\input zb-basic
\input zb-ioport
\iteman{io-port 00856669}
\itemau{Heskes, Tom}
\itemti{On Fokker-Planck approximations of on-line learning processes.}
\itemso{J. Phys. A, Math. Gen. 27, No.15, 5145-5160 (1994).}
\itemab
Summary: There are several ways to describe on-line learning in neural networks. The two major ones are a continuous-time master equation and a discrete-time random-walk equation. The random-walk equation is obtained in the case of fixed time intervals between subsequent learning steps, the master equation results when the time intervals are drawn from a Poisson distribution. Following van Kampen, we give a rigorous expansion of both the master and the random-walk equation in the limit of small learning parameters. The results explain the difference between the Fokker-Planck approaches proposed by {\it G. Radons}, {\it H. Schuster} and {\it D. Werner} [Fokker-Planck description of learning in backpropagation networks, Int. Neural Network Conf. Paris, 993-996 (1990)] and {\it L. K. Hansen}, {\it R. Pathria} and {\it P. Salamon} [J. Phys. A, Math. Gen. 26, No. 1, 63-71 (1993; Zbl 0773.92001)]. Furthermore, we find that the mathematical validity of these approaches is restricted to local properties of the learning process. Yet Fokker-Planck approaches are often suggested as models to study global properties, such as mean first passage times and stationary solutions. To check their accuracy and usefulness in these situations we compare simulations of two learning procedures with exactly the same drift and diffusion matrix, the only moments that are considered in Fokker-Planck approximation. The simulation shows that the mean first passage times for these two learning procedures diverge rather than converge for small learning parameters. We reach the conclusion that Fokker-Planck approaches are not accurate enough to compute global properties of on-line learning processes.
\itemrv{~}
\itemcc{}
\itemut{on-line learning; neural networks}
\itemli{doi:10.1088/0305-4470/27/15/015}
\end