×

Self-learning control of finite Markov chains. (English) Zbl 0960.93001

Control Engineering (Boca Raton) 4. New York, NY: Marcel Dekker. 312 p. (1999).
The book presents approaches to the adaptive control of finite Markov chains. The structure of the Markov chains is assumed to be given but the transition probabilities are considered unknown a priori. The goal of the control process is to optimize a linear cost function subject to a set of linear inequality constraints. The time horizon of the control process is infinite and the problem is to optimize the asymptotic estimated cost function over time assuming the limit exists.
The book consists of eight chapters divided into two parts. The first four chapters introduce basic definitions and address the unconstrained optimization problem. The other four chapters of the second part deal with the theoretical aspects of the constrained cases and include one chapter containing simulation results for the various algorithms covered in the book.
The approach adopted throughout the book falls in the category of direct approaches to adaptive control where the control actions are directly estimated using the available information as in the learning automata literature. There appears to be no comparison with the indirect approaches relying on the so-called certainty equivalence principle in adaptive control. All the algorithms presented in the book are based on well-known stochastic approximation and gradient optimization techniques. The book is rich with theorems and mathematical descriptions but falls short in presenting viable practical applications. The content of the book is mostly not new and the chapters seem to be loosely coupled with each chapter containing a list of references with some redundancy.
The book may serve as a textbook for graduate study for engineers and mathematicians in the stochastic control area.

MSC:

93-02 Research exposition (monographs, survey articles) pertaining to systems and control theory
93E35 Stochastic learning and adaptive control
93E20 Optimal stochastic control
62L20 Stochastic approximation
PDFBibTeX XMLCite