Fernández-Gaucherand, Emmanuel; Arapostathis, Aristotle; Marcus, Steven I. On the adaptive control of a partially observable binary Markov decision process. (English) Zbl 0712.93063 Advances in computing and control, Sel. Pap. Int. Conf. Adv. Commun. Control Syst., Baton Rouge/LA (USA) 1988, Lect. Notes Control Inf. Sci. 130, 217-228 (1989). [For the entire collection see Zbl 0708.00019.] Some initial steps are taken toward the development of a methodology for the optimal adaptive control of markov chains with incomplete state observations and unknown model parameters. One set of problems for which some results are available, when all the parameters are known, are those involving quality control and machine maintenance/replacement. These results are reviewed; in particular, the adaptive control of a problem with simple structure is studied: the two-state binary replacement problem. An enforced certainty equivalent adaptive control algorithm is formulated, with a recursive parameter estimator of the stochastic gradient type. Initial results are given in the direction of using the ODE method of analysis to show convergence and optimality of the combined estimation and control scheme. However, the presence of feedback makes a complete analysis difficult. The principal technical contribution of the paper is the derivation of some (uniformly geometric) ergodic properties for an augmented state process. More recent work by the authors substantially improve on the results of this paper. Reviewer: E.Fernández-Gaucherand Cited in 1 Document MSC: 93E20 Optimal stochastic control 93C40 Adaptive control/observation systems 90B25 Reliability, availability, maintenance, inspection in operations research Keywords:markov chains; incomplete state observations; adaptive control; two-state binary replacement problem Citations:Zbl 0708.00019 PDFBibTeX XML