On the stability of some controlled Markov chains and its applications to stochastic approximation with Markovian dynamic

Research output: Contribution to journalArticle (Academic Journal)

3 Citations (Scopus)

Abstract

We develop a practical approach to establish the stability, that is the recurrence in a given set, of a large class of controlled Markov chains. These processes arise in various areas of applied science and encompass important numerical methods. We show in particular how individual Lyapunov functions and associated drift conditions for the parametrised family of Markov transition probabilities and the parameter update can be combined to form Lyapunov functions for the joint process, leading to the proof of the desired stability property. Of particular interest is the fact that the approach applies even
in situations where the two components of the process present a time-scale separation, which is a crucial feature of practical situations. We then move on to show how such a recurrence property can be used in the context of stochastic approximation in order to prove the convergence of the parameter sequence,
including in the situation where the so-called stepsize is adaptively tuned. We finally show that the results apply to various algorithms of interest in computational statistics and cognate areas.
Original languageEnglish
Number of pages31
JournalAnnals of Applied Probability
DOIs
Publication statusPublished - 1 Feb 2015

Fingerprint Dive into the research topics of 'On the stability of some controlled Markov chains and its applications to stochastic approximation with Markovian dynamic'. Together they form a unique fingerprint.

  • Cite this