A System-theoretic Note on Model-Based Actor-Critic

In this paper, we provide preliminary results toward the direction of using systems theory tools for the design and analysis of reinforcement learning algorithms. Specifically, we analyze the convergence properties of a model-based scheme with an actor-critic structure. The distinctive feature of our scheme is that the actor and critic updates are equipped with auxiliary variables that allow for the use of a constant step size. Although idealized due to the assumption on access to the underlying Markov Decision Process (MDP), the investigated setting is a starting point toward a genuine (model-free) actorcritic scheme. A key contribution is the interpretation of this algorithmic framework in terms of discrete-time, interconnected dynamical systems. Specifically, by resorting to Singular Perturbations (S P), we reinterpret the whole algorithm as the interconnection of a fast subsystem (auxiliary variables' mechanism), an intermediate one (critic), and a slow one (actor). We separately analyze three auxiliary systems each corresponding to one of the identified subsystems. These preparatory results, combined with SP and LaSalle arguments, allow us to prove that the overall method asymptotically converges to a problem stationary point. Some numerical simulations confirm our theoretical findings.