Skip to content

Adaptive Optimal Control via Continuous-Time Q-Learning for Unknown Nonlinear Affine Systems

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Original languageEnglish
Title of host publication2019 IEEE Conference on Decision and Control (CDC)
Publisher or commissioning bodyInstitute of Electrical and Electronics Engineers (IEEE)
DateAccepted/In press - 19 Jul 2019
EventIEEE Conference on Decision and Control - Nice, France
Duration: 11 Dec 201913 Dec 2019
Conference number: 58

Publication series

NameIEEE Conference on Decision and Control


ConferenceIEEE Conference on Decision and Control
Abbreviated titleCDC2019
Internet address


This paper proposes two novel adaptive optimal control algorithms for continuous-time nonlinear affine systems based on reinforcement learning: i) generalized policy iteration (GPI) and ii) Q-learning. As a result, the a priori knowledge of the system drift f (x) is not needed via GPI, which gives us a partially model-free and online solution. We then for the first time extend the idea of Q-learning to the nonlinear continuous time optimal control problem in a noniterative manner. Thisleads to a completely model-free method where neither the system drift f (x) nor the input gain g(x) is needed. For both methods, the adaptive critic and actor are continuously and simultaneously updating each other without iterative steps, which effectively avoids the hybrid structure and the need or an initial stabilizing control policy. Moreover, finite-time convergence is guaranteed by using a sliding mode technique in the new adaptive approach, where the persistent excitation (PE) condition can be directly verified online. We also prove the overall Lyapunov stability and demonstrate the effectiveness of the proposed algorithms using numerical examples.

    Research areas

  • adaptive critic, approximate dynamic programming, reinforcement learning, nonlinear systems, Q-learning, adaptive optimal control


IEEE Conference on Decision and Control

Abbreviated titleCDC2019
Conference number58
Duration11 Dec 201913 Dec 2019
Web address (URL)
Degree of recognitionInternational event

Event: Conference



  • Full-text PDF (accepted author manuscript)

    Rights statement: This is the accepted author manuscript (AAM). The final published version (version of record) is available online via IEEE at [insert hyperlink]. Please refer to any applicable terms of use of the publisher.

    Accepted author manuscript, 182 KB, PDF document

    Embargo ends: 1/01/99

    Request copy

    Licence: Other

View research connections

Related faculties, schools or groups