Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions

Tom Bewley; Freddy Lecue

Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions

Tom Bewley, Freddy Lecue

Bristol Doctoral College

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

8 Citations (Scopus)

18 Downloads (Pure)

Abstract

The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.

Original language	English
Title of host publication	The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022)
Publisher	The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)
Pages	118-126
Number of pages	9
ISBN (Electronic)	9781450392136
Publication status	Published - 13 May 2022
Event	AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems - , New Zealand Duration: 9 May 2022 → 13 May 2022

Publication series

Name	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems
ISSN (Print)	1548-8403
ISSN (Electronic)	1558-2914

Conference

Conference	AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems
Country/Territory	New Zealand
Period	9/05/22 → 13/05/22

Keywords

cs.LG
cs.AI

Access to Document

Full-text PDF (accepted manuscript)
This is the accepted author manuscript (AAM) of the article which has been made Open Access under the University of Bristol's Scholarly Works Policy. The final published version (Version of Record) can be found on the publisher's website. The copyright of any third-party content, such as images, remains with the copyright holder.
Accepted author manuscript, 4.3 MBLicence: CC BY

https://ifaamas.org/Proceedings/aamas2022/forms/contents.htm

Handle.net

Persistent link

Cite this

Bewley, T., & Lecue, F. (2022). Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. In The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022) (pp. 118-126). (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems ). The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). https://ifaamas.org/Proceedings/aamas2022/forms/contents.htm

Bewley, Tom ; Lecue, Freddy. / Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022). The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. pp. 118-126 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems ).

@inproceedings{73533c9bf5614b33bc7b26239b03b7ea,

title = "Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions",

abstract = "The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.",

keywords = "cs.LG, cs.AI",

author = "Tom Bewley and Freddy Lecue",

year = "2022",

month = may,

day = "13",

language = "English",

series = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems ",

publisher = "The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)",

pages = "118--126",

booktitle = "The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022)",

note = "AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems ; Conference date: 09-05-2022 Through 13-05-2022",

}

Bewley, T & Lecue, F 2022, Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. in The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022). Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems , The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), pp. 118-126, AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems, New Zealand, 9/05/22. <https://ifaamas.org/Proceedings/aamas2022/forms/contents.htm>

Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. / Bewley, Tom; Lecue, Freddy.
The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022). The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS), 2022. p. 118-126 (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems ).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions

AU - Bewley, Tom

AU - Lecue, Freddy

PY - 2022/5/13

Y1 - 2022/5/13

N2 - The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.

AB - The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.

KW - cs.LG

KW - cs.AI

UR - https://aamas2022-conference.auckland.ac.nz/

M3 - Conference Contribution (Conference Proceeding)

T3 - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems

SP - 118

EP - 126

BT - The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022)

PB - The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)

T2 - AAMAS ' 22: International Conference on Autonomous Agents and Multi-Agent Systems

Y2 - 9 May 2022 through 13 May 2022

ER -

Bewley T, Lecue F. Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions. In The Proceedings of the 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS-2022). The International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS). 2022. p. 118-126. (Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems ).

Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions

Abstract

Publication series

Conference

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this