Right for the Right Reason: Training Agnostic Networks

Sen Jia, Thomas Lansdall-Welfare, Nello Cristianini

Research output: Chapter in Book/Report/Conference proceedingConference Contribution (Conference Proceeding)

7 Citations (Scopus)
115 Downloads (Pure)


We consider the problem of a neural network being requested to classify images (or other inputs) without making implicit use of a “protected concept”, that is a concept that should not play any role in the decision of the network. Typically these concepts include information such as gender or race, or other contextual information such as image backgrounds that might be implicitly reflected in unknown correlations with other variables, making it insufficient to simply remove them from the input features. In other words, making accurate predictions is not good enough if those predictions rely on information that should not be used: predictive performance is not the only important metric for learning systems. We apply a method developed in the context of domain adaptation to address this problem of “being right for the right reason”, where we request a classifier to make a decision in a way that is entirely ‘agnostic’ to a given protected concept (e.g. gender, race, background etc.), even if this could be implicitly reflected in other attributes via unknown correlations. After defining the concept of an ‘agnostic model’, we demonstrate how the Domain-Adversarial Neural Network can remove unwanted information from a model using a gradient reversal layer.
Original languageEnglish
Title of host publicationAdvances in Intelligent Data Analysis XVII
Subtitle of host publication17th International Symposium, IDA 2018, ’s-Hertogenbosch, The Netherlands, October 24–26, 2018, Proceedings
PublisherSpringer, Cham
Number of pages11
ISBN (Electronic)9783030017682
ISBN (Print)9783030017675
Publication statusPublished - 5 Oct 2018

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743


  • Agnostic models
  • Explainable AI
  • Fairness in AI
  • Trust


Dive into the research topics of 'Right for the Right Reason: Training Agnostic Networks'. Together they form a unique fingerprint.

Cite this