Failing on Bias Mitigation: A Case Study on the Challenges of Fairness in Government Data

The potential for bias and unfairness in AI-supporting government services raises ethical and legal concerns. Using crime rate prediction with the Bristol City Council data as a case study, we examine how these issues persist. Rather than auditing real-world deployed systems or producing a generalizable benchmark, our position is that a case-based investigation to understand why bias mitigations applied to government data are not always effective. Experimental analysis reveals that the failure occurs not because of flaws in model architecture or metric selection, but due to the inherent properties of the data itself, further reinforcing that the origin of bias lies in the structure and history of government datasets. We then explore the reasons for the mitigation failures in predictive models on government data and highlight the potential sources of unfairness posed by data distribution shifts, accumulated historical bias, and delays in data release. This study provides a crucial exi stence proof and serves as a critical ‘early warning’ that biases in government data may persist even with standard mitigation methods.