Detecting and reducing heterogeneity of error in acoustic classification

Oliver C. Metcalf; Jos Barlow; Yves Bas; Erika Berenguer; Christian Devenish; Filipe França; Stuart Marsden; Charlotte Smith; Alexander C. Lees

doi:10.1111/2041-210X.13967

Detecting and reducing heterogeneity of error in acoustic classification

Oliver C. Metcalf^*, Jos Barlow, Yves Bas, Erika Berenguer, Christian Devenish, Filipe França, Stuart Marsden, Charlotte Smith, Alexander C. Lees

^*Corresponding author for this work

Research output: Contribution to journal › Article (Academic Journal) › peer-review

14 Citations (Scopus)

67 Downloads (Pure)

Abstract

Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere?

We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data.

We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F-score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species.

We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.

Original language	English
Pages (from-to)	2559-2571
Number of pages	13
Journal	Methods in Ecology and Evolution
Volume	13
Issue number	11
Early online date	31 Aug 2022
DOIs	https://doi.org/10.1111/2041-210X.13967 https://doi.org/10.1111/2041-210X.13967
Publication status	Published - 3 Nov 2022

Bibliographical note

Funding Information:
We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Renílson Freitas and Josué Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD‐RAS (CNPq/CAPES/PELD 441659/2016‐0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate).

Funding Information:
We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Renílson Freitas and Josué Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD-RAS (CNPq/CAPES/PELD 441659/2016-0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate).

Publisher Copyright:
© 2022 The Authors. Methods in Ecology and Evolution published by John Wiley & Sons Ltd on behalf of British Ecological Society.

Access to Document

10.1111/2041-210X.13967Licence: CC BY
10.1111/2041-210X.13967Licence: CC BY

Full-text PDF (final published version)
This is the final published version of the article (version of record). It first appeared online via Wiley at https://doi.org/10.1111/2041-210X.13967 . Please refer to any applicable terms of use of the publisher.
Final published version, 3.08 MBLicence: CC BY

Handle.net

Persistent link

Cite this

@article{7be1c9dcd143432684d1c8c50acc788f,

title = "Detecting and reducing heterogeneity of error in acoustic classification",

abstract = " Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F-score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.",

author = "Metcalf, \{Oliver C.\} and Jos Barlow and Yves Bas and Erika Berenguer and Christian Devenish and Filipe Fran{\c c}a and Stuart Marsden and Charlotte Smith and Lees, \{Alexander C.\}",

note = "Funding Information: We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Ren{\'i}lson Freitas and Josu{\'e} Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD‐RAS (CNPq/CAPES/PELD 441659/2016‐0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate). Funding Information: We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Ren{\'i}lson Freitas and Josu{\'e} Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD-RAS (CNPq/CAPES/PELD 441659/2016-0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate). Publisher Copyright: {\textcopyright} 2022 The Authors. Methods in Ecology and Evolution published by John Wiley \& Sons Ltd on behalf of British Ecological Society.",

year = "2022",

month = nov,

day = "3",

doi = "10.1111/2041-210X.13967",

language = "English",

volume = "13",

pages = "2559--2571",

journal = "Methods in Ecology and Evolution",

issn = "2041-210X",

publisher = "Wiley",

number = "11",

}

TY - JOUR

T1 - Detecting and reducing heterogeneity of error in acoustic classification

AU - Metcalf, Oliver C.

AU - Barlow, Jos

AU - Bas, Yves

AU - Berenguer, Erika

AU - Devenish, Christian

AU - França, Filipe

AU - Marsden, Stuart

AU - Smith, Charlotte

AU - Lees, Alexander C.

N1 - Funding Information: We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Renílson Freitas and Josué Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD‐RAS (CNPq/CAPES/PELD 441659/2016‐0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate). Funding Information: We would like to thank the RAS field and laboratory assistants: Marcos Oliveira, Gilson Oliveira, Renílson Freitas and Josué Jesus de Oliveira for their hard work and assistance, without whom this would not be possible. We are also grateful to Joice Ferreira and Liana Chessini Rossi for logistical field support in Brazil. Additional thanks to the Cornell Lab of Ornithology, particularly Matthew Medler and Jay McGowan for advising and helping with data from the Macaulay Library. Fieldwork in Brazil and later analysis was supported by research grants ECOFOR (NE/K016431/1), and AFIRE (NE/P004512/1), PELD-RAS (CNPq/CAPES/PELD 441659/2016-0) and the BNP Paribas Foundation's Climate and Biodiversity Initiative (Project Bioclimate). Publisher Copyright: © 2022 The Authors. Methods in Ecology and Evolution published by John Wiley & Sons Ltd on behalf of British Ecological Society.

PY - 2022/11/3

Y1 - 2022/11/3

N2 - Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F-score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.

AB - Passive acoustic monitoring can be an effective method for monitoring species, allowing the assembly of large audio datasets, removing logistical constraints in data collection and reducing anthropogenic monitoring disturbances. However, the analysis of large acoustic datasets is challenging and fully automated machine learning processes are rarely developed or implemented in ecological field studies. One of the greatest uncertainties hindering the development of these methods is spatial generalisability—can an algorithm trained on data from one place be used elsewhere? We demonstrate that heterogeneity of error across space is a problem that could go undetected using common classification accuracy metrics. Second, we develop a method to assess the extent of heterogeneity of error in a random forest classification model for six Amazonian bird species. Finally, we propose two complementary ways to reduce heterogeneity of error, by (i) accounting for it in the thresholding process and (ii) using a secondary classifier that uses contextual data. We found that using a thresholding approach that accounted for heterogeneity of precision error reduced the coefficient of variation of the precision score from a mean of 0.61 ± 0.17 (SD) to 0.41 ± 0.25 in comparison to the initial classification with threshold selection based on F-score. The use of a secondary, contextual classification with thresholding selection accounting for heterogeneity of precision reduced it further still, to 0.16 ± 0.13, and was significantly lower than the initial classification in all but one species. Mean average precision scores increased, from 0.66 ± 0.4 for the initial classification, to 0.95 ± 0.19, a significant improvement for all species. We recommend assessing—and if necessary correcting for—heterogeneity of precision error when using automated classification on acoustic data to quantify species presence as a function of an environmental, spatial or temporal predictor variable.

UR - https://doi.org/10.1111/2041-210X.13967

U2 - 10.1111/2041-210X.13967

DO - 10.1111/2041-210X.13967

M3 - Article (Academic Journal)

SN - 2041-210X

VL - 13

SP - 2559

EP - 2571

JO - Methods in Ecology and Evolution

JF - Methods in Ecology and Evolution

IS - 11

ER -

Detecting and reducing heterogeneity of error in acoustic classification

Abstract

Bibliographical note

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this