A quantitative comparison of manual vs. automated facial coding using real life observations of fathers

Romana Burgess; Iryna Culpin; Helen Bould; Rebecca Pearson; Ian Nabney

doi:10.1007/978-3-031-34586-9_25

A quantitative comparison of manual vs. automated facial coding using real life observations of fathers

Romana Burgess^*, Iryna Culpin, Helen Bould, Rebecca Pearson, Ian Nabney

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

Abstract

This work explores the application of an automated facial recognition software “FaceReader” [1] to videos of fathers (n = 36), taken using headcams worn by their infants during interactions in the home. We evaluate the use of FaceReader as an alternative method to manual coding – which is both time and labour intensive – and advance understanding of the usability of this software in naturalistic interactions. Using video data taken from the Avon Longitudinal Study of Parents and Children (ALSPAC), we first manually coded fathers’ facial expressions according to an existing coding scheme, and then processed the videos using FaceReader. We used contingency tables and multivariate logistic regression models to compare the manual and automated outputs. Our results indicated low levels of facial recognition by FaceReader in naturalistic interactions (approximately 25.17% compared to manual coding), and we discuss potential causes for this (e.g., problems with lighting, the headcams themselves, and speed of infant movement). However, our logistic regression models showed that when the face was found, FaceReader predicted manually coded expressions with a mean accuracy of M = 0.84 (range = 0.67–0.94), sensitivity of M = 0.64 (range = 0.27–0.97), and specificity of M = 0.81 (range = 0.51–0.97).

Original language	English
Title of host publication	Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings
Subtitle of host publication	- 16th EAI International Conference, PervasiveHealth 2022, Proceedings
Editors	Athanasios Tsanas, Andreas Triantafyllidis
Publisher	Springer Nature
Pages	379-396
Number of pages	18
Volume	48
ISBN (Electronic)	978-3-031-34586-9
ISBN (Print)	978-3-031-34585-2
DOIs	https://doi.org/10.1007/978-3-031-34586-9_25
Publication status	Published - 2023
Event	16th EAI International Conference on Pervasive Computing Technologies for Healthcare, PH 2022 - Thessaloniki, Greece Duration: 12 Dec 2022 → 14 Dec 2022

Publication series

Name	Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST
Volume	488 LNICST
ISSN (Print)	1867-8211
ISSN (Electronic)	1867-822X

Conference

Conference	16th EAI International Conference on Pervasive Computing Technologies for Healthcare, PH 2022
Country/Territory	Greece
City	Thessaloniki
Period	12/12/22 → 14/12/22

Bibliographical note

Funding Information:
We are extremely grateful to all the families who took part in the ALSPAC study, the midwives for their help in recruiting them, and the whole ALSPAC team, which includes interviewers, computer and laboratory technicians, clerical workers, research scientists, volunteers, managers, receptionists, and nurses. The UK Medical Research Council and Wellcome (Grant ref: 217065/Z/19/Z) and the University of Bristol provide core support for ALSPAC. We would also like to thank the two double coders who were involved in this work, Lottie Relph and Maddy Stephens. This publication is the work of the authors RB, IC, HB, and RP, who will serve as guarantors for the contents of this paper. A comprehensive list of grants funding is available on the ALSPAC website (http://www.bristol.ac.uk/alspac/external/documents/grant-acknowledgements.pdf); This work is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 758813; MHINT). Additionally, RB was supported by the Engineering and Physical Sciences Research Council (EPSRC) Digital Health and Care Centre for Doctoral Training (CDT) at the University of Bristol (UKRI Grant No. EP/S023704/1). IC was supported by the Wellcome Trust Research Fellowship in Humanities and Social Science (Grant ref: 212664/Z/18/Z).

Publisher Copyright:
© 2023, ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.

Keywords

ALSPAC
Automated facial coding
FaceReader

Access to Document

10.1007/978-3-031-34586-9_25

https://doi.org/10.1007/978-3-031-34586-9_25Licence: Unspecified

Handle.net

Persistent link

Cite this

Burgess, R., Culpin, I., Bould, H., Pearson, R., & Nabney, I. (2023). A quantitative comparison of manual vs. automated facial coding using real life observations of fathers. In A. Tsanas, & A. Triantafyllidis (Eds.), Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings: - 16th EAI International Conference, PervasiveHealth 2022, Proceedings (Vol. 48, pp. 379-396). (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 488 LNICST). Springer Nature. https://doi.org/10.1007/978-3-031-34586-9_25

Burgess, Romana ; Culpin, Iryna ; Bould, Helen et al. / A quantitative comparison of manual vs. automated facial coding using real life observations of fathers. Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings: - 16th EAI International Conference, PervasiveHealth 2022, Proceedings. editor / Athanasios Tsanas ; Andreas Triantafyllidis. Vol. 48 Springer Nature, 2023. pp. 379-396 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST).

@inproceedings{b3476baf3acf4f9c8eb6e2475859d8a2,

title = "A quantitative comparison of manual vs. automated facial coding using real life observations of fathers",

abstract = "This work explores the application of an automated facial recognition software “FaceReader” [1] to videos of fathers (n = 36), taken using headcams worn by their infants during interactions in the home. We evaluate the use of FaceReader as an alternative method to manual coding – which is both time and labour intensive – and advance understanding of the usability of this software in naturalistic interactions. Using video data taken from the Avon Longitudinal Study of Parents and Children (ALSPAC), we first manually coded fathers{\textquoteright} facial expressions according to an existing coding scheme, and then processed the videos using FaceReader. We used contingency tables and multivariate logistic regression models to compare the manual and automated outputs. Our results indicated low levels of facial recognition by FaceReader in naturalistic interactions (approximately 25.17% compared to manual coding), and we discuss potential causes for this (e.g., problems with lighting, the headcams themselves, and speed of infant movement). However, our logistic regression models showed that when the face was found, FaceReader predicted manually coded expressions with a mean accuracy of M = 0.84 (range = 0.67–0.94), sensitivity of M = 0.64 (range = 0.27–0.97), and specificity of M = 0.81 (range = 0.51–0.97).",

keywords = "ALSPAC, Automated facial coding, FaceReader",

author = "Romana Burgess and Iryna Culpin and Helen Bould and Rebecca Pearson and Ian Nabney",

note = "Funding Information: We are extremely grateful to all the families who took part in the ALSPAC study, the midwives for their help in recruiting them, and the whole ALSPAC team, which includes interviewers, computer and laboratory technicians, clerical workers, research scientists, volunteers, managers, receptionists, and nurses. The UK Medical Research Council and Wellcome (Grant ref: 217065/Z/19/Z) and the University of Bristol provide core support for ALSPAC. We would also like to thank the two double coders who were involved in this work, Lottie Relph and Maddy Stephens. This publication is the work of the authors RB, IC, HB, and RP, who will serve as guarantors for the contents of this paper. A comprehensive list of grants funding is available on the ALSPAC website (http://www.bristol.ac.uk/alspac/external/documents/grant-acknowledgements.pdf); This work is part of a project that has received funding from the European Research Council (ERC) under the European Union{\textquoteright}s Horizon 2020 research and innovation programme (Grant agreement No. 758813; MHINT). Additionally, RB was supported by the Engineering and Physical Sciences Research Council (EPSRC) Digital Health and Care Centre for Doctoral Training (CDT) at the University of Bristol (UKRI Grant No. EP/S023704/1). IC was supported by the Wellcome Trust Research Fellowship in Humanities and Social Science (Grant ref: 212664/Z/18/Z). Publisher Copyright: {\textcopyright} 2023, ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.; 16th EAI International Conference on Pervasive Computing Technologies for Healthcare, PH 2022 ; Conference date: 12-12-2022 Through 14-12-2022",

year = "2023",

doi = "10.1007/978-3-031-34586-9_25",

language = "English",

isbn = "978-3-031-34585-2",

volume = "48",

series = "Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST",

publisher = "Springer Nature",

pages = "379--396",

editor = "Athanasios Tsanas and Andreas Triantafyllidis",

booktitle = "Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings",

address = "United Kingdom",

}

Burgess, R, Culpin, I, Bould, H , Pearson, R & Nabney, I 2023, A quantitative comparison of manual vs. automated facial coding using real life observations of fathers. in A Tsanas & A Triantafyllidis (eds), Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings: - 16th EAI International Conference, PervasiveHealth 2022, Proceedings. vol. 48, Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, vol. 488 LNICST, Springer Nature, pp. 379-396, 16th EAI International Conference on Pervasive Computing Technologies for Healthcare, PH 2022, Thessaloniki, Greece, 12/12/22. https://doi.org/10.1007/978-3-031-34586-9_25

A quantitative comparison of manual vs. automated facial coding using real life observations of fathers. / Burgess, Romana; Culpin, Iryna; Bould, Helen et al.
Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings: - 16th EAI International Conference, PervasiveHealth 2022, Proceedings. ed. / Athanasios Tsanas; Andreas Triantafyllidis. Vol. 48 Springer Nature, 2023. p. 379-396 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 488 LNICST).

Research output: Chapter in Book/Report/Conference proceeding › Conference Contribution (Conference Proceeding)

TY - GEN

T1 - A quantitative comparison of manual vs. automated facial coding using real life observations of fathers

AU - Burgess, Romana

AU - Culpin, Iryna

AU - Bould, Helen

AU - Pearson, Rebecca

AU - Nabney, Ian

N1 - Funding Information: We are extremely grateful to all the families who took part in the ALSPAC study, the midwives for their help in recruiting them, and the whole ALSPAC team, which includes interviewers, computer and laboratory technicians, clerical workers, research scientists, volunteers, managers, receptionists, and nurses. The UK Medical Research Council and Wellcome (Grant ref: 217065/Z/19/Z) and the University of Bristol provide core support for ALSPAC. We would also like to thank the two double coders who were involved in this work, Lottie Relph and Maddy Stephens. This publication is the work of the authors RB, IC, HB, and RP, who will serve as guarantors for the contents of this paper. A comprehensive list of grants funding is available on the ALSPAC website (http://www.bristol.ac.uk/alspac/external/documents/grant-acknowledgements.pdf); This work is part of a project that has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 758813; MHINT). Additionally, RB was supported by the Engineering and Physical Sciences Research Council (EPSRC) Digital Health and Care Centre for Doctoral Training (CDT) at the University of Bristol (UKRI Grant No. EP/S023704/1). IC was supported by the Wellcome Trust Research Fellowship in Humanities and Social Science (Grant ref: 212664/Z/18/Z). Publisher Copyright: © 2023, ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.

PY - 2023

Y1 - 2023

N2 - This work explores the application of an automated facial recognition software “FaceReader” [1] to videos of fathers (n = 36), taken using headcams worn by their infants during interactions in the home. We evaluate the use of FaceReader as an alternative method to manual coding – which is both time and labour intensive – and advance understanding of the usability of this software in naturalistic interactions. Using video data taken from the Avon Longitudinal Study of Parents and Children (ALSPAC), we first manually coded fathers’ facial expressions according to an existing coding scheme, and then processed the videos using FaceReader. We used contingency tables and multivariate logistic regression models to compare the manual and automated outputs. Our results indicated low levels of facial recognition by FaceReader in naturalistic interactions (approximately 25.17% compared to manual coding), and we discuss potential causes for this (e.g., problems with lighting, the headcams themselves, and speed of infant movement). However, our logistic regression models showed that when the face was found, FaceReader predicted manually coded expressions with a mean accuracy of M = 0.84 (range = 0.67–0.94), sensitivity of M = 0.64 (range = 0.27–0.97), and specificity of M = 0.81 (range = 0.51–0.97).

AB - This work explores the application of an automated facial recognition software “FaceReader” [1] to videos of fathers (n = 36), taken using headcams worn by their infants during interactions in the home. We evaluate the use of FaceReader as an alternative method to manual coding – which is both time and labour intensive – and advance understanding of the usability of this software in naturalistic interactions. Using video data taken from the Avon Longitudinal Study of Parents and Children (ALSPAC), we first manually coded fathers’ facial expressions according to an existing coding scheme, and then processed the videos using FaceReader. We used contingency tables and multivariate logistic regression models to compare the manual and automated outputs. Our results indicated low levels of facial recognition by FaceReader in naturalistic interactions (approximately 25.17% compared to manual coding), and we discuss potential causes for this (e.g., problems with lighting, the headcams themselves, and speed of infant movement). However, our logistic regression models showed that when the face was found, FaceReader predicted manually coded expressions with a mean accuracy of M = 0.84 (range = 0.67–0.94), sensitivity of M = 0.64 (range = 0.27–0.97), and specificity of M = 0.81 (range = 0.51–0.97).

KW - ALSPAC

KW - Automated facial coding

KW - FaceReader

UR - http://www.scopus.com/inward/record.url?scp=85164106811&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-34586-9_25

DO - 10.1007/978-3-031-34586-9_25

M3 - Conference Contribution (Conference Proceeding)

AN - SCOPUS:85164106811

SN - 978-3-031-34585-2

VL - 48

T3 - Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST

SP - 379

EP - 396

BT - Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings

A2 - Tsanas, Athanasios

A2 - Triantafyllidis, Andreas

PB - Springer Nature

T2 - 16th EAI International Conference on Pervasive Computing Technologies for Healthcare, PH 2022

Y2 - 12 December 2022 through 14 December 2022

ER -

Burgess R, Culpin I, Bould H , Pearson R , Nabney I. A quantitative comparison of manual vs. automated facial coding using real life observations of fathers. In Tsanas A, Triantafyllidis A, editors, Pervasive Computing Technologies for Healthcare - 16th EAI International Conference, PervasiveHealth 2022, Proceedings: - 16th EAI International Conference, PervasiveHealth 2022, Proceedings. Vol. 48. Springer Nature. 2023. p. 379-396. (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST). doi: 10.1007/978-3-031-34586-9_25

A quantitative comparison of manual vs. automated facial coding using real life observations of fathers

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this