Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis

Scott D. Gronlund; John T. Wixted; Laura Mickes

doi:10.1177/0963721413498891

Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis

Scott D. Gronlund
, John T. Wixted
, Laura Mickes

Bristol Neuroscience

Research output: Contribution to journal › Article (Academic Journal) › peer-review

140 Citations (Scopus)

Abstract

Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio-based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.

Original language	English
Pages (from-to)	3-10
Number of pages	8
Journal	Current Directions in Psychological Science
Volume	23
Issue number	1
DOIs	https://doi.org/10.1177/0963721413498891
Publication status	Published - Feb 2014

Bibliographical note

Funding Information:
Gronlund, S. D., Andersen, S. M., & Perry, C. (2013) . (See References). Review of the debate surrounding simultaneous and sequential lineups. Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A., Wetmore, S. A., Wooten, A., & Graham, M. (2012) . (See References). Receiver operating characteristic analysis evaluated performance from simultaneous and sequential lineups and showups. Mickes, L., Flowe, H. D., & Wixted, J. T. (2012) . (See References). Receiver operating characteristic analysis showed no evidence for sequential advantage. Wixted, J. T., & Mickes, L. (2012). The field of eyewitness memory should abandon “probative value” and embrace receiver operating characteristic analysis. Perspectives on Psychological Science, 7 , 275–278. Illustrates the limitations of ratio-based probative value measures and the advantages of receiver-operating-characteristic-based analysis of simultaneous and sequential lineup performance. Declaration of Conflicting Interests The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article. Funding This article was supported by National Science Foundation (NSF) Grants SES-1060902 to Scott D. Gronlund and SES-1155248 to John T. Wixted and Laura Mickes. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the NSF. Supplemental Material Additional supporting information, including a tutorial on ROC analysis, may be found at http://cdp.sagepub.com/content/by/supplemental-data 1. One could simply compute d ′ from Lindsay and Wells’s (1985) data as a proxy for a full receiver operating characteristic (ROC) analysis; it shows a sequential advantage (1.58 vs. 0.61). Had it been performed, ROC analysis may have shown a sequential advantage as well. However, the only way to be certain about that is to actually perform ROC analysis, which is superior to computing d ′ because it is an assumption-free method. By contrast, d ′ is based on a specific model that applies to a standard single-item old/new recognition procedure, not to a lineup recognition procedure; d ′ also makes assumptions regarding normally distributed memory strength values. Although d ′ is almost certainly more informative than the more typically used diagnosticity ratio, it is less informative than ROC analysis. 2. There are several measures of probative value of a suspect ID that utilize the ratio of correct and false IDs in various configurations. All of these measures are subject to the criticism we level later in the article.

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 16 Peace, Justice and Strong Institutions

Keywords

diagnosticity ratio
eyewitness identification
probative value
receiver operating characteristic (ROC) analysis
simultaneous and sequential lineups

Access to Document

10.1177/0963721413498891

Handle.net

Persistent link

Cite this

@article{654e9c0ffd5647479cd3efba0749a320,

title = "Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis",

abstract = "Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio-based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.",

keywords = "diagnosticity ratio, eyewitness identification, probative value, receiver operating characteristic (ROC) analysis, simultaneous and sequential lineups",

author = "Gronlund, \{Scott D.\} and Wixted, \{John T.\} and Laura Mickes",

note = "Funding Information: Gronlund, S. D., Andersen, S. M., \& Perry, C. (2013) . (See References). Review of the debate surrounding simultaneous and sequential lineups. Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A., Wetmore, S. A., Wooten, A., \& Graham, M. (2012) . (See References). Receiver operating characteristic analysis evaluated performance from simultaneous and sequential lineups and showups. Mickes, L., Flowe, H. D., \& Wixted, J. T. (2012) . (See References). Receiver operating characteristic analysis showed no evidence for sequential advantage. Wixted, J. T., \& Mickes, L. (2012). The field of eyewitness memory should abandon “probative value” and embrace receiver operating characteristic analysis. Perspectives on Psychological Science, 7 , 275–278. Illustrates the limitations of ratio-based probative value measures and the advantages of receiver-operating-characteristic-based analysis of simultaneous and sequential lineup performance. Declaration of Conflicting Interests The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article. Funding This article was supported by National Science Foundation (NSF) Grants SES-1060902 to Scott D. Gronlund and SES-1155248 to John T. Wixted and Laura Mickes. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the NSF. Supplemental Material Additional supporting information, including a tutorial on ROC analysis, may be found at http://cdp.sagepub.com/content/by/supplemental-data 1. One could simply compute d ′ from Lindsay and Wells{\textquoteright}s (1985) data as a proxy for a full receiver operating characteristic (ROC) analysis; it shows a sequential advantage (1.58 vs. 0.61). Had it been performed, ROC analysis may have shown a sequential advantage as well. However, the only way to be certain about that is to actually perform ROC analysis, which is superior to computing d ′ because it is an assumption-free method. By contrast, d ′ is based on a specific model that applies to a standard single-item old/new recognition procedure, not to a lineup recognition procedure; d ′ also makes assumptions regarding normally distributed memory strength values. Although d ′ is almost certainly more informative than the more typically used diagnosticity ratio, it is less informative than ROC analysis. 2. There are several measures of probative value of a suspect ID that utilize the ratio of correct and false IDs in various configurations. All of these measures are subject to the criticism we level later in the article. ",

year = "2014",

month = feb,

doi = "10.1177/0963721413498891",

language = "English",

volume = "23",

pages = "3--10",

journal = "Current Directions in Psychological Science",

issn = "0963-7214",

publisher = "SAGE Publications Inc.",

number = "1",

}

TY - JOUR

T1 - Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis

AU - Gronlund, Scott D.

AU - Wixted, John T.

AU - Mickes, Laura

N1 - Funding Information: Gronlund, S. D., Andersen, S. M., & Perry, C. (2013) . (See References). Review of the debate surrounding simultaneous and sequential lineups. Gronlund, S. D., Carlson, C. A., Neuschatz, J. S., Goodsell, C. A., Wetmore, S. A., Wooten, A., & Graham, M. (2012) . (See References). Receiver operating characteristic analysis evaluated performance from simultaneous and sequential lineups and showups. Mickes, L., Flowe, H. D., & Wixted, J. T. (2012) . (See References). Receiver operating characteristic analysis showed no evidence for sequential advantage. Wixted, J. T., & Mickes, L. (2012). The field of eyewitness memory should abandon “probative value” and embrace receiver operating characteristic analysis. Perspectives on Psychological Science, 7 , 275–278. Illustrates the limitations of ratio-based probative value measures and the advantages of receiver-operating-characteristic-based analysis of simultaneous and sequential lineup performance. Declaration of Conflicting Interests The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article. Funding This article was supported by National Science Foundation (NSF) Grants SES-1060902 to Scott D. Gronlund and SES-1155248 to John T. Wixted and Laura Mickes. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the NSF. Supplemental Material Additional supporting information, including a tutorial on ROC analysis, may be found at http://cdp.sagepub.com/content/by/supplemental-data 1. One could simply compute d ′ from Lindsay and Wells’s (1985) data as a proxy for a full receiver operating characteristic (ROC) analysis; it shows a sequential advantage (1.58 vs. 0.61). Had it been performed, ROC analysis may have shown a sequential advantage as well. However, the only way to be certain about that is to actually perform ROC analysis, which is superior to computing d ′ because it is an assumption-free method. By contrast, d ′ is based on a specific model that applies to a standard single-item old/new recognition procedure, not to a lineup recognition procedure; d ′ also makes assumptions regarding normally distributed memory strength values. Although d ′ is almost certainly more informative than the more typically used diagnosticity ratio, it is less informative than ROC analysis. 2. There are several measures of probative value of a suspect ID that utilize the ratio of correct and false IDs in various configurations. All of these measures are subject to the criticism we level later in the article.

PY - 2014/2

Y1 - 2014/2

N2 - Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio-based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.

AB - Eyewitness identification is a pivotal issue in applied research because, in practice, a correct identification can help to remove a dangerous criminal from society, but a false identification can lead to the erroneous conviction of an innocent suspect. Consequently, psychologists have tried to ascertain the best procedures for collecting identification evidence, evaluating them using measures based on the ratio of correct to false identification rates. Unfortunately, ratio-based measures are ambiguous because they change systematically as a function of a witness's willingness to choose. In other words, a measure thought to index discriminability is instead fully confounded with response bias. A better method involves constructing receiver operating characteristic (ROC) curves. Using ROC curves, researchers can trace out discriminability across levels of response bias for each procedure. We illustrate the shortcomings of ratio-based measures and demonstrate why ROC analysis is required. In recent studies, researchers comparing simultaneous and sequential lineup procedures using ROC analyses have provided no evidence for the sequential superiority effect and instead have shown that the simultaneous procedure may be diagnostically superior. It is not yet clear which lineup procedure will prove to be generally superior, but it is clear that ROC analysis is the only way to make that determination.

KW - diagnosticity ratio

KW - eyewitness identification

KW - probative value

KW - receiver operating characteristic (ROC) analysis

KW - simultaneous and sequential lineups

UR - https://www.scopus.com/pages/publications/84893601680

U2 - 10.1177/0963721413498891

DO - 10.1177/0963721413498891

M3 - Article (Academic Journal)

AN - SCOPUS:84893601680

SN - 0963-7214

VL - 23

SP - 3

EP - 10

JO - Current Directions in Psychological Science

JF - Current Directions in Psychological Science

IS - 1

ER -

Evaluating Eyewitness Identification Procedures Using Receiver Operating Characteristic Analysis

Abstract

Bibliographical note

UN SDGs

Keywords

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this