Realistic precision and accuracy of online experiment platforms, web browsers, and devices

Alexander Anwyl-Irvine, Edwin S Dalmaijer, Nick Hodges, Jo K Evershed

Research output: Contribution to journalArticle (Academic Journal)peer-review

7 Citations (Scopus)

Abstract

Due to increasing ease of use and ability to quickly collect large samples, online behavioural research is currently booming. With this popularity, it is important that researchers are aware of who online participants are, and what devices and software they use to access experiments. While it is somewhat obvious that these factors can impact data quality, the magnitude of the problem remains unclear. To understand how these characteristics impact experiment presentation and data quality, we performed a battery of automated tests on a number of realistic set-ups. We investigated how different web-building platforms (Gorilla v.20190828, jsPsych v6.0.5, Lab.js v19.1.0, and psychoJS/PsychoPy3 v3.1.5), browsers (Chrome, Edge, Firefox, and Safari), and operating systems (macOS and Windows 10) impact display time across 30 different frame durations for each software combination. We then employed a robot actuator in realistic set-ups to measure response recording across the aforementioned platforms, and between different keyboard types (desktop and integrated laptop). Finally, we analysed data from over 200,000 participants on their demographics, technology, and software to provide context to our findings. We found that modern web platforms provide reasonable accuracy and precision for display duration and manual response time, and that no single platform stands out as the best in all features and conditions. In addition, our online participant analysis shows what equipment they are likely to use.

Original languageEnglish
Pages (from-to)1407-1425
Number of pages19
JournalBehavior Research Methods
Volume53
Issue number4
DOIs
Publication statusPublished - Aug 2021

Bibliographical note

Funding Information:
AAI, NH, and JKE are employed by Cauldron, which operates the Gorilla experiment platform. AAI’s PhD is funded by the Templeton World Charity Foundation (TWCF no. 0159) and the Medical Research Council.

Publisher Copyright:
© 2020, The Author(s).

Keywords

  • Behavioral Research
  • Data Accuracy
  • Humans
  • Internet
  • Reaction Time
  • Software
  • Web Browser

Fingerprint

Dive into the research topics of 'Realistic precision and accuracy of online experiment platforms, web browsers, and devices'. Together they form a unique fingerprint.

Cite this