Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication

Charles R. Ebersole; Olivia E. Atherton; Aimee L. Belanger; Hayley M. Skulborstad; Jill M. Allen; Jonathan Banks; Erica Baranski; Michael J. Bernstein; Diane B. V. Bonfiglio; Leanne Boucher; Elizabeth R. Brown; Nancy I. Budiman; Athena H. Cairo; Colin A. Capaldi; Christopher R. Chartier; Joanne M. Chung; David C. Cicero; Jennifer A. Coleman; Brian A. Nosek; John G. Conway; William E. Davis; Thierry Devos; Melody M. Fletcher; Komi German; Jon E. Grahe; Anthony D. Hermann; Joshua A. Hicks; Nathan Honeycutt; Brandon Humphrey; Matthew Janus; David J. Johnson; Jennifer A. Joy-Gaba; Hannah Juzeler; Ashley Keres; Diana Kinney; Jacqueline Kirshenbaum; Richard A. Klein; Richard E. Lucas; Christopher J.N. Lustgraaf; Daniel Martin; Madhavi Menon; Mitchell Metzger; Jaclyn M. Moloney; Patrick J. Morse; Radmila Prislin; Timothy Razza; Daniel E. Re; Nicholas O. Rule; Donald F. Sacco; Kyle Sauerberger; Emily Shrider; Megan Shultz; Courtney Siemsen; Karin Sobocko; Amy Summerville; Konstantin O. Tskhay; Zack van Allen; Leigh Ann Vaughn; Ryan J. Walker; Ashley Weinberg; John Paul Wilson; James H. Wirth; Jessica Wortman; Weylin Sternglanz

doi:10.1016/j.jesp.2015.10.012

Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication

Charles R. Ebersole
, Olivia E. Atherton
, Aimee L. Belanger
, Hayley M. Skulborstad
, Jill M. Allen
, Jonathan Banks
, Erica Baranski
, Michael J. Bernstein
, Diane B. V. Bonfiglio
, Leanne Boucher
, Elizabeth R. Brown
, Nancy I. Budiman
, Athena H. Cairo
, Colin A. Capaldi
, Christopher R. Chartier
, Joanne M. Chung
, David C. Cicero
, Jennifer A. Coleman
, Brian A. Nosek
, John G. Conway

William E. Davis, Thierry Devos, Melody M. Fletcher, Komi German, Jon E. Grahe, Anthony D. Hermann, Joshua A. Hicks, Nathan Honeycutt, Brandon Humphrey, Matthew Janus, David J. Johnson, Jennifer A. Joy-Gaba, Hannah Juzeler, Ashley Keres, Diana Kinney, Jacqueline Kirshenbaum, Richard A. Klein, Richard E. Lucas, Christopher J.N. Lustgraaf, Daniel Martin, Madhavi Menon, Mitchell Metzger, Jaclyn M. Moloney, Patrick J. Morse, Radmila Prislin, Timothy Razza, Daniel E. Re, Nicholas O. Rule, Donald F. Sacco, Kyle Sauerberger, Emily Shrider, Megan Shultz, Courtney Siemsen, Karin Sobocko, Amy Summerville, Konstantin O. Tskhay, Zack van Allen, Leigh Ann Vaughn, Ryan J. Walker, Ashley Weinberg, John Paul Wilson, James H. Wirth, Jessica Wortman, Weylin Sternglanz

Research output: Contribution to journal › Article › peer-review

Abstract

The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools ( N = 2696) and with an online sample ( N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

Original language	American English
Journal	Journal of Experimental Social Psychology
Volume	67
DOIs	https://doi.org/10.1016/j.jesp.2015.10.012
State	Published - Nov 1 2016

Keywords

academic semester
participant pool quality
replication
Social psychology
Cognitive psychology
Participant pool
Individual differences
Sampling effects
Situational effects

Disciplines

Psychology

Access to Document

10.1016/j.jesp.2015.10.012License: CC BY

https://doi.org/10.1016/j.jesp.2015.10.012

Cite this

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H., Capaldi, C. A., Chartier, C. R., Chung, J. M., Cicero, D. C., Coleman, J. A., Nosek, B. A., ... Sternglanz, W. (2016). Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication. Journal of Experimental Social Psychology, 67. https://doi.org/10.1016/j.jesp.2015.10.012

Ebersole, CR, Atherton, OE, Belanger, AL, Skulborstad, HM, Allen, JM, Banks, J, Baranski, E, Bernstein, MJ, Bonfiglio, DBV, Boucher, L, Brown, ER, Budiman, NI, Cairo, AH, Capaldi, CA, Chartier, CR, Chung, JM, Cicero, DC, Coleman, JA, Nosek, BA, Conway, JG, Davis, WE, Devos, T, Fletcher, MM, German, K, Grahe, JE, Hermann, AD, Hicks, JA, Honeycutt, N, Humphrey, B, Janus, M, Johnson, DJ, Joy-Gaba, JA, Juzeler, H, Keres, A, Kinney, D, Kirshenbaum, J, Klein, RA, Lucas, RE, Lustgraaf, CJN, Martin, D, Menon, M, Metzger, M, Moloney, JM, Morse, PJ, Prislin, R, Razza, T, Re, DE, Rule, NO, Sacco, DF, Sauerberger, K, Shrider, E, Shultz, M, Siemsen, C, Sobocko, K, Summerville, A, Tskhay, KO, van Allen, Z, Vaughn, LA, Walker, RJ, Weinberg, A, Wilson, JP, Wirth, JH, Wortman, J & Sternglanz, W 2016, 'Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication', Journal of Experimental Social Psychology, vol. 67. https://doi.org/10.1016/j.jesp.2015.10.012

@article{8175a16df6184338af6dcc58cf5cf791,

title = "Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication",

abstract = " The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools ( N = 2696) and with an online sample ( N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.",

keywords = "academic semester, participant pool quality, replication, Social psychology, Cognitive psychology, Participant pool, Individual differences, Sampling effects, Situational effects",

author = "Ebersole, \{Charles R.\} and Atherton, \{Olivia E.\} and Belanger, \{Aimee L.\} and Skulborstad, \{Hayley M.\} and Allen, \{Jill M.\} and Jonathan Banks and Erica Baranski and Bernstein, \{Michael J.\} and Bonfiglio, \{Diane B. V.\} and Leanne Boucher and Brown, \{Elizabeth R.\} and Budiman, \{Nancy I.\} and Cairo, \{Athena H.\} and Capaldi, \{Colin A.\} and Chartier, \{Christopher R.\} and Chung, \{Joanne M.\} and Cicero, \{David C.\} and Coleman, \{Jennifer A.\} and Nosek, \{Brian A.\} and Conway, \{John G.\} and Davis, \{William E.\} and Thierry Devos and Fletcher, \{Melody M.\} and Komi German and Grahe, \{Jon E.\} and Hermann, \{Anthony D.\} and Hicks, \{Joshua A.\} and Nathan Honeycutt and Brandon Humphrey and Matthew Janus and Johnson, \{David J.\} and Joy-Gaba, \{Jennifer A.\} and Hannah Juzeler and Ashley Keres and Diana Kinney and Jacqueline Kirshenbaum and Klein, \{Richard A.\} and Lucas, \{Richard E.\} and Lustgraaf, \{Christopher J.N.\} and Daniel Martin and Madhavi Menon and Mitchell Metzger and Moloney, \{Jaclyn M.\} and Morse, \{Patrick J.\} and Radmila Prislin and Timothy Razza and Re, \{Daniel E.\} and Rule, \{Nicholas O.\} and Sacco, \{Donald F.\} and Kyle Sauerberger and Emily Shrider and Megan Shultz and Courtney Siemsen and Karin Sobocko and Amy Summerville and Tskhay, \{Konstantin O.\} and \{van Allen\}, Zack and Vaughn, \{Leigh Ann\} and Walker, \{Ryan J.\} and Ashley Weinberg and Wilson, \{John Paul\} and Wirth, \{James H.\} and Jessica Wortman and Weylin Sternglanz",

year = "2016",

month = nov,

day = "1",

doi = "10.1016/j.jesp.2015.10.012",

language = "American English",

volume = "67",

journal = "Journal of Experimental Social Psychology",

issn = "1096-0465",

}

TY - JOUR

T1 - Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication

AU - Ebersole, Charles R.

AU - Atherton, Olivia E.

AU - Belanger, Aimee L.

AU - Skulborstad, Hayley M.

AU - Allen, Jill M.

AU - Banks, Jonathan

AU - Baranski, Erica

AU - Bernstein, Michael J.

AU - Bonfiglio, Diane B. V.

AU - Boucher, Leanne

AU - Brown, Elizabeth R.

AU - Budiman, Nancy I.

AU - Cairo, Athena H.

AU - Capaldi, Colin A.

AU - Chartier, Christopher R.

AU - Chung, Joanne M.

AU - Cicero, David C.

AU - Coleman, Jennifer A.

AU - Nosek, Brian A.

AU - Conway, John G.

AU - Davis, William E.

AU - Devos, Thierry

AU - Fletcher, Melody M.

AU - German, Komi

AU - Grahe, Jon E.

AU - Hermann, Anthony D.

AU - Hicks, Joshua A.

AU - Honeycutt, Nathan

AU - Humphrey, Brandon

AU - Janus, Matthew

AU - Johnson, David J.

AU - Joy-Gaba, Jennifer A.

AU - Juzeler, Hannah

AU - Keres, Ashley

AU - Kinney, Diana

AU - Kirshenbaum, Jacqueline

AU - Klein, Richard A.

AU - Lucas, Richard E.

AU - Lustgraaf, Christopher J.N.

AU - Martin, Daniel

AU - Menon, Madhavi

AU - Metzger, Mitchell

AU - Moloney, Jaclyn M.

AU - Morse, Patrick J.

AU - Prislin, Radmila

AU - Razza, Timothy

AU - Re, Daniel E.

AU - Rule, Nicholas O.

AU - Sacco, Donald F.

AU - Sauerberger, Kyle

AU - Shrider, Emily

AU - Shultz, Megan

AU - Siemsen, Courtney

AU - Sobocko, Karin

AU - Summerville, Amy

AU - Tskhay, Konstantin O.

AU - van Allen, Zack

AU - Vaughn, Leigh Ann

AU - Walker, Ryan J.

AU - Weinberg, Ashley

AU - Wilson, John Paul

AU - Wirth, James H.

AU - Wortman, Jessica

AU - Sternglanz, Weylin

PY - 2016/11/1

Y1 - 2016/11/1

N2 - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools ( N = 2696) and with an online sample ( N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

AB - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools ( N = 2696) and with an online sample ( N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

KW - academic semester

KW - participant pool quality

KW - replication

KW - Social psychology

KW - Cognitive psychology

KW - Participant pool

KW - Individual differences

KW - Sampling effects

KW - Situational effects

UR - https://nsuworks.nova.edu/cps_facarticles/1660

UR - https://aquila.usm.edu/fac_pubs/17817

UR - https://doi.org/10.1016/j.jesp.2015.10.012

U2 - 10.1016/j.jesp.2015.10.012

DO - 10.1016/j.jesp.2015.10.012

M3 - Article

SN - 1096-0465

VL - 67

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

ER -

Many Labs 3: Evaluating Participant Pool Quality across the Academic Semester via Replication

Abstract

Keywords

Disciplines

Access to Document

Other files and links

Fingerprint

Cite this