CloudResearch
CloudResearch is a technology company based in Queens, NY that facilitates online behavioral science research (e.g., surveys, polling, behavioral experiments, market research, etc.). The products CloudResearch creates are used by researchers at top universities worldwide, Fortune 500 companies, government agencies, and nonprofit organizations to conduct credible, high-quality research that drives academic science, business decisions, and policy.
CloudResearch employs a team of social scientists and software engineers who work to make online participant recruitment, the protection of online data, and the execution of complex online projects more efficient and more trustworthy.
History
Started in 2015 as TurkPrime, CloudResearch began by helping academic researchers make Amazon’s Mechanical Turk into a useful platform for behavioral science research[1]. Because of CloudResearch’s academic roots—both founders were researchers and professors—TurkPrime developed tools and sampling solutions that academic researchers needed and that facilitated further adoption of Mechanical Turk[2][3]. As a result, TurkPrime quickly developed a niche within the academic community.
According to an article published in Behavior Research Methods, from 2016 to 2022 “about 10,000 researchers have used CloudResearch to conduct over 340,000 MTurk studies with approximately 500,000 unique participants and over 50 million completed assignments.”[4] This level of activity is thanks largely to the popularity of MTurk among research psychologists, political scientists, sociologists, and other behavioral researchers. However, the MTurk Toolkit continues to be used widely by all kinds of researchers in and outside of academia to manage projects on Mechanical Turk today, including ones that touch on timely political and social issues[5][6][7][8].
Over the years, CloudResearch expanded to provide research participants from a variety of online sources. In 2017, for example, the company launched Prime Panels. Prime Panels allowed researchers to tap into nonprobability panels (also sometimes called market research panels) that are traditionally used for industry and market research, as well as many nonprobability political polls[9].
The innovation CloudResearch brought to this space was an approach to vetting survey respondents and elevating data quality. Several peer reviewed publications have identified data quality as a critical issue within online panels[10][11], with the Pew Research Center going so far as to label some of this data “bogus.”[12] CloudResearch developed a tool called Sentry to improve data quality by finding people committing survey fraud, removing inattentive participants, and ensuring that the people who take a survey are actually the people they claim to be.
With the introduction of Prime Panels and Sentry, CloudResearch began to develop a reputation for data quality among the many market research firms, nonprofits, and government agencies in need of high quality data. For example, a 2021 article in the Harvard Business Review made clear the difference between the status quo in online research and what a serious approach to data quality can do, like the one advanced by CloudResearch[13].
In 2022, CloudResearch launched its own participant platform called Connect. Built by CloudResearch, Connect is completely separate from the previous sources of participants offered by CloudResearch. The company has described Connect as an attempt to learn from years of experience with other online platforms and to do better. The better in this sentence is understood as doing better for researchers by improving the research experience with better data quality and doing better for participants by improving the experience of participating in studies.
Products and Services
CloudResearch has four products: Connect, Prime Panels, the Mechanical Turk Toolkit, and Sentry. CloudResearch also offers to conduct research studies for a fee through its managed research service.
Connect
Connect is a participant recruitment platform that CloudResearch built and manages. Released in 2022, CloudResearch has described Connect as “an attempt to learn from years of experience with other online platforms and to provide a better research experience.”
On the CloudResearch website Connect is described as a platform “that puts data quality at the forefront while giving researchers the tools to make their online studies better.” The advantages of Connect are generally understood to be its low service fees, high data quality, and unique tools for researcher collaboration.
Prime Panels
Prime Panels began in 2017. As a tool that taps into market research panels, Prime Panels offers access to over 100 million research participants worldwide. However, what separates CloudResearch’s Prime Panels from the several other sources of market research participants is CloudResearch’s recognition that participants from these panels need to be vetted and that data quality has to be actively managed.
To do that, CloudResearch uses a patented pre-survey vetting system called Sentry. Sentry quickly vets participants and only allows those who show evidence of quality responses to continue the survey. Fraudulent, inattentive, or otherwise undesirable participants are blocked from the study. The principles behind CloudResearch’s Sentry system have been accepted in multiple peer-reviewed publications.
Mechanical Turk Toolkit
CloudResearch’s MTurk Toolkit allows researchers to sample from Amazon Mechanical Turk (MTurk), while using the tools and data quality controls that CloudResearch has built over the years. According to SAGE, the most prestigious academic publisher on research methods, “CloudResearch provides enhanced controls in seven general areas: tools for communicating with workers such as chat, audio, and video portals for live interviewing; tools for running longitudinal studies; control over who participates in a study; control over launching and running HITs; batched functionality for communication; payment tools for enhanced sampling; and advanced dashboard indicators.” For this reason, “CloudResearch has been used in numerous scientific studies (e.g., Goodman & Paolacci, 2017; Stewart, Chandler, & Paolacci, 2017).”[14]
Perhaps the most notable control is the CloudResearch Approved Participants. CloudResearch Approved Particiaptns are a subset of more than 100,000 workers from the MTurk population. CloudResearch has vetted each and every one of the participants in the approved group to ensure data quality. Several publications from CloudResearch and independent researchers attest to the quality of this solution.
Sentry
Sentry is CloudResearch’s patented data quality solution. Although many platforms offer tools to block duplicate IPs or use reCAPTCHA’s, Sentry is the gold-standard for ensuring high data quality using behavioral measures in addition to technological detection. Sentry works as a pre-survey instrument that puts participants through a series of attention checks in under 30 seconds, using a large database of validated questions. It can also measure whether participants are copy-pasting, going off screen, or translating the questions into another language. Sentry can be used with any sample source and is automatically applied to Prime Panels studies as a first line of protection against low-effort or fraudulent responses.
Managed Research
Managed research is a service CloudResearch offers to help researchers and organizations gather data for a project.
Managed Research is sometimes used to help researchers more efficiently gather data from participants that are “hard-to-reach" and therefore would not be efficient using a do-it-yourself approach. At other times, Managed Research is used for complicated online projects that would be too difficult or time consuming for teams to manage themselves. As an example of this type of project, CloudResearch was selected to help execute a massively intense longitudinal project in cooperation with The Office of the Director of National Intelligence[15]. In the project, more than 1,200 participants completed a weekly, 2-3 hour task for 35 consecutive weeks.
Uses
CloudResearch enables all kinds of online behavioral research and human subjects data collection. For example, some of the ways researchers have used CloudResearch include:
- Conducting hundreds of thousands of surveys, correlational studies, and experiments
- Performing user experience, or UX, research
- Running market research and brand tracking studies
- Gathering training data for machine learning
- Administering online video interviews
- Recruiting rare populations for in person studies
- Running mock juror tasks
- Performing language translation tasks
- Executing data tagging tasks
- Conducting real-time interactive tasks and competitions among participants
- Administering complex forecasting and longitudinal studies
References
- ↑ Litman, Leib; Robinson, Jonathan; Abberbock, Tzvi (2017-04-01). "TurkPrime.com: A versatile crowdsourcing data acquisition platform for the behavioral sciences". Behavior Research Methods. 49 (2): 433–442. doi:10.3758/s13428-016-0727-z. ISSN 1554-3528. PMC 5405057. PMID 27071389.
- ↑ Sassenberg, Kai; Ditrich, Lara (June 2019). "Research in Social Psychology Changed Between 2011 and 2016: Larger Sample Sizes, More Self-Report Measures, and More Online Studies". Advances in Methods and Practices in Psychological Science. 2 (2): 107–114. doi:10.1177/2515245919838781. ISSN 2515-2459. Unknown parameter
|s2cid=ignored (help) - ↑ Anderson, Craig A.; Allen, Johnie J.; Plante, Courtney; Quigley-McBride, Adele; Lovett, Alison; Rokkum, Jeffrey N. (June 2019). "The MTurkification of Social and Personality Psychology". Personality and Social Psychology Bulletin. 45 (6): 842–850. doi:10.1177/0146167218798821. ISSN 0146-1672. PMID 30317918. Unknown parameter
|s2cid=ignored (help) - ↑ Hauser, David J.; Moss, Aaron J.; Rosenzweig, Cheskie; Jaffe, Shalom N.; Robinson, Jonathan; Litman, Leib (2022-11-03). "Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk". Behavior Research Methods. doi:10.3758/s13428-022-01999-x. ISSN 1554-3528. PMID 36326997 Check
|pmid=value (help). - ↑ Sanchez, Patricia Y. (2022-06-27). "Whiteness is still associated with leadership in the United States, study suggests". PsyPost. Retrieved 2023-02-16.
- ↑ Jackson, Emily B.; Torres-Beltran, Angie (August 16, 2022). "What does the Kansas vote tell us about the future of abortion?". The Washington Post. Retrieved 16 February 2023.
- ↑ "Researchers determine public attitudes towards monkeypox infections". News-Medical.net. 2022-06-27. Retrieved 2023-02-16.
- ↑ Paridon, Bradley van (2021-04-29). "COVID-19 Death Rates and Risk Perception Led to More Time Spent At Home". Infectious Disease Advisor. Retrieved 2023-02-16.
- ↑ Baker, Reg; Brick, J. Michael; Bates, Nancy A.; Battaglia, Mike; Couper, Mick P.; Dever, Jill A.; Gile, Krista J.; Tourangeau, Roger (26 September 2013). "Summary Report of the AAPOR Task Force on Non-probability Sampling". Journal of Survey Statistics and Methodology. 1 (2): 90–143. doi:10.1093/jssam/smt008.
- ↑ Kees, Jeremy; Berry, Christopher; Burton, Scot; Sheehan, Kim (2017-01-02). "An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk". Journal of Advertising. 46 (1): 141–155. doi:10.1080/00913367.2016.1269304. ISSN 0091-3367. Unknown parameter
|s2cid=ignored (help) - ↑ Berry, Christopher; Kees, Jeremy; Burton, Scot (2022-08-08). "Drivers of Data Quality in Advertising Research: Differences across MTurk and Professional Panel Samples". Journal of Advertising. 51 (4): 515–529. doi:10.1080/00913367.2022.2079026. ISSN 0091-3367. Unknown parameter
|s2cid=ignored (help) - ↑ Mitchell, Travis (2020-02-18). "Assessing the Risks to Online Polls From Bogus Respondents". Pew Research Center Methods. Retrieved 2023-02-16.
- ↑ Hartman, Rachel (2021-04-20). "Did 4% of Americans Really Drink Bleach Last Year?". Harvard Business Review. ISSN 0017-8012. Retrieved 2023-02-16.
- ↑ Litman, Leib; Robinson, Jonathan (2021). Conducting Online Research on Amazon Mechanical Turk and Beyond. Thousand Oaks: CA: SAGE. ISBN 9781506391151. Search this book on
- ↑ PhD, Aaron Moss (2022-03-04). "How CloudResearch and IARPA Completed the Largest Longitudinal Online Research Project Ever". CloudResearch. Retrieved 2023-02-16.
This article "CloudResearch" is from Wikipedia. The list of its authors can be seen in its historical and/or the page Edithistory:CloudResearch. Articles copied from Draft Namespace on Wikipedia could be seen on the Draft Namespace of Wikipedia and not main one.
