Resources and analysis for my master's thesis
I did multiple iterations of the study when I was working out kinks in the manipulations. First, I tried the original design of the study on MTurk:
In the first test, the participants did not seem to be responding accurately. We were worried that they were misunderstanding the vignette. I reworked the manipulations a bit to make them very straightforward and tried again:
The updated vignettes produced no change in the accuracy of the data. I became concerned about the possibility that the participants were simply not engaging with the survey, because it seemed highly unlikely that they were reading the short passages and still failing to correctly respond to the manipulation checks.
At this point, I tweaked things a bit more on the survey but also moved it to Cloud Research, which claims to weed out MTurk workers who do not show engagement:
Manipulation Check v0.3 on Cloud Research
The results showed a marked improvement, but still fell short of the accuracy I wanted to see. Hoping to see more improvement in a larger sample, I launched the full survey on Cloud Research. After about 40 responses, accuracy was hovering at around 65% (where pure chance would put it at 50%) on a question that should have had close to 100% accuracy.
I added an additional validation check in between the vignettes and the manipulation checks. In order to move on from the vignettes, participants now must demonstrate that they have read the vignettes and have the most basic understanding. Finally, that seemed to have boosted the accuracy substantially, to about 80%. That is still low, considering that accuracy really should be closer to 100% given how straightforward the manipulation checks are, but it was good enough that I decided to continue with the full data collection.
Here were the results of the manipulation check in the (almost) full sample: