Mechanical Turk (mTurk) is a great resource for gathering a large sample in a short period of time; however, there is a real concern that participants may end up participating in multiple related studies. This makes running a line of research on mTurk problematic. Luckily, Qualtrics (along with a little magic of my own) offers a way to screen participants so that the problem of duplicate participants can be mitigated.
Note: I have now set up a better way to screen participants, using TurkGate. These instructions are here for those who have existing studies using this method, but I recommend that you switch over to TurkGate for new lines of research.
The basic process is to pass the mTurk worker ID into your Qualtrics survey, check this ID from a list of IDs for people who have participated in previous surveys, and then sending workers whose IDs match an existing ID to a disqualification page. Just follow the steps below to set up this process.
-
The first step is to download this zip file, unzip it, and move the “exclude.php” file somewhere onto your artsresearch account. You will probably find it easiest to put it in its own folder, because you will be adding more files that will be your list of worker IDs.
-
In the same folder as your “exclude.php” file, add as many files as you wish (in .txt or .csv format) with lists of the worker IDs you wish to exclude. There is a sample file in the zip file above. It is up to you how to organize these sets of IDs, but you might want to group IDs based on who has completed a particular task, manipulation, or deception. Make a separate file for each group (they can be called whatever you like). Each file should list mTurk IDs on a separate line. If you want to keep track of other information associated with these IDs (e.g., date they completed a study, what study they completed), you can add this information after each ID, and separate it using spaces, tabs, or commas. The only requirement is that the worker ID must come first on the line.
-
Next, add a script to your mTurk HIT that will grab each worker’s ID and append it to your Qualtrics study link. Insert the script below into the HTML code of your HIT and change the highlighted URL to your Qualtrics study URL (thanks to Peer, Paolacci, Chandler, & Mueller, 2012 for the initial version of this script):
<span id="myelementLink">URL not shown because there is an error with Javascript on your computer. To perform this HIT, you must have Javascript and cookies enabled on your browser.</span> <script type="text/javascript" language="javascript"><!-- var studyUrl = 'http://psychologyuwaterloo.qualtrics.com/SE/?SID=YOUR_SURVEY'; var assignment_id_field = document.getElementById('myelementLink'); var paramstr = window.location.search.substring(1); var parampairs = paramstr.split("&"); var mturkworkerID = null; for (var i = 0; i < parampairs.length; i++) { var pair = parampairs[i].split("="); if (pair[0] == "workerId") { mturkworkerID = pair[1]; break; } } if (mturkworkerID === null) { assignment_id_field.innerHTML = 'The link will appear here only if you accept this HIT.'; } else { assignment_id_field.innerHTML = '<a target="_blank" href="'+studyUrl+'&mid='+mturkworkerID+'"><span style="color: red; font-weight: bold">Click here to take the survey</span></a>'; } // --></script>Be sure that the formatting of the code (i.e., line breaks) does not change—some people have run into problems when the line breaks were removed for some reason.
Please note that after you paste the script in the HTML code of the HIT and view it in the Design Layout mode, it will show this error message: “URL not shown because there is an error with Javascript on your computer. To perform this HIT, you must have Javascript and cookies enabled on your browser.” This is normal. Once you move to the Preview and Finish mode, this text should change to: “The link will appear here only if you accept this HIT.” This means that the script works and workers will be able to use your link after they accept the HIT.
Note: If you use Crowdflower as an intermediary instead of going directly through mTurk, please see below for an alternate method to capture participant IDs.
-
In your Qualtrics study, click on “Survey Flow”, then click the option to “Add a New Element Here”. Add an Embedded Data element from the list of elements, and name it “mid”. You do not need to set a value, as it will be set through the URL. Click and drag this element to the very top of the study using the “Move” button.
-
OPTIONAL: If you wish, you can add a failsafe to catch participants whose worker ID was not passed along for whatever reason. (I have run several studies using this script and never had a case where this happened.) Add a block with a text entry item asking them to confirm their worker ID. In the options on the right, you may set this item to “Force Response”. Using the purple gear button to the left of the question, you can set a default response option: Set this to
${e://Field/mid}.
Go back into Survey Flow and add a Branch to check if the “mid” field is empty. Move the block with your screener question underneath this branch. Below this block, add another Embedded Data field, also called “mid”, and set the value by piping in the response from your text field. The code will depend on what question number your text field is, but it should be something similar to this:${q://QID2/ChoiceTextEntryValue}.
-
Below your Embedded Data element and your (optional) screener question, add a new “Web Service” element. Insert the URL to your “exclude.php” here. Directly after the URL, you will need to feed two pieces of information through the URL: The “mid” data field and the names of the files that you want to check for matching IDs. Here are a couple examples:
If I had one file with all worker IDs I wanted to screen for, I could write my URL as follows:
https://artsresearch.uwaterloo.ca/~username/mturk_exclude/exclude.php?id=${e://Field/mid}&file=mturk_ids.txtThis URL would check the file “mturk_ids.txt” for the particular worker ID passed to it through the “mid” field.
If I wanted to check worker IDs from more than one file (for instance, I might have a list for participants who have previously completed a self-affirmation task and a list for participants who have completed a mood induction task, and this new study uses both tasks), I could list both filenames, separated by commas:
https://artsresearch.uwaterloo.ca/~username/mturk_exclude/exclude.php?id=${e://Field/mid}&file=self_affirmation_ids.csv,mood_induction_ids.csvThis URL would check both files to see whether the particular worker ID shows up in either list.
After inserting your URL, click “Add Embedded Data” and set “match” to equal “match”.
-
The Web Service element above will set an embedded data field called “match” that will equal one of three things: “true”, “false”, or “error”. If it equals “true”, the script has found a match between the current worker ID and the list of IDs, which means you want to disqualify this participant. If it equals “false”, no match was found and the participant should be able to continue on. If it equals “error”, one of a variety of things may have happened, but you probably want to allow the participant to proceed and sort it out later.
The last step, then, is to send the matching participants to a disqualification page. Below the Web Service element, created a Branch that checks if “match” is equal to “true”. Then, underneath that, add an “End of Survey” element. You can customize this element to display an end of survey message explaining that the participant has been disqualified.
The process above should result in a survey flow design that looks something like this:
The “exclude.php” script can also be used to exclude participants on other criteria that you may have available. As long as these criteria can be fed into Qualtrics somehow, and the disqualifying data are placed at the beginning of the file(s) just like the worker IDs would be, it will work just fine. Email me (Jeff Hughes) if you would like to set something like this up, or have questions about the process outlined above.
Using Crowdflower as an Intermediary
For those using Crowdflower as an intermediary, you will not be able to capture participants’ mTurk worker IDs. However, you are able to get their Crowdflower ID, which remains consistent between jobs. In order to capture their Crowdflower ID, append the following code to the end of the study URL that you pass in through the Excel spreadsheet: &mid=. For example: http://psychologyuwaterloo.qualtrics.com/SE/?SID=123456789&mid=. Then, when you insert your URL into the study description (under the “data elements” section), you need to put this directly after the URL: {{_worker_id}}.
Of course, with Crowdflower, nothing could ever be that easy, so you will end up running into issues because they put a space in front of data elements for no reason as soon as you save it. To avoid this, set up everything about your study as usual, save it, then click “Edit in CML Editor”. Delete any spaces in between your URL data element and the {{_worker_id}} element. If you save it at this point, it will work properly. But you cannot save the study in the graphical editor after this point, or it will break again. (Yeah, I know, it’s a pain in the ass. Believe me, I know.) To avoid this, you can insert this code in any HTML tag in the editor: only-if="". This is just a little hack the Crowdflower support let me know about that will open up the CML editor by default for that study.
You should also ignore step 5 above regarding the “failsafe”, since people are not going to know their Crowdflower ID. Otherwise, just follow the instructions above regarding Qualtrics.
