Mechanical Turk (mTurk) is a great resource for gathering a large sample in a short period of time; however, there is a real concern that participants may end up participating in multiple related studies. This makes running a line of research on mTurk problematic. Luckily, TurkGate offers a way to screen participants so that the problem of duplicate participants can be mitigated.
TurkGate operates as an intermediary between mTurk and your survey. When a participant accepts your HIT and clicks the survey link, they are first sent to TurkGate to determine whether they have completed any related studies that would make them ineligible for this study. If they are eligible, they are passed onto the survey; if they are not, they are not given access.
One nice feature of TurkGate is that participants can check whether they are eligible to complete the survey before accepting the HIT. TurkGate includes a “check eligibility” link that participants can click so they can find out if they will be able to complete the survey. This means that they don’t have to accept the HIT, find out they are not eligible, and then return the HIT (which impacts their worker ratings).
Using TurkGate
Using TurkGate is very simple. You just need your survey URL and access to mTurk. I have set up an instance of TurkGate that you can use. Clicking that link will bring you to the main TurkGate page to set up your study.
Input your survey URL into the first textbox, and then create a group name for your group of surveys. If this is the first survey in the group, you can name the group anything you like. (Note: Participants will be able to see the group name, so don’t name it anything that blows your cover story.) If you have previous related studies for which you have used TurkGate, clicking on the “Group Name” textbox will bring up a list of previously created groups. You can select the appropriate group. Note that multiple people can use the same instance of TurkGate. This means that if you and another colleague do related research, you can group your surveys together as well!
The rest of the settings on the page can be kept as-is (you want to use the web interface unless you do more complicated things with mTurk), so just click the button to generate the HIT. This will produce some code on the right side of the page. Highlight and copy this, and then move over to your mTurk account.
When you create your project on mTurk, one of the tabs says “Design Layout”. This is where the description of the study appears. The easiest thing to do is to insert your study description like you normally would into that box—but don’ include your survey URL, since the code TurkGate gave you will insert that itself. When your study description looks the way you want it to, then click the “Source” button at the top of the big textbox, which will allow you to edit the HTML code of the description. Find the bottom of your description, and then paste the TurkGate code in the bottom. When you click the “Source” button again, you should see some text starting with “For technical reasons, you cannot preview the study…”. This means that the TurkGate code has been inserted successfully.
Note that because the TurkGate code does not display the survey link until workers have accepted the HIT, the link also will not appear in the preview. Don”t worry; it will show up!
After that code has been inserted, that’s all you need to do to have your study participants screened for eligibility! You can set up your HIT and run it like you normally would.
Capturing Worker IDs
TurkGate can also pass along worker IDs through the URL to your study. This means that worker IDs are appended to the URL. To enable this in TurkGate, check the “Pass Worker ID” checkbox before generating your HIT. If you use Qualtrics, and you wish to capture worker IDs for use in your survey, or just to associate data with worker IDs directly (which can be useful to verify that workers actually completed the study), just add an embedded data field to the top of your survey flow, with a variable called “workerID”. Make sure you get the capitalization of that variable name correct! When placed at the top of the survey flow, that will capture the worker ID from the URL and store it in your data.
Verifying Codes with TurkGate
One other feature of TurkGate is the ability to create individualized completion codes for each participant. This can be useful if you have concerns that participants might share your completion code with others by posting it online, etc. (I haven’t found this to be a major concern.)
NOTE: This feature is optional. You are still able to give participants your own completion codes at the end of the study if you wish.
If you wish to create individualized codes, simply copy the URL that was given when you generated your TurkGate code. This URL appears directly under the box with the generated code. Copy this URL, and then go to your survey flow in Qualtrics. Place an “End of Survey” element at the very end of your survey flow, and then in the customization options, send your participants to that URL. What will happen is that participants will automatically get redirected to a page that gives them a unique code, which they can copy and paste into mTurk.
The issue here is that you now cannot easily scan through the inputted codes on mTurk and tell which ones are real or fake. However, TurkGate offers a code verification page that makes it simple enough. Download your results by clicking on the “Results” button in mTurk, then clicking “Download CSV” in the top-right corner. Then open this file, highlight and copy the whole thing, and then paste it into the code verification page. TurkGate will go through each code and make sure everything matches up properly, and will either let you know that the code is valid or invalid. Then, you may assign your HITs appropriately.
That‘s all there is to working with TurkGate! Now you can run your studies knowing that the same participants are not (non-naively) participating in multiple related studies, which will help you do good science.
