Tips & Tricks Feed Simple tricks for Managing Workers on Mechanical Turk

Did you know that you can manage your Workers from the Mechanical Turk Requester website?  This feature has been available for a while now but we find there are a few Myths that cause confusion:
Myth: I can't manage Workers via the Requester website if I create my HITs via the API.
Reality:  You CAN manage Workers from the Requester website regardless of how your HITs are created.  Just visit the Manage>Workers section.

You'll see the list of Workers who have worked on your HITs (and their approval rate on your HITs).  Click on an individual Worker to bonus, block or grant the Worker a Qualification.

Myth: I have to page through LOTS of pages of my Workers to find the one I want to bonus, block or assign a Qualification.
Reality: Actually there is a really simple way to manage a Worker without paging through the crowd.  The Manage Workers page includes a list of every Worker who has ever submitted a HIT for you.   If you click on an individual's ID you will be taken to the page that lets you manage this individual Worker.  The url for this detail page includes the Worker's ID in the format https://requester.mturk.com/workers/WORKERID.  Here's an example for a specific Worker:

If you want to bonus a different Worker (suppose it's a Worker with Worker ID 3ABQ167TW). Put this Worker ID after the last backslash in the url (resulting in the url https://requester.mturk.com/workers/3ABQ167TW.)  Once you're on the page for that Individual Worker, you can bonus, block and manage that Worker's Qualifications.

Myth: I have to block Workers individually and that takes too long.
Reality: You can take action on multiple Workers at once by using the downloaded csv file.  To do this download the file of your Workers from the Manage>Workers page.  Follow the instructions from the download wizard to block/unblock Workers or Grant/Revoke a Qualification by marking the appropriate columns and rows in the file and upload it to us.  We'll do the rest.  (Sorry you can't grant bonuses this way.)

Using Mechanical Turk for Categorization 

Response to our last HIT critique was so positive, we’re going to continue offering suggestions here.
This HIT asks Requesters to categorize a web page.

There are several things I like about this HIT:
1)      The HIT requires Workers have the Categorization Masters Qualification.  Workers with this Qualification (aka Masters) have demonstrated consistent, accurate performance on categorization HITs from many Requesters.  This is the fastest way to get consistently accurate results.
2)      Not only does the Requester provide the link to the website, he also loads the website into an IFrame window within the HIT so the Worker doesn’t need to switch tabs or windows.  The Worker has everything they need to complete the HIT right in front of them.  This makes Workers more efficient and improves their effective hourly rate without increasing your costs.  Everybody wins!
3)      The Worker needs to select from a list of 10 categories.

These are placed so that the Worker can see them all without scrolling.  As a Worker hovers over a category a pop-up provides the definition of the category and several examples.  And I LOVE that the submit button is placed right near the categories selection so Workers don’t have to scroll to submit.  That’s a BIG Plus for Workers!
4)      The “Notes” are really instructions.  I like that they provide a clear example:

Without these instructions, I might have put the etsy blog page into the Retail category because etsy is primarily a retailer.  The example makes it clear I should look only at the page provided and categorize the broader site.
5)      I did several of these HITs in preparing this blog post. (Yes, I have earned the Master Qualification the hard way, just like all Master Workers).   One of the websites I categorized didn’t load properly so I selected “Error Loading” and  submitted.  Only after hitting “submit” did I see this additional instruction:

I’d recommend this be added to the Instructions/Notes section.   It would have saved me from a wrong answer.
This HIT follows many of our best practices for Qualifying Workers, providing clear instructions with examples and making Workers efficient by limiting scrolling and multiple windows. Do your HITs?

Mechanical Turk - HIT Critique

We’re often asked to critique a Requester’s HIT to make it better.  This can be of great benefit to the individual Requester (and the Worker who is on the receiving end of the HIT) but it doesn’t allow others to learn.  So I thought I’d take a HIT that’s currently available in the system and critique it.
Below is an image of a HIT that was published for Workers on Mechanical Turk. I don’t know this Requester but my guess is they're goal is to generate metadata to improve the search experience on a website. The HIT does some things quite well but with a few easily adjustments I think this Requester will get better results.
 


Design
I LOVE that the Requester provides a link to the search site with the product pre-populated rather than make the Worker type in the product.  This limits operator error (typos) and makes Workers more efficient.
I also LOVE that the link opens in a new tab. When the link opens inside the HIT window, the website replaces the HIT and the Worker can't see the questions anymore.  Opening the reference link in a new tab makes it easier for the Worker to flip between the HIT and the website you provided for reference.

Worker Qualification
This HIT is available to any Worker with a 96% or higher approval rate.  This includes Workers who’ve only completed 2 Assignments and achieved a 96% approval rate on them.  I think you want more proven Workers than that.  I would recommend you either use Mechanical Turk Masters (Workers who have demonstrated accuracy in completing moderation and categorization HITs) or stick with our recommended Qualifications for other types of HITs:
                                       HIT Approval rate is greater than or equal to 95%
                                                                       AND
                                       Number of HITs Approved is greater than 1000 

Question #1
The instructions for the first question, ask Workers to select which of the provided keywords fit the product. I like that Workers are told that it’s ok if they select NONE of the provided keywords.  Otherwise Workers would likely return the HITs that have no relevant keywords for fear that if they submitted with no keywords selected the work would be rejected.
Also, the instructions don’t tell the worker how to determine which key words are appropriate.  For example, key words for a product are sometimes listed in the product description on the link or inferred from looking at the picture from the link.  Provide examples of what is a good keyword and what is not. 
Question #2
The second question seems ambiguous to me.  Is it mandatory that I provide descriptors?  How many do I need to provide to get paid?  Will I get a bonus if I provide more than required number?
Also you’ll get better descriptors if you explain what makes a “valid descriptor.”   Something like:

• Valid descriptors include generic product categories (for instance if the link is to a TV set then “TV” would be a valid descriptor). 
• Valid descriptors also include features that differentiate this product from others in the generic category (for instance a “LCD” or “Plasma” would be valid for certain TVs.) 
• Color, brand name, accessories and dimensions are not valid descriptors.

Cooking Tip #5 Ask Multiple Workers to complete a HIT

Many Requesters ask multiple Workers to complete the same HIT.  We often refer to this as using “plurality”.  For example, a travel site may ask Workers to categorize images provided by a hotel so they know which ones to feature on their travel site.  The Requester might ask 2 Workers to indicate if the image is of the pool, bedroom, lobby or other location in the hotel.  When 2 Workers come to the same conclusion and provide the same answer independently, the Requester has more confidence in the answer then if one Worker was asked or if 2 Workers provided different answers. 
Often, the goal of using plurality is to get high confidence in Worker answers with the minimum amount of work.  To make sure plurality is working well for you, you need to make sure Workers agree MOST of the time.   The best way to do this is to tightly control which Workers can work on your HITs and make sure your instructions are clear and cover MOST of the “gray areas” or “corner cases”.
 A good way to check that your instructions are clear and complete is to complete 100 of your own HITs with the real data. An even better way is to have a friend or colleague do them – pick someone who isn’t too close to the project. Have them follow the instructions you provide and make a list of the questions they had that your instructions didn’t cover. Adjust your instructions based on this feedback.
Many Requesters are afraid to restrict the Worker Group – won’t my work get done slower? Maybe but it will be done more accurately.  You can restrict your Workers by using Masters – an elite group of Workers who have demonstrated proficiency in certain types of HITs.  If you don’t want to use Masters, you can also create your own custom group of Qualified Workers.   The better qualified your Workers, the more likely each will provide an accurate answer.
If you follow this advice, when 2 qualified Workers disagree, it will likely be that there is an issue with the HIT data or the instructions.  For instance, in our use case what is the correct category for an image of the couch/sitting area of the hotel room?
Plurality is a great tool but like any tool, you have to know when and how to use it.  It’s easy to know what to do when Workers agree – approve the work and use the answer.  But what do you do when they disagree?  Many Requesters ask another Worker and if 2 out of the 3 Workers agree on an answer they use that answer.  That’s fine.  But do you approve assignments from the Worker who disagreed? You should.
Why? Well let’s think about this from a Worker’s perspective.  Workers, especially really good Workers, are reluctant to do HITs with plurality if you are rejecting based on majority rules.  Why is that? Workers, especially good Workers, know they submitted a good answer.  But they don’t know if the other Worker (I’ll call him Worker B) is another great Worker just like them or a not so good Worker.  If Worker B and C didn’t read the instructions carefully or submitted the wrong answer (whatever the reason) you are penalizing Worker A unfairly by rejecting their work.  As good Workers get penalized unfairly, they may choose not to do your HITs (because they want their work to be graded fairly and they want to protect their Approval Rate) and more of your HITs are being completed by Workers like B and C.  It’s a vicious cycle – as more good Workers avoid your HITs, a larger portion of your HITs are completed by less accurate Workers.
To avoid this trap
1)      Tightly qualify your Workers (either by testing and auditing Workers directly or by using Masters)
2)      Make sure your instructions are good and revise them based on patterns in Worker disagreement
3)      Don’t reject assignments based solely on “majority” – either audit it or approve it.   Only reject when you are sure the Worker’s answer was wrong.

 Mturk mail bag: How do I create HITs that require a Worker have an approval rate of 95% on MY HITs?

We recently received the following question from a Requester.
I know I can create HITs that require a Worker have an approval rate of 95% but how do I create HITs that require a Worker have an approval rate of 95% on MY HITs?
This is a common use case so I thought I’d share the answer on the blog.
Mechanical Turk provides a “System Qualification” called HIT Approval Rate.  A Worker’s HIT Approval Rate is calculated as the Workers' Lifetime Assignments Approved divided by the Worker's Lifetime Number of Assignments Submitted -- on ALL Assignments.   Because it’s calculated across ALL assignments that a Worker completes, it doesn’t necessarily reflect a Worker’s ability to accurate complete YOUR HITs.  For instance if a Worker completes mostly photo moderation tasks, they may have a high HIT Approval Rate but they may perform poorly on your HIT that asks them to “Write an article about the motives of the Occupy Wall Street Movement.”
But there is an easy way to require that Workers have a certain approval rate on YOUR HITs. To do this you need to create a Qualification Type and grant it to Workers who have achieved the approval rate you decide is acceptable.  Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools), you can create a Qualification Type via the Requester web site(Requester.mturk.com >Manage>Qualification Types).
Once you create a Qualification Type, you can grant it to Workers with the acceptable approval rate.  So how do you know a Worker’s approval rate on YOUR HITs? Regardless of whether you create HITs via the Requester web site or the developer tools (API or Command Line Tools),you can see your Workers’ approval rates on the Worker Management page of the Requester web site (Requester.Mturk.com > Manage > Workers).   Here is the list of Workers who have submitted assignments for you.  You can filter to show only Workers who have submitted assignments for you in the last 7 days, 30 days or forever.  To grant the Qualification to multiple Workers, use the csv file download/upload option on this page. To grant the Qualification to one Worker, just click on the Worker ID.  You’ll be taken to the Worker’s detail page where you can grant them the Qualification as well as block or bonus the Worker.
BTW, the Manage Worker page also shows you how many assignments each Worker has completed for you.  This lets you find the Workers who've earned a 95% approval rate on your HITs AND completed enough HITs so that 95% Approval Rate matters.
Once you create and grant the Qualifiction, you can use it to control who can work on your HITs. The next time you create HITs (via the Requester web site or the developer tools) just require Workers have the Qualification you created in order to complete your HITs.