We're going into our new partnership with MTurk Suite by taking a deep dive review of our site, processes, and how we're serving the worker community.

It's been over two years since TurkerView launched, and in that time we've seen amazing progress around fair wage practices. We're blessed with a strong community of workers, and with their expertise on the platform we want to share what we've learned and share insight on how we can approach payments for crowd workers moving forward.

Measuring Progress

We selected users who submitted data to TurkerView during our first few months of operation and who have continued to interact with the platform through 2019 in order to create our core userbase. Their data is at the forefront of understanding how wages are impacted by down time, wasted effort, task searches, and tax obligations. They are the driving force behind our efforts, and we examined their earnings on the platform in order to get a more controlled overview of wage trends.

In Task Earnings for Core users, 2018-2019 (all tasks)

Overall these users have seen a 30% total increase in their average task wage over the last 2 years. That's fantastic progress but it's important to remember these users generally operate in the higher efficiency spectrum of workers and this doesn't cover time spent in between tasks.

Wage Sentiment Analysis

As we've seen wages increase across our core userbase we've also noticed a shift in their expectations for payment. On TurkerView users are asked to rate how they felt about the payment of a task they're reviewing and given a basic guideline on suggested ranges—and despite not updating that guideline since its initial launch, workers have begun proactively shifting their expectations and relaying them to fellow workers.

Average Wage Sentiment, 2018-2019 (tasks paying $7.25/hr-$10.50/hr)

For tasks existing in the same pay range worker sentiment on those tasks has dropped almost 11%. Across time, and as their own wages increase, workers are rating tasks with stagnant wages as no longer acceptable.

These shifts match across most of our currently proposed ratings guidelines which are displayed to workers every time they leave a review on our website. It's great to see workers continuing to value their own work and time, and we want to facilitate that by updating our services to reflect those expectations.

What We're Doing

Sentiment Ratings

Continuing to use historical sentiment ratings makes it difficult to convey our user's current feelings. While helpful in the moment perspective changes over time and we're seeing a lot of indications the sentiment ratings aren't doing a great job of projecting accurate information. In response to this we'll be removing them from the web platform and turning them into an internal metric for workers only (similar to our return review system).

Workers won't see any adverse effects to their current tools, and our API will continue to return these ratings to them for evaluation in their workflow if desired, but these changes will allow us to offer time based metrics on requester accounts so past payment practices carry less weight.

Wage Guidelines

As we push for the fair payment for all workers it's important to set new standards on what is deemed "fair". The tables below represent our plan for keeping our platform up to date, and in the service of workers, in 2020. We've used feedback from workers (and a special thanks to Mark Whiting for publishing his team's data on this topic) to settle on ranges that more accurately reflect their associated sentiment tier. We plan to update our guidelines quarterly, and to reflect the tax burdens self employed individuals face (which apply to all crowdworkers on MTurk and similar platforms).

2020 (Q3) Wage Guidelines (reviewed quarterly)

We understand the budgetary strains increasing our guidelines can place on requesters, but when used properly MTurk is a source of high quality, convenient participant samples. We want to move in a direction that shares the value the platform provides with the people actually completing work. We are always available to mediate discussions with funding sources or IRBs if researchers need help conveying the importance of paying fair wages.

These guidelines will be applied to tasks posted in Q3 2020 and onward to give requesters a fair amount of time to adjust. As it would be unfair to retroactively apply them to researchers who used our original guidelines for pricing we'll be splitting historical wage & sentiment statistics from the old guidelines for display on our web app. This means only tasks posted after July 1st will be measured under the new guidelines.

Representing Fairness - Return Reviews

In response to the removal of our sentiment ratings from the web platform we'll be replacing them with something we feel more accurately measures the rating's purpose of understanding if a requester is treating workers fairly.

Currently our users have access to an internal review system for "bad" tasks that allows them to quickly alert other workers that a task violated a set of worker generated guidelines (including low payment, unfair screening practices, broken tasks) that generally lend themselves to quicker, less permanent submissions.

This data is rather transient in the current system as the reports are only tied to individual tasks. It's difficult for users to tell if a requester has a history of problems using this feature, so we'll be bringing the aggregate statistics from these reviews onto the web platform for more long form worker review. We'll only display filtered aggregate data, so workers can continue to feel secure in sharing their experiences, but we hope this information will add some nuance to requester profiles that don't always paint a full picture. We hope they'll also give insights for requesters to see where their task design may be falling short in helping workers provide quality data.

We're Making Progress

We cannot express enough gratitude to the hundreds of researchers, businesses, and workers who engage in open discussion about online crowd work. The positive trends we've seen over the last two years are impressive.

We'll be adding extra emphasis around our public facing applications to ensure attention is brought towards what constitutes generous payments in order to help facilitate these discussions, and will continue to pop in on social media to share these thoughts as well!

For workers, we're excited to bring these changes to our platform and, as always, look forward to feedback from the broader MTurk community.

TurkerView ❤️ MTurk Suite

We’re excited to announce that Brandon Hellman, well-known in the worker community as "Kadauchi", has officially joined TurkerView as our acting Technical Co-Founder!

Best known for his all-inclusive extension MTurk Suite, Brandon will be leading our technical development of worker productivity tools. While we’ve worked side by side for years, entering into a formal partnership will allow us to communicate more clearly, divide tasks and development responsibilities, and better serve workers who rely on us. This partnership will bring a more streamlined, efficient experience to workers and allow us to focus on our shared goal of improving working conditions on the internet’s largest crowdsourcing platform, Amazon’s Mechanical Turk.

Combined Efforts

TurkerViewJS, our flagship tool, has grown far too complex for a userscript. Building a browser extension from the ground up would take weeks of development time, but a full integration with MTurk Suite—utilizing Brandon’s extensive product development experience—will open up a faster path to a wide range of productivity enhancements. Pooling our resources is also allowing us to keep core functions and services free for workers who have come to rely on them.

Additionally, TurkerView has a wealth of informational services that will now be easily available to workers through the extension. Did you know that upwards of 10% of HITs returned by new workers are the result of simply misunderstanding how the platform functions? We do, and it represents thousands of dollars in missed rewards for workers (and valuable data for researchers!) every month. Mturk Suite’s intelligent, user friendly design unlocks powerful avenues for us to help guide workers to better, more productive rewards.

Looking Ahead

The already-in-progress rewrite of MTS v3 has offered us an opportunity to more closely integrate our services, both old and new. We’ll be focused on crafting an experience that allows new workers to get up to speed, and reduce the friction of transitioning to an online work environment with educational resources, guided informational analytics, and improved UI. Simultaneously, we’ll be bolstering the productivity of workers well-versed with the existing systems by offering faster, lower response rate APIs; better filtering of existing data sets to reduce information not relevant to your experience level; and more streamlined access to services pertinent to an advanced worker’s daily usage.

In the future we’ll be focused on the larger ecosystem. The MTurk team has continued to support workers in ways most platforms don’t, and we’re happy the infrastructure they’re building on AWS is the premiere platform for business and academia to recruit workers for their AI, machine learning, and research projects. We want to support that with worker-guided resources and interaction through projects such as our recently launched Rejection Dispute System.

Finally, we believe that it’s important to make sure we’re looking at features and services beyond the data. Online work can be hectic, and since TurkerView started as a simple forum community, we want to keep that spirit by taking a deep dive into ways we can improve worker quality of life.

Update Path

We’re doing everything we can to ensure this transition is smooth for workers. Updates are scheduled for the lowest impact times for our core userbase (targeting weekends, early morning EST in the 1-3AM range), but otherwise things will hum along as expected. The transition should actually allow us to immediately improve response times an estimated 30%, and improve our ability to respond to disruptions with our service providers. In an environment where milliseconds matter, this is exciting news!

If you have any feedback or suggestions, please reach out and let us know what we can do to make your experience on the platform better! Your support drives the next generation of worker produced tools, and we’re incredibly excited to see where this next step takes us.

Connecting workers has always been the core of our mission. Today, we're expanding that mission with the goal of bridging the communication gap between workers and requesters on MTurk.

TurkerView Bridge is a new development in our web app that we're excited to launch. Over time we hope to build multiple systems to improve interactions between workers and requesters. The first feature, Rejection Disputes, is aimed directly at improving one of the most contentious issues on MTurk: rejected work.

Over the years we've found that most rejections tend to stem from simple misunderstandings. A requester doesn't understand how long their HIT should actually take, a technical issue arises with Qualtrics, or any number of things crop up that lead to rejecting the work without understanding the impact that has on a worker account. Even once the issue is cleared up it can be a monumental task to fix the mistake for requesters not familiar with the platform or those who don't have experience with coding in MTurk's API.

This leaves the burden of explaining why a HIT should be approved, and how to approve it, to the worker. They must compile documentation, converse with the requester, and in some cases teach them how to overturn the rejection manually. We've assisted workers in hundreds of these cases and found the process is tedious, filled with technical pitfalls for the requester, and usually frustrating for the worker since it takes time away from actual work.

Rejection Disputes aims to provide a less cumbersome experience for handling rejections.

Anatomy Of A Dispute

Creating A Dispute

Workers can use TurkerViewJS to open a dispute directly from their MTurk interface. Dispute creation links are available next to the status details of rejected HITs. The process is streamlined and simple so workers don't have to spend time chasing down documentation.

Secure URLs

Once a dispute is created, the worker is forwarded to MTurk's message system to share the dispute with the requester. We fill in all the information for them, and allow the worker to double check the message before sending. They can also view their open rejection disputes from their TurkerView Bridge Account Dashboard

Dispute URLs are formed with two parts:

1. A Case Identifier
2. A Lock Key

The lock key can be thought of as a shared password that can be attached to the dispute link for simplicity and ease of access. If someone tries to access a case in our system without providing it they will be prompted to enter the correct code before being allowed to view any details of a dispute.

Both parts are required to view a dispute in our system. Because links contain all of the security protocols to unlock a case file, workers and requesters should take caution not to share the links with outside parties unless necessary.

Dispute Details

For the requester, we provide relevant information on overturning the rejection, bonusing a worker, or messaging them through MTurk's platform if they want to notify workers of overturned rejections. Alternatively, they may reply to the dispute themselves and we'll send a notification to the worker.

Dispute Message

A worker may submit a markdown formatted message detailing any evidence they might have or reasoning the HIT should be overturned.

We provide a number of pre-generated responses to workers for common rejection scenarios and encourage them to save time by submitting them as-is or as a template to expand on. Workers may also save their own templates for future use, and make edits to templates through the web app. All communication should follow MTurk's guidelines on polite, professional contact between parties.

Replies

Workers, requesters, and affiliated parties (such as an IRB manager or collaborating researcher) may submit replies to a dispute. A worker's TurkerView account name will never be disseminated through this system, and workers should take caution not to divulge their TurkerView username.

Requesters who have a verified TurkerView profile will have their display name shown and a verified badge displayed to give the worker confidence they're conversing directly with the requester.

Unverified requesters, or third parties, will have a generated hash displayed as their username.

Evidence

Workers often take screenshots of studies as proof of completion when a code isn't present, or they think issues might arise with payment. We've kept this in mind and disputes currently support a secure upload of two (2) pieces of image evidence as well as an area to link to a YouTube video if the worker has video evidence. They should ensure the video is marked private if it contains sensitive information or reveals contents of the HIT.

Action Panel

To help requesters who receive a dispute in our system we link directly to documentation from MTurk detailing the steps to overturn a rejection. This saves both the worker and the requester time, streamlining the process of overturning the rejection if a worker's presented evidence and work quality are sufficient.

CSV Generator

In order to provide the best experience for all parties, and to improve the odds of a positive outcome, we've built a system capable of generating a properly formatted CSV file in accordance with MTurk's standards to approve previously rejected HITs.

Less Friction, Better Outcomes

We've seen the frustration that misunderstandings in this space cause on both sides and we're excited to start offering systems that bring all parties together. The majority of requesters on the platform, especially those affiliated with academic institutions, are great people. We hope this system will help alleviate some of the more common issues new requesters face when making approval/rejection decisions, and reduce the amount of wasted time required of workers to fix issues when they do arise.

Workers can begin utilizing this new system by updating or installing TurkerViewJS. There's also a fully functional Example Dispute Case on our website, and we've generated a quick Info section about the system with answers to frequently asked questions.

Imagine if someone told you they’d decided they were going to work for Uber—despite not having a car, not having a license to drive a car, or even having ridden in a car.

Sound ridiculous? Well, that's how the article about Mechanical Turk (MTurk for short), published by The New York Times, sounds to many workers experienced with the platform.

Articles like this don't hurt Amazon; they hurt you. They hurt me. They drive requesters off the platform and towards platforms that have the outward appearance of caring (like Prolific) while giving significantly less of a shit than Amazon ever could.

Our site, TurkerView, is dedicated to tracking wages on Mechanical Turk and providing workers with tools and services to increase their productivity. TurkerVew is now the largest active community of workers, and the only worker operated review platform: it's an eclectic group of college kids, stay at home parents, system administrators, freelance writers, and even a prospective alpaca farm owner on one memorable occasion.

We serve thousands of workers every day across our platform's various services/APIs. Contrary to what The New York Times article would suggest, wages have actually risen over 35% (from $14.50/hr in Q1 2018 to $20.00/hr in Q4 2019) among workers in our community—in fact, the change has been so significant we've started overhauling our 2020 wage guidelines in response.

We're publishing this because we, the administrators and active users of TurkerView, believe the Times's article portrays the MTurk platform poorly and inaccurately.

People working on MTurk are not making $0.97/hr

Wage Trends for TurkerView users, 2018-2019 (all tasks)

TurkerView makes it easy to browse through the most active requesters, or even the specific universities and colleges that successfully utilize MTurk. No doubt there are poorly paid tasks on MTurk. That's the nature of any marketplace. Unique to other marketplaces, where the best rise to the top, MTurk works in the opposite direction: the worst of the worst tend to be the most visible when you first log in because people aren't doing them.

MTurk employees at conferences like re:Invent have advocated for paying people fairly for work on the platform. Nonetheless, some businesses still use MTurk to outsource their microtasks to the sizeable portion of foreign workers on the platform. We're not going to make ethical judgements about that since many of those tasks do provide a livable wage to those workers, but it's important to note the author largely focused his efforts on those tasks that most American workers rightly avoid.

That article is condescending.

If a stranger walked into your job Monday morning, could they expect to perform your duties as deftly as you can? Probably not! It takes years of education, training, and experience to excel at many jobs. If someone wants to treat MTurk as such by discussing wages, the same applies. Using MTurk successfully requires workers to be disciplined, computer-savvy, and flexible with what they do at any given moment. Focus is a must as they manage complex workflows and quickly infer the critical parts of tasks — some of which help surgeons save lives, deliver medicine to villages in Africa, or a myriad of other projects that touch on all parts of digital life we all enjoy.

Perhaps what's most important to note about the article is that when the author attempted to interview workers, he was largely rebuffed. Most MTurk communities have seen this article rehashed time and time again, and we all knew what this was going to say long before it came out. One worker, Doc3D, explained this in the author's initial call for quotes:

You will likely get a very narrow view by the person who would take an offer like this. This would likely be the same people who do the lowest paying hits. There is a somewhat wide range in type of turker. There are batch workers only, survey workers only, transcribers, etc, and then people who do a combination of those. There are workers who will work a 30 minute hit for only 10 cents, and then there are workers who would only do that same hit if it paid at least 3.50. There are workers who have closed quals from several institutions and work those almost exclusively. Then there are workers who will work for just about anything and for any amount. You won't get the full scope of what it's like to turk by just interviewing a single person, and by only offering a dinner - the chances are pretty great that the majority of anyone who would respond is the same people working the absolute minimum hits and that will just skew your perception of the whole thing.

We're not here to disagree with the viewpoints of workers who believe that MTurk doesn't do them justice, and we certainly don't fault them for sharing their experiences. Our job is to support them as best we can. In fact, if you're struggling on MTurk, let us know—we'll do a workflow review & do our best to help.

However, the TurkerView community has dedicated the last two years to building tools and a platform for the purpose of helping workers, and we would argue that few people are more familiar with the stories of workers than we are. As it is, the media has been interested in MTurk from only one angle, and many workers find that to be as exploitative as the articles claim MTurk to be. Experienced workers know the value of their time, and aren't interested in any unpaid platform-related work. As one Reddit user said while explaining why workers did not want to participate:

This isn't the same as meeting a friend and discussing your job. It is work for him that will earn him money and ultimately bring money to The Times. The online version of the story will benefit the publisher and this journalist via website visits, ads, subscriptions, shares and improved brand and individual recognition and reputation.

The truth is, MTurk is the most ethical way to recruit crowd workers.

Bold statement—we know. But we believe it's true and needs to be directly stated.

MTurk's fees are some of the most reasonable in the space—for example, drawing from MTurk's worker pool allows Positly to supply requesters with a reliable set of workers with total fees lower than similar offerings. Amazon has continued to allow workers freedom to work, build, and tweak the platform to suit their individual needs, and offers both workers and requesters powerful APIs in addition to their default web interfaces.

We're not going to mince words here: the Times's willingness to profit off workers by using one sided, emotionally charged titles and stories is not a good reason to abandon MTurk. Neither is methodologically unsound "research" that claims wages are bad across the board (yes, we'll address that paper in-depth later).

Still, the ultimate problem with these articles is that they aren't just one sided: they're damaging.

...articles like this shred every bit of integrity of what we do for a "living".

Imagine trying to explain to a corporate recruiter that for months/years, you worked on a service and support platform where you managed personal micro-projects for a variety of requesters, earning a viable wage in return for self-motivated work for major universities, Amazon, Google, Microsoft, etc. all the while building skillsets in various technological tools and areas of interest, just to have them ask what the name of that platform was.

You say "Mechanical Turk".

They laugh, because they've read the articles. They've read "Ghost Work" by Siddharth Suri et. al. They know that you're not a motivated self-starter; they think you're a cog in the machine, earning $2/hr. They liken you to whats-her-name in India, who in Ghost Work was incapable of performing a HIT if her big brother wasn't in the room. Your microtasking work is suddenly no longer skilled, or integral, or even meaningful. You're a pleb that types lines on a receipt. Hell, you may have even bought your account from a black market in the third world; after all, Amazon has no controls in place, right?

They laugh, because they've read the coverage on the Bot Scare(TM). They think that there's no way your work can be skilled, because a rudimentary computer program can do the same thing and avoid detection in most cases. In the case that you are somehow able to prove you made decent money (1099 anyone?), you must be a scammer. After all, only the top 0.00242% of workers make over $7.25/hr, or whatever the latest bullshit "research" statistic is. Certainly you didn't get there on your own accord; you must be cheating.

Articles like this don't hurt Amazon; they hurt you. They hurt me. They drive requesters off the platform and towards platforms that have the outward appearance of caring (like Prolific) while giving significantly less of a shit than Amazon ever could.

Good requesters who don't want the stigma associated with the platform wind up leaving, or never coming at all—often turning to competing platforms that charge even higher fees (meaning less of their budget actually goes to the worker) or have worse experiences for their workers (including loss of freedom, transparency, and lower wages). Articles like this make using MTurk taboo, and workers do not deserve to be punished because of it.

MTurk is, by and large, an incredibly solid platform for online crowd work. Its fees are reasonable, workers have a lot of freedom in choosing tasks, the quality of data remains high for requesters following best practices, and it allows for development on top of the platform that gives it unparalleled flexibility through Amazon's AWS offerings.

Moreover, many workers get a real sense of satisfaction out of completing scientific research on the platform.

MTurk is, in our opinion, the best option for online crowdsourcing—both economically and ethically—and potential requesters should feel confident using it for any project they have, be it professional or personal. We at TurkerView are going to keep working on strengthening lines of communication between all parties, which means listening to the experiences of all types of workers—not just pretending to be one for a week, then moving on.

If you want to join us, we'd love for you to share this on social media or get in touch with us. There's an endless list of projects that could be completed to improve worker's experience on the platform, and that's what we're here to help facilitate. Drop in, chat directly with workers, use your profile to get a sense of what they're saying about your work and improve it.

If we focus more on mutual respect and less on attention-grabbing clickbait, we can all benefit from an ecosystem that is continually evolving.

Every day academic researchers, prospective PhDs, and university labs post hundreds of studies to Amazon's Mechanical Turk (MTurk). Most workers are proud of the part we play in helping researchers across the globe gather the data they need to publish their work.

TurkerView, for those unfamiliar with us, is a website dedicated to helping workers increase their earnings by providing tools & information about the platform, mostly in the form of wage data & user experience reviews with requesters. Nowadays we do a lot to connect workers with researchers (hundreds have already claimed their profiles & engaged with workers on our site) and we're aiming to improve that service with our new Institution Profile system.

Reputation Building, Reputation Sharing

It's common for workers to be wary of "new" requesters entering the ecosystem, which can limit the pool of subjects those researchers can draw from. This is especially true for new workers, who are often told to avoid working for "unrated" requesters.

Over the last few years, however, we've noticed that many institutions do an outstanding job of guiding researchers to higher quality data and helping them support workers in an ever evolving technical ecosystem. Wharton, Duke ^{1 2 3}, and Yale ^{1 2 3} are prime examples of institutions with generally well established reputations, but no way to share their reputation across their numerous labs & individual researchers.

That's where Institution Profiles, now live on TurkerView, enter in. We hope that highlighting their contributions to crowd work at a higher level will allow workers to feel more confident working with researchers from well respected and established institutions and help requesters of all types find positive examples to emulate.

Building Better, Together

The salient point consistently lost through online work platforms is that we're all human. We're all dealing with other humans from incredibly diverse, often eclectic backgrounds with their own unique motivations for being here coupled with one of the most difficult mediums of human connection & communication we've had since smoke signals were invented.

So if your requester profile or school profile doesn't look like a Wharton case study yet, that's okay! No one wants it to get there more than us. We're always available to discuss your profile & provide avenues to pave the way to a better experience for everyone.

Mechanical Turk is an amazing framework built on the rock solid foundation of the world's leading cloud computing provider. The worker community has a long, storied history of building amazing things on top of that framework and we believe the next step of that is to bridge the digital gap by working with academics.

What can you do to help build your school's reputation?

It's "simple" really: post great work!

We know that isn't nearly descriptive enough (subscribe to our blog where we'll discuss this in depth in the future) but here is a short checklist to get started:

• Use the sandbox environments (Requester | Worker) to check for errors / bugs
• If a worker can accept the task, they should be able to submit it (use correct qualifications & pre-screening practices)
• Set appropriate reward amounts ($12-$15/hr for the lower quartile of participants as a minimum baseline, we recommend requesters aim for $20-$25/hr average)
• Your research must be approved by the IRB or ERB associated with your school to qualify, and that information must be presented to participants prior to the task.

It's important to us to have a wide array of positive profiles on our platform because it means our worker pool is successfully finding engaging, rewarding tasks while contributing to meaningful studies. Hopefully unsurprisingly, contributing to research is actually important to a lot of workers

The best thing you can do is spend resources investing in higher quality study design, efficient task flow, and increased wages. The current in-task average wage for an academic study posted to MTurk in 2019 is hovering around ~$20/hr according to our datasets, so it's important to price your tasks competitively so workers will give their full time & attention to the task!

Average in Task Wages, 2019 (surveys only)

We know anything involving dollar signs can be a sticking point, especially for some researchers funding studies out of pocket, but it's important to note that better wages doesn't always mean increased task reward amounts. Improving task design, or simply posting tasks that work the first time, can go a long way in this area. If you're not sure how to do this always feel free to reach out for a second opinion to see if there are problems, pitfalls, or areas for improvement from a worker's perspective. Especially when considering the budgetary waste associated with tasks that produce unusable results the time spent publishing a well designed, thoroughly tested study can pay rewarding dividends for both parties.

Contributing positively to the ecosystem & using it responsibly will help build both your profile on our platform and the institution you publishing research at. You can also claim your individual profile to signal to workers that you're engaged with understanding best practices and available for contact. Otherwise, we'll do the rest!

If you're a worker looking for more information on how this new feature can help you, and where we're taking it in the future, please check this post.