Connecting workers has always been the core of our mission. Today, we're expanding that mission with the goal of bridging the communication gap between workers and requesters on MTurk.
TurkerView Bridge is a new development in our web app that we're excited to launch. Over time we hope to build multiple systems to improve interactions between workers and requesters. The first feature, Rejection Disputes, is aimed directly at improving one of the most contentious issues on MTurk: rejected work.
Over the years we've found that most rejections tend to stem from simple misunderstandings. A requester doesn't understand how long their HIT should actually take, a technical issue arises with Qualtrics, or any number of things crop up that lead to rejecting the work without understanding the impact that has on a worker account. Even once the issue is cleared up it can be a monumental task to fix the mistake for requesters not familiar with the platform or those who don't have experience with coding in MTurk's API.
This leaves the burden of explaining why a HIT should be approved, and how to approve it, to the worker. They must compile documentation, converse with the requester, and in some cases teach them how to overturn the rejection manually. We've assisted workers in hundreds of these cases and found the process is tedious, filled with technical pitfalls for the requester, and usually frustrating for the worker since it takes time away from actual work.
Rejection Disputes aims to provide a less cumbersome experience for handling rejections.
Anatomy Of A Dispute
Creating A Dispute
Workers can use TurkerViewJS to open a dispute directly from their MTurk interface. Dispute creation links are available next to the status details of rejected HITs. The process is streamlined and simple so workers don't have to spend time chasing down documentation.
Secure URLs
Once a dispute is created, the worker is forwarded to MTurk's message system to share the dispute with the requester. We fill in all the information for them, and allow the worker to double check the message before sending. They can also view their open rejection disputes from their TurkerView Bridge Account Dashboard
Dispute URLs are formed with two parts:
- A Case Identifier
- A Lock Key
The lock key can be thought of as a shared password that can be attached to the dispute link for simplicity and ease of access. If someone tries to access a case in our system without providing it they will be prompted to enter the correct code before being allowed to view any details of a dispute.
Both parts are required to view a dispute in our system. Because links contain all of the security protocols to unlock a case file, workers and requesters should take caution not to share the links with outside parties unless necessary.
Dispute Details
For the requester, we provide relevant information on overturning the rejection, bonusing a worker, or messaging them through MTurk's platform if they want to notify workers of overturned rejections. Alternatively, they may reply to the dispute themselves and we'll send a notification to the worker.
Dispute Message
A worker may submit a markdown formatted message detailing any evidence they might have or reasoning the HIT should be overturned.
We provide a number of pre-generated responses to workers for common rejection scenarios and encourage them to save time by submitting them as-is or as a template to expand on. Workers may also save their own templates for future use, and make edits to templates through the web app. All communication should follow MTurk's guidelines on polite, professional contact between parties.
Replies
Workers, requesters, and affiliated parties (such as an IRB manager or collaborating researcher) may submit replies to a dispute. A worker's TurkerView account name will never be disseminated through this system, and workers should take caution not to divulge their TurkerView username.
Requesters who have a verified TurkerView profile will have their display name shown and a verified badge displayed to give the worker confidence they're conversing directly with the requester.
Unverified requesters, or third parties, will have a generated hash displayed as their username.
Evidence
Workers often take screenshots of studies as proof of completion when a code isn't present, or they think issues might arise with payment. We've kept this in mind and disputes currently support a secure upload of two (2) pieces of image evidence as well as an area to link to a YouTube video if the worker has video evidence. They should ensure the video is marked private if it contains sensitive information or reveals contents of the HIT.
Action Panel
To help requesters who receive a dispute in our system we link directly to documentation from MTurk detailing the steps to overturn a rejection. This saves both the worker and the requester time, streamlining the process of overturning the rejection if a worker's presented evidence and work quality are sufficient.
CSV Generator
In order to provide the best experience for all parties, and to improve the odds of a positive outcome, we've built a system capable of generating a properly formatted CSV file in accordance with MTurk's standards to approve previously rejected HITs.
Less Friction, Better Outcomes
We've seen the frustration that misunderstandings in this space cause on both sides and we're excited to start offering systems that bring all parties together. The majority of requesters on the platform, especially those affiliated with academic institutions, are great people. We hope this system will help alleviate some of the more common issues new requesters face when making approval/rejection decisions, and reduce the amount of wasted time required of workers to fix issues when they do arise.
Workers can begin utilizing this new system by updating or installing TurkerViewJS. There's also a fully functional Example Dispute Case on our website, and we've generated a quick Info section about the system with answers to frequently asked questions.
