Gamification

At first glance, the essential skills needed to become a data annotator are generally quite straightforward: a minimum of computer literacy, attention to detail and, of course, some free time on your hands. In certain cases, these requirements are insufficient, and some platforms even require software development skills.

As we go deeper into data labelling, it becomes increasingly clearer that people need to be instructed on what they have to do. Additionally, we need dependable methods for assessing their performance in carrying out instructions. Is it enough to just provide them with guidelines on how the data needs to be labelled? Appen has proved that in some cases it is sufficient, but also that things can go horribly, horribly wrong.

At Timeworx, we’re taking a different approach. We’re reducing friction and removing barriers, so that human labellers can start solving tasks and receiving rewards without any effort. The ability to reach large and varied audiences in a very short time leads to an increase in fairness, a reduction in bias, and also gives us a glimpse into how the human mind works regardless of geography, background and culture. With many users going through the same flows day after day, we are able to validate our hypotheses by designing and running experiments in which we observe the behaviour of our Human Agents across a given period of time. This allows us to speed up our learning process, pivot when necessary, and build automatic quality control measures based on facts instead of hunches.

Throughout our experiments in the Closed alpha testing phase, we have processed publicly available datasets which allowed us to validate that Human Agents are able to reach an accuracy upward of 97%. However, we have also encountered Human Agents that exhibit free-riding behaviour - they want the rewards, but don’t want to work for them. So they look for exploits and try to make the most of them. We have discovered that the incentive to solve a task must outweigh the effort of exploiting the task itself.

The platform tackles these issues head on by implementing a range of gamified incentives for ensuring that our Human Agents are responsible, engaged and are constantly seeking to improve their performance for increasing their rewards.

Awards and Penalties

Aside from constant monitoring of KPIs, an additional measure of determining user performance & intentions is implemented in the platform through speculative Tasks. The outcome of a speculative Task is known in advance, and is used to swiftly determine if a user has bad intentions, or is treating task solving with little-to-no rigour. These Tasks, which are indistinguishable from others, are randomly distributed to Human Agents in a manner that eventually leads to the entire data processing community passing through this additional check at some point in time, at least once. As opposed to regular Tasks that have to go through Consensus to determine the performance of users, speculative Tasks provide instant feedback which can be acted upon on the spot.

In the true spirit of reinforcement learning, Timeworx.io rewards good performers, suggests paths for improvement to poor performers, and discourages intentional free-riding or exploitative behaviour.

As Human Agents improve their KPIs and get better results at solving Tasks, the platform periodically selects the best performers from the leaderboard and provides incentives such as task reward multipliers from the gamification pool of tokens, staking reward multipliers, Discord roles, on-chain proofs of accomplishment, and many more.

On the other hand, when detecting free-riding or exploitative behaviour, the platform issues penalties in the form of fines. As such, Human Agents are withheld an amount of TIX from their task rewards, corresponding to the proceeds that had been obtained from unethical actions. Offending users can then pay off their penalty through solving Tasks without the benefit of the reward. Needless to say, such users are served speculative Tasks with a much higher rate, and, in case their performance does not improve (or worsens), they are flagged and the platform stops serving them with Tasks altogether. All of the rewards corresponding to Tasks solved while paying penalties are automatically transferred to the TIX gamification pool, to the benefit of the entire community.

Custom-tailored Improvement Paths

Timeworx.io does not require any prior training from Human Agents and, based on monitoring their KPIs, can provide feedback and insights through a gamified in-app experience:

1. Practice sessions: Human Agents can get acquainted with new types of Tasks by running test trials on speculative tasks. Mistakes are highlighted and corrective instructions are provided to the users.

2. Tips & tricks: in-app messages and notifications that guide the users towards improving their KPIs. The mobile app is able to provide hints about new Tasks that a Human Agent can try to improve Versatility, and can suggest Practice sessions for Tasks that can help improve Speed or Proficiency.

3. Daily streak: Reliability is improved by keeping track of the number of consecutive days of solving tasks for each Human Agent. Users are incentivised to build higher streaks with each run and are also notified when their streaks are in danger of expiring.

4. Performance Review: Human Agents can view a detailed list of Task outcomes that were not marked as correct during Consensus runs. This can help users better understand how to improve their performance in the future.

Leaderboards & Challenges

Competition has always pushed people towards progress. Whether it is the sense of accomplishment when improving and going up places in a leaderboard, the thrill of the chase when reaching the top, or the overwhelming and dopamine-ridden pride when reaching the #1 spot, we are all passionate about achieving our goals.

Therefore, we have designed a set of gamified mechanics to engage people into friendly & fun competition:

1. Leaderboards: daily, weekly, monthly and all-time statistics allowing Human Agents to get a better understanding of their current standing, their progress and how they compare with the rest of the community.

2. H2H: Human Agents are able to challenge each other to go head-to-head in a round of solving tasks. In this battle of KPIs, the winner takes all of the rewards associated with all of the tasks solved during the match.

3. Public challenges: Based on eligibility criteria, a common goal and a clear timeline, a public challenge is issued to all Human Agents, for example: “Reach Level 5 in the next 12 hours”. Any eligible individual can participate by locking in their commitment using a minimum amount of TIX before the challenge begins. At the of the event’s timeline, the entire pot of locked TIX is distributed to all Agents that have reached the goal set by the challenge, proportional to their initial wager.