Book a 30 minute meeting now
Built BY RESEARCHERS from
reduction
Compare
Routes to the mathematically optimal combination of labelers
Each time you add more qualified human labelers, train more AI, and update the router and combiner, you improve the quality and lower cost.
Quality
It’s important to know each part of the pipeline is high quality.
1. Skill Tree
They must pass skill tests such as understanding English, drawing image annotations, following instructions, etc.
2. Template qualification
They must qualify on the data program template. This is useful if you haven’t yet uploaded enough ground-truth data.
3. Instance qualification
The final test is to be able to pass data program tasks based on your specific data. This is the most important qualitification.
Tasks we already know the answers to (ground truth) are sent to labelers to randomly check if they are still giving high-quality answers.
Labelers who are very good get sent less checks, labelers who are questionable get sent more checks. The frequency is determined by reinforcement learning (based up a partially observed Markov decision process).
Consistency score is set up for evaluating consistency of translation quality. For instance, if your last 3 scores are "7.0", "7.0", "7.0", your average score is "7.0". Meanwhile, if your last 3 scores are "10.0", "10.0", "1.0", your average score is sitll "7.0", but your "consistency score" will be signicantly lower than the previous case.
Sending the same task again but with some permutations
is a good way to assess this drop in performance.
Labers get compensated based on the value they add to you. This creates a deep incentive to get the answer correct, improve their skills, and level up. High performing people get more responsibility over time.
The generative model, models correlations between labelers, estimates a latent trust score, and predicts how high quality a labeler will be on a specific task given other labelers working on the same task. If two labelers are very similar in their predictions it doesn’t make sense to combine them in a consensus task, but it does make sense to use them to predict and assess quality.
1.
Skill Tree
They must pass skill tests such as understanding English, drawing image annotations, following instructions, etc.
2.
Template qualification
They must qualify on the data program template.
This is useful if you haven’t yet uploaded enough ground-truth data.
3.
instance qualification
The final test is to be able to pass data program tasks based on your specific data. This is the most important qualitification.
Tasks we already know the answers to (ground truth) are sent to labelers to randomly check if they are still giving high-quality answers.
Labelers who are very good get sent less checks, while labelers who are questionable get sent more checks. The frequency is determined by reinforcement learning (based upon a partially observed Markov decision process).
Consistency scores are powerful because they can be used even if not ground truth is present. Sometimes there are temporary drops in performance, due to fatigue or distraction.
Sending the same task again but with some permutations
is a good way to assess this drop in performance.
Labelers get compensated based on the value they provide to you. This creates a deep incentive to get the answer correct, improve their skills,
and level up. High performing people get more responsibility over time.
The generative model, models correlations between labelers, estimates a latent trust score, and predicts how high quality a labeler will be on a specific task given other labelers working on the same task.
If two labelers are very similar in their predictions it doesn’t make sense to combine them in a consensus task, but it does make sense to use them to predict and assess quality.
The most trust worthy and uncorrelated labelers are and they agree, the higher quality the end result. If labelers disagree and the entropy of this disagreement goes above a threadhold (based on your desired performance) then the task goes down a further processing funnel to guarantee quality.
Not all tasks and data are the same difficultly level. Some tasks can be answered confidently by 1 labeler, some tasks need a combination of multiple labelers. The incremental relabeling system determines how many labelers are needed for a particular task to get a high quality result.
The most trust worthy and uncorrelated labelers are and they agree, the higher quality the end result. If labelers disagree and the entropy of this disagreement goes above a threadhold (based on your desired performance) then the task goes down a further processing funnel to guarantee quality.
After a task is passed through the QA pipeline it will either get resolved as high enough quality, or it will eventually get passed to an expert review team to ensure ultimate quality.
This is a committee of trained experts who follow a control procedure to make a final determination on a task. You can also use your own experts. The goal is to use this commitee as little as possible.
A likelihood classifier is use to create a distribution of the expected bounds of outputs. If the actual output deviates too far far the expected statistical distribution then the task is flagged for further review.
The most trust worthy and uncorrelated labelers are and they agree, the higher quality the end result. If labelers disagree and the entropy of this disagreement goes above a threadhold (based on your desired performance) then the task goes down a further processing funnel to guarantee quality.
Not all tasks and data are the same difficultly level. Some tasks can be answered confidently by 1 labeler, some tasks need a combination of multiple labelers. The incremental relabeling system determines how many labelers are needed for a particular task to get a high quality result.
The most trust worthy and uncorrelated labelers are and they agree, the higher quality the end result. If labelers disagree and the entropy of this disagreement goes above a threadhold (based on your desired performance) then the task goes down a further processing funnel to guarantee quality.
After a task is passed through the QA pipeline it will either get resolved as high enough quality, or it will eventually get passed to an expert review team to ensure ultimate quality.
This is a committee of trained experts who follow a control procedure to make a final determination on a task. You can also use your own experts. The goal is to use this commitee as little as possible.
A likelihood classifier is use to create a distribution of the expected bounds of outputs. If the actual output deviates too far far the expected statistical distribution then the task is flagged for further review.
Instructions are created as modules. Each module can be audited and tested separately. If tasks in a data program categorically drop below the quality baseline, then instructions are sent through a master data program, broken down into their smallest pieces and run to check for consistency.
If the automatic escalation process does not yiled higher quality instructions, or an anomaly is detected, the instructions get sent to customer success representatives who are experts in writing and debugging instructions.
An NLP classifier is run to check for proper sytactic and semantic structure. The classifier is trained on millions of successful instructions and can automatically detect the most common errors and give an assessment of what to fix. You can set it to fix things automatically.
Instructions are created as modules. Each module can be audited and tested separately. If tasks in a data program categorically drop below the quality baseline, then instructions are sent through a master data program, broken down into their smallest pieces and run to check for consistency.
If the automatic escalation process does not yiled higher quality instructions, or an anomaly is detected, the instructions get sent to customer success representatives who are experts in writing and debugging instructions.
An NLP classifier is run to check for proper sytactic and semantic structure. The classifier is trained on millions of successful instructions and can automatically detect the most common errors and give an assessment of what to fix. You can set it to fix things automatically.
The data linting routines automatically 1) Flag potiential issues in the data (e.g. numerical features with widely different scales), and 2) Suggest fixes or potientially useful feature transformations of the data. It works on structured and unstructed data.
The noise model is trained over billions of unstructured data points (images, audio, video, etc.) to determine a proper range of the signal to noise ratio of your particular data. It merges Gaussian additative noise, Poisson noise, and multiplicative noise to identify bad data. In addition to the pior distribution of noise, the noise models are trained on your successfully labeled data to decrease the variance of the proper signal to noise ratio of your data.
If the certainty of the noise model or the data linter is not high enough yet, the data will get passed the the master data quality data program to get audited by a crowd of expert labelers. If the data is determined to be below a threshold needed for the desired quality, you’ll get alerted and suggestions will be made on how to fix it.
The data linting routines automatically 1) Flag potiential issues in the data (e.g. numerical features with widely different scales), and 2) Suggest fixes or potientially useful feature transformations of the data. It works on structured and unstructed data.
The noise model is trained over billions of unstructured data points (images, audio, video, etc.) to determine a proper range of the signal to noise ratio of your particular data. It merges Gaussian additative noise,
Poisson noise, and multiplicative noise to identify bad data. In addition to the pior distribution of noise, the noise models are trained on your successfully labeled data to decrease the variance of the proper signal to noise ratio of your data.
If the certainty of the noise model or the data linter is not high enough yet, the data will get passed the the master data quality data program to get audited by a crowd of expert labelers. If the data is determined to be below a threshold needed for the desired quality, you’ll get alerted and suggestions will be made on how to fix it.
Data Engine
Fastest way to train your machine learning models.
super.AI’s Data Engine makes efficient use of your data by sampling only the most important parts, and finding data similar to it.
Security
Industry standard enterprise security and data privacy.
ISO 9001 and ISO 27001 certification for information security management, GDPR compliant, in controlled and monitored facilities.
PCI DSS Level 1
Payment Card Standards
CSA
Cloud Security Alliance Controls
SOC 1 / 2 / 3
Audit Controls Report / Security, Availability, & Confidentiality Report / General Controls Report
ISO 9001 / 27001 / 27017 / 27018
Global Quality Standard / Security Management Controls / Cloud Specific Controls / Personal Data Protection
HIPPA
Protected Health Information
GDPR
Personal Data Protection
Meet some of our team
We experienced AI in different ways in our careers, we came together to allow companies of all sizes to use AI.
Brad Cordova
.png)
Lars Wulfken
.png)
Enrique Garcia
.png)
Adriana Puchianu
Lisa Torlina
Purnawirman
Herry Sutanto
Michael Lee
INVESTORS
Backed by amazing investors
We are fortunate to have an exceptional set of investors backing us. Chances are you use the products they help build.
Toby Coppel
Mosaic Ventures
Julie Sandler
Pioneer Square Labs
Grant Ries
Liveramp
Old & New
A new programming paradigm for AI
Making access to AI available to everyone
50 years ago
Only the smartest people in the world could use computers 50 years ago. They were expensive and could only be programmed in binary. Over time people built abstractions: assembly, compiled languages, interpreted languages, GUI. Now almost anyone can use a computer. This changed the world.
Today
Only the smartest and most well funded companies can effectively use AI. Just like 50 years ago, when computers only understood binary, today's AI only understands labeled data. At super.AI, we built abstractions on top of labeled data, AI assembly language, AI compiler and Data Programming.
Vision
But we aren't even close to being done. We made big steps with the AI Compiler and Data Programming, but our mission is to ultimately make AI available for everyone.
Way
way
Old way
New way
Old way
New Way
Feature
Feature Resources
Over time, more and more data gets routed to AI.
Inside Canotic
