AnnotatePlus has the following features for automated quality control of annotations:
- Validation set configuration for a task
- Job validation on job finish ("Immediate feedback")
- Review mode for problems found
- Quality analytics
In this section we only highlight the key steps in quality estimation. Read the detailed guide on quality estimation in AnnotatePlus in the Advanced section.
How to enable quality control
- Go to task creation
- Select the source media, configure other task parameters
- Scroll down to the Quality Control section
Select one of the
validation modes
available
- Create the task
- Upload or create Ground Truth annotations in the Ground Truth job in the task
- Switch the Ground Truth job into the
acceptancestage andcompletedstate 
How to enable immediate job feedback
Note: This feature requires a configured validation set in the task. Read more in How to enable quality control and in the full guide.
- Open the task Actions menu > Quality control > Settings
- Set Max validations per job to above zero. 3 is a good starting number.
- Save the updated settings
- Assign an annotator to an annotation job
- Annotate the job
- Mark the job finished using the corresponding button in the menu
- Once the job is completed, you'll see the job validation dialog
Each assignee gets no more than the specified number of validation attempts.
Read more about this functionality in the Immediate Feedback section.
How to check task quality metrics
- Open the task Actions menu > Quality control
- (optional) Request quality metrics computation, wait for completion
- Review summaries or detailed reports
Read more about this functionality here.
How to review problems found
- Open the task Actions menu > Quality control
- Find an annotation job to be reviewed, it must have at least 1 validation frame
- Click the job link
- Switch to the Review mode
- Enable display of Ground Truth annotations and conflicts
Read more about this functionality here.