Quality control

Overview of quality control features

CVAT has the following features for automated quality control of annotations:

In this section we only highlight the key steps in quality estimation. Read the detailed guide on quality estimation in CVAT in the Advanced section.

How to enable quality control

  1. Go to task creation
  2. Select the source media, configure other task parameters
  3. Scroll down to the Quality Control section
  4. Select one of the validation modes available

Create task with validation mode

  1. Create the task
  2. Upload or create Ground Truth annotations in the Ground Truth job in the task
  3. Switch the Ground Truth job into the acceptance stage and completed state

Set job status

For already existing tasks only the Ground Truth validation mode is available. If you want to use Honeypots for your task, you will need to recreate the task.

  1. Open the task page
  2. Click the + button next to the job list

Create job

  1. Select Job Type Ground truth and configure the job parameters

Configure job parameters

  1. Upload or create Ground Truth annotations in the Ground Truth job in the task
  2. Switch the Ground Truth job into the acceptancestage and completed state

Set job status

How to enable immediate job feedback

Note: This feature requires a configured validation set in the task. Read more in How to enable quality control and in the full guide.

  1. Open the task Actions menu > Quality control > Settings
  2. Set Max validations per job to above zero. 3 is a good starting number.

Configure job validations

  1. Save the updated settings
  2. Assign an annotator to an annotation job
  3. Annotate the job
  4. Mark the job finished using the corresponding button in the menu
  5. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Read more about this functionality in the Immediate Feedback section.

How to check task quality metrics

  1. Open the task Actions menu > Quality control
  2. (optional) Request quality metrics computation, wait for completion
  3. Review summaries or detailed reports

Quality Analytics page

Read more about this functionality here.

How to review problems found

  1. Open the task Actions menu > Quality control
  2. Find an annotation job to be reviewed, it must have at least 1 validation frame
  3. Click the job link
  4. Switch to the Review mode
  5. Enable display of Ground Truth annotations and conflicts

GT conflict

Read more about this functionality here.