Quality control

Overview of quality control features

CVAT has the following features for automated quality control of annotations:

In this section, we highlight only the key steps in quality estimation. Consult the detailed guide on quality estimation in CVAT in the Advanced section.

How to enable quality control

  1. Go to task creation

  2. Select the source media, configure other task parameters

  3. Scroll down to the Quality Control section

  4. Select one of the validation modes available

    Create task with validation mode

  5. Create the task

  6. Upload or create Ground Truth annotations in the Ground Truth job in the task

  7. Switch the Ground Truth job into the acceptance stage and completed state

Set job status

For already existing tasks only the Ground Truth validation mode is available. If you want to use Honeypots for your task, you will need to recreate the task.

  1. Open the task page

  2. Select the + button next to the job list

    Create job

  3. Select Job Type Ground truth and configure the job parameters

    Configure job parameters

  4. Upload or create Ground Truth annotations in the Ground Truth job in the task

  5. Switch the Ground Truth job into the acceptancestage and completed state

    Set job status

How to enable immediate job feedback

Note: This feature requires a configured validation set in the task. Learn more in How to enable quality control and in the full guide.

  1. Open the task Actions menu > Quality control > Settings

  2. Set Max validations per job to above zero. 3 is a good starting number

    Configure job validations

  3. Save the updated settings

  4. Assign an annotator to an annotation job

  5. Annotate the job

  6. Mark the job finished using the corresponding button in the menu

  7. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Learn more about this functionality in the Immediate Feedback section.

How to check task quality metrics

  1. Open the task Actions menu > Quality control

  2. (Optional) Request quality metrics computation, and wait for completion

  3. Review summaries or detailed reports

    Quality Analytics page

Learn more about this functionality here.

How to review problems found

  1. Open the task Actions menu > Quality control
  2. Find an annotation job to be reviewed, it must have at least 1 validation frame
  3. Select the job link
  4. Switch to the Review mode
  5. Enable display of Ground Truth annotations and conflicts

GT conflict

Learn more about this functionality here.