Automated QA, Review & Honeypots

In CVAT, it’s possible to evaluate the quality of annotation through the creation of a validation subset of images. To estimate the task quality, CVAT compares all other jobs in the task against the established Ground truth job, and calculates annotation quality based on this comparison.

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish (“Immediate feedback”)
• Review mode for problems found
• Quality analytics

Basics

There are several approaches to quality estimation used in the industry. In CVAT, we can use a method known as Ground Truth or Honeypots. The method assumes there are Ground Truth annotations for images in the dataset. This method is statistical, which means that we can use only a small portion of the whole dataset to estimate quality on the full dataset, so we don’t need to annotate the whole dataset twice. Here we assume that the images in the dataset are similar (represent the same task).

We will call the validation portion of the whole dataset (or a task in CVAT) a validation set. In practice, it is typically expected that annotations in the validation set are carefully validated and curated. It means that they are more expensive - creating them might require expert annotators or just several iterations of annotation and validation. It means that it’s desirable to keep the validation set small enough. At the same time, it must be representative enough to provide reliable estimations. To achieve this, it’s advised that the validation set images are sampled randomly and independently from the full dataset. That is, for the quality assurance to function correctly, the validation set must have some portion of the task frames, and the frames must be chosen randomly.

Depending on the dataset size, data variance, and task complexity, 5-15% of the data is typically good enough for quality estimation, while keeping extra annotation overhead for the Ground Truth acceptable.

For example, in a typical task with 2000 frames, selecting just 5%, which is 100 extra frames to annotate, is enough to estimate the annotation quality. If the task contains only 30 frames, it’s advisable to select 8-10 frames, which is about 30%. It is more than 15%, but in the case of smaller datasets, we need more samples to estimate quality reliably, as data variance is higher.

Ground truth jobs

A Ground Truth job (GT job) is a way to represent the validation set in a CVAT task. This job is similar to regular annotation jobs - you can edit the annotations manually, use auto-annotation features, and import annotations in this job. There can be no more than 1 Ground Truth job in a task.

To enable quality estimation in a task, you need to create a Ground truth job in the task, annotate it, switch the job stage to acceptance, and set the job state to completed. Once the Ground Truth job is configured, CVAT will start using this job for quality estimation.

Read more about Ground Truth management here.

Configuring quality estimation

There are 2 key components related to quality estimation configuration: Ground Truth jobs and quality requirements. Ground Truth jobs are configured at the Task level. In this section, we explain how to set up a Ground Truth job. Read more about quality settings here.

A configured Ground Truth job is required for all quality computations in CVAT.

• In a new task
• In an existing task

For already existing tasks only the Ground Truth validation mode is available. If you want

Validation modes

Currently, there are 2 validation modes available for tasks: Ground Truth and Honeypots. These names are often used interchangeably, but in CVAT they have some differences. Both modes rely on the use of Ground Truth annotations in a task, stored in a Ground Truth job, where they can be managed.

Ground Truth

In this mode some of the task frames are selected into the validation set, represented as a separate Ground Truth job. The regular annotation jobs in the task are not affected in any way.

Ground Truth jobs can be created at the task creation automatically or manually at any moment later. They can also be removed manually at any moment. This validation mode is available for any tasks and annotations.

This is a flexible mode that can be enabled or disabled at any moment without any disruptions to the annotation process.

Frame selection

This validation mode can use several frame selection methods.

Random

This is a simple method that selects frames into the validation set randomly, representing the basic approach, described above.

Parameters:

• frame count - the number or percent of the task frames to be used for validation. Can be specified as an absolute number in the Frame count field or a percent in the Quantity field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Random per job

This method selects frames into the validation set randomly from each annotation job in the task.

It solves one of the issues with the simple Random method that some of the jobs can get no validation frames, which makes it impossible to estimate quality in such jobs. Note that using this method can result in increased total size of the validation set.

Parameters:

• frame count per job - the percent of the job frames to be used for validation. This method uses segment size of the task to select the same number of validation frames in each job, if possible. Can be specified as an absolute number in the Frame count field or a percent in the Quantity per job field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Honeypots

In this mode some random frames of the task are selected into the validation set. Then, validation frames are randomly mixed into regular annotation jobs. This mode can also be called “Ground Truth pool”, reflecting the way validation frames are used. This mode can only be used at task creation and cannot be changed later.

The mode has some limitations on the compatible tasks:

• It’s not possible to use it for an already existing task, the task has to be recreated.
• This mode assumes random frame ordering, so it is only available for image annotation tasks and not for ordered sequences like videos.
• Tracks are not supported in such tasks.

The validation set can be managed after the task is created - annotations can be edited, frames can be excluded and restored, and honeypot frames in the regular jobs can be changed. However, it’s not possible to select new validation frames after the task is created. The Ground truth job created for this validation mode cannot be deleted.

Parameters:

• frame count per job (%) - the percent of job frames (segment size) to be added into each annotation job from the validation set. Can be specified in the Overhead per job field.
• total frame count (%) - the percent of the task frames to be included into the validation set. This value must result in at least frame count per job * segment size frames. Can be specified in the Total honeypots field.

Mode summary

Here is a brief comparison of the validation modes:

Choosing the right mode

Here are some examples on how to choose between these options. The general advice is to use Ground Truth for better flexibility, but keep in mind that it can require more resources for validation set annotation. Honeypots, on the other hand, can be beneficial if you want to minimize the number of validation images required, but the downside here is that there are some limitations on where this mode can be used.

Example: a video annotation with tracks. In this case there is only 1 option - the Ground Truth mode, so just use it.

Example: an image dataset annotation, image order is not important. Here you can use both options. You can choose Ground Truth for better flexibility in validation. This way, you will have the full control of validation frames in the task, annotation options won’t be limited, and the regular jobs will not be affected in any way. However, if you have a limited budget for the validation (for instance, you have only a small number of validation frames) or you want to allow more scalability (with this approach the number of validation frames doesn’t depend on the number of regular annotation jobs), it makes sense to consider using Honeypots instead.

Quality management

If a task has a validation configured, there are several options to manage validation set images. With any of the validation modes, there will be a special Ground Truth (GT) job in the task.

Validation set management

Validation frames can be managed on the task Quality Management page. Here it’s possible to check the number of validation frames, current validation mode and review the frame details. For each frame you can see the number of uses in the task. When in the Ground Truth mode, this number will be 1 for all frames. With Honeypots, these numbers can be 0, 1 or more.

Frame changes

In both validation modes it’s possible to exclude some of the validation frames from being used for validation. This can be useful if you find that some of the validation frames are “bad”, extra, or if they have incorrect annotations, which you don’t want to fix. Once a frame is marked “excluded”, it will not be used for validation. There is also an option to restore a previously excluded frame if you decide so.

There is an option to exclude or restore frames in bulk mode. To use it, select the frames needed using checkboxes, and click one of the buttons next to the table header.

Ground Truth job management

In the Ground Truth validation mode, there will be an option to remove the Ground Truth job from the task. It can be useful if you want to change validation set frames completely, add more frames, or remove some of the frames for any reason. This is available in the job Actions menu.

In the Honeypots mode, it’s not possible to add or remove the GT job, so it’s not possible to add more validation frames.

Create

A Ground Truth job can be added manually in a task without a selected validation mode or in a task with the Ground Truth validation mode, after the existing Ground Truth job is deleted manually.

Delete

To delete the Ground Truth job, do the following:

1. Open the task and find the Ground Truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Delete.

Import annotations

If you want to import annotations into the Ground truth job, do the following:

1. Open the task and find the Ground truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Import annotations.
4. Select import format and select file.
5. Click OK.

Export annotations

To export annotations from the Ground Truth job, do the following:

1. Open the task and find a job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Export annotations.

Annotation management

Annotations for validation frames can be displayed and edited in a special Ground Truth job in the task. You can edit the annotations manually, use auto-annotation features, import and export annotations in this job.

In the Ground Truth task validation mode, annotations of the ground Truth job do not affect other jobs in any way. The Ground Truth job is just a separate job, which can only be changed directly. Annotations from Ground truth jobs are not included in the dataset export, they also cannot be imported during task annotations import or with automatic annotation for the task.

In the Honeypots task validation mode, the annotations of the GT job also do not affect other jobs in any way. However, import and export of task annotations works differently. When importing task annotations, annotations for validation frames will be copied both into GT job frames and into corresponding honeypot frames in annotation jobs. When exporting task annotations, honeypot frames in annotation jobs will be ignored, and validation frames in the resulting dataset will get annotations from the GT job.

Import and export of Ground Truth job annotations works the same way in both modes.

Ground Truth jobs are included in task backups, so can be saved and restored this way.

Import, Export, and Delete options are available from the Ground Truth job Actions menu. Read more.

Annotation quality settings

Quality settings can be set up at the task or project level. If a task is not bound to a project, it uses its own settings. Tasks inside a project can use individual settings or inherit the project settings. Read more about project settings here.

To configure them, open a task or project Actions menu, select Quality control, and open the Settings tab.

The Settings tab has the following sections:

• General contains the Job selection filter. Only matching annotation and consensus-replica jobs are included in the next quality report.
• Job validation contains Max validations per job. A value above zero enables Immediate feedback for the task.
• Requirements configuration defines the criteria used to evaluate annotations.

Requirements configuration

Each enabled quality requirement has its own annotation type, target metric, threshold, filter, and comparison options. CVAT provides non-removable base requirements for supported annotation types. You can rename a base requirement, change its settings, or disable it.

The table shows the requirement hierarchy, target annotation type, metric, threshold, and enabled state. Use the arrow beside a row to expand its child requirements. The actions in each row let you create a child rule, edit the rule, and, for custom rules, copy or delete it.

Root requirements and custom rules

Root requirements are at the top of the hierarchy. CVAT creates one for every supported annotation type, and they cannot be deleted. They are useful as broad defaults, such as “all rectangles must reach 70% accuracy”. Disable a root requirement when that annotation type must not participate in the report or immediate feedback.

Use + button on a root or custom requirement to add a child. The parent and target annotation type are fixed for a child rule. Give the child a descriptive name and use its filter and comparison options to define a narrower check. For example, a root rectangle requirement can have a child rule for the vehicle label with a higher score threshold. Custom requirements can be copied when you need a similar rule and deleted when no longer needed.

Inheritance

Children inherit their target annotation type, target metric, threshold, comparison options, and attribute-comparison configuration from their parent. This avoids repeating the same comparison policy for every label or subset of annotations.

Change a value in a parent to update the effective value for descendants that have not overridden it. A child can override its metric, threshold, and applicable comparison options when it needs a more specific policy. Overridden fields show a revert control; use it to return to the parent value. A child requirement always keeps its parent and annotation type.

Annotation filters

The Filter control selects the annotations evaluated by a requirement. Use it to narrow a rule by label, annotation type, area, source, visibility, track properties, or label-specific attributes. Combine rules in the filter builder to describe the subset that the requirement must evaluate.

Parent filters are displayed alongside the current filter. They are always applied to child requirements, so a child can only narrow its parent’s scope. For example, a parent filtered to vehicle annotations can have a child filtered to vehicle annotations with a specific attribute value; the child cannot include annotations outside the parent’s filter.

Attribute rules

Use Attribute comparison when matching annotations must also verify attribute values. Enable Match unspecified attributes exactly to compare every attribute without a custom rule using the Exact comparator. This is useful when all attributes are part of the expected annotation result.

Add an Attribute rule when one attribute needs different handling. Select the attribute, enable or disable its comparison, then choose either Exact or Levenshtein. The Levenshtein comparator accepts similar text values and exposes a normalized similarity threshold from 0 to 1. Attribute rules, like the rest of the comparison configuration, can be inherited and overridden by child requirements.

Comparison options

When editing a requirement, set its Target metric and Target metric threshold to define its completion condition. The remaining form sections configure comparison behavior for shapes, groups, object visibility, points and skeletons, polylines, and attributes. Inherited controls can be overridden or reverted to the parent value. Hover the ? icon next to a control for a detailed description.

The available controls depend on the requirement target:

Save the settings after editing them. Changes apply when the next quality report is calculated; they do not alter existing reports.

Project quality settings

In CVAT, it is possible to group tasks into projects to share common configurations or establish a logical grouping for datasets. In this section, we explain options for quality management inside projects.

When tasks are inside a project, it can be convenient to reuse the same quality setup for all the project tasks. There is an option to use quality settings from the parent project for all or only for specific tasks inside a project. This is controlled by the corresponding toggle on the Quality control page of the task:

If some of the project tasks have individual settings, a notification is displayed on the project settings page. You can enforce project settings for all the project tasks by clicking the Force project settings button.

By default, new tasks inside a project inherit the quality settings of the project. You can freely switch between these 2 modes without losing the individual configuration for the task.

Updating project quality includes quality computation for all the nested tasks. If the task quality is updated in a specific task manually, project quality has to be recomputed to display relevant values.

Job filtering

Depending on the situation, you may need or don’t need to include specific jobs in the quality report. For example, your workflow may require quality checks only for completed jobs or maybe you want to exclude jobs from a specific task from a project quality report. There is an option to filter which jobs are included in quality computations. You can configure this by changing the Job selection filter in quality settings. Only jobs matching the filter criteria will be included in the quality results. If a filter is changed, quality must be recomputed for the filter to take effect.

Comparisons

Tags

The equality is used for matching.

Shapes

A pair of shapes is considered matching, if both their shapes and labels match. For each shape, spatial parameters are matched first, then labels are matched.

Each shape type can have their own spatial matching details. Specifically:

• bounding box - IoU (including rotation). For example, for a pair of bounding boxes it can be visualized this way:

  

  
  IoU = intersection area / union area.
  The green part is the intersection, and green, yellow and red ones together are the union.

• polygons, masks - IoU. Polygons and masks are considered interchangeable, which means a mask can be matched with a polygon and vice versa. Polygons and masks in groups are merged into a single object first. If Panoptic comparison is enabled, objects will be cut to only the visible (non-covered) parts only, which is determined by the shape z order.

• skeletons - The OKS metric from the COCO dataset is used. Briefly, each skeleton point gets a circular area around, determined by the object size (bounding box side) and Point size values, where this point can be matched with the specified probability. If a bounding box is grouped with the skeleton, it is used for object size computation, otherwise a bounding box of visible points of the skeleton is used.

  For example, consider a skeleton with 6 points and a square bounding box attached:

  

  In this example, the Point size parameter is 0.05 (5%) of the bbox side. Areas shown in the green color cover ~68.2% (1 sigma) of the points, corresponding to each GT point. A point on the boundary of such an area will have ~88% of probability to be correct. The blue-colored zone contains ~95% (2 sigma) of the correct points for the corresponding GT point. A point on the boundary of such an area will have ~60% of probability to be correct. These probabilities are then averaged over the visible points of the skeleton to calculate their similarity. The point-size value corresponds to one standard deviation in the normal distribution.

• points - The OKS metric is used for each point group annotation. Same as for skeletons, Point size determines relative point sizes. The Point size base setting allows configuring whether points in point groups should use the group bounding box or the image space. Using image space for object size can be useful if you want to treat each point as a separate annotation.

• polylines - A pair of lines is considered matching if all the points of one line lie within a “hull” of the other one. The “hull” is determined as the area around the polyline, such as if the line had some “thickness”. For example, the black polyline can have a hull shown in the green color:

  

  The line thickness can be configured via the Line thickness setting. The value is relative to the image side and determines a half of the hull width.

• ellipses - IoU, described in more detail above.

Tracks

Tracks are split into separate shapes and compared on the per-frame basis with other tracks and shapes.

Quality Analytics

Once quality estimation is enabled in a task and its Ground Truth job is configured, quality reports are available for the task and its jobs. Projects can also have an aggregated report for their tasks and jobs.

Open the task or project Actions menu > Quality control and select the Requirements tab. Select Calculate a new quality report. If no report has been calculated yet, the page shows a prompt instead of an empty table. The date of the most recent report and a download action appear after the report is available.

Analytics page contents

The Requirements tab shows every enabled requirement, its target metric, score, and whether it meets the threshold. A score of N/A with a warning icon means CVAT could not calculate that requirement, for example because there were no applicable annotations. Hover the icon to see the reason. A requirement that cannot be calculated is not treated as failed.

Use the confusion-matrix action next to a requirement to inspect its results, select another requirement, or download the selected matrix as CSV or JSON.

The Jobs tab lists evaluated jobs and their completion rate. It shows the visible requirement scores; use the table controls to reveal other requirement columns, filter, sort, or download the data. In a project, the Tasks tab provides the same requirement-oriented overview for tasks. Use the arrow action in a task or requirement row to open the related filtered list of jobs.

For tasks with a configured Ground Truth job, the Management tab controls validation frames and the Ground Truth job. The Settings tab is available for tasks and projects whether or not a report has been calculated.

Problem Reporting

All currently reported conflicts have error severity.

Quality Reports

Use Download in the report header to download the task or project quality report in JSON format. Each job row also provides a download action for its report. These files are useful for processing quality results in scripts.

Quality reports contain the requirement scores, metrics, and conflicts shown in the UI, along with detailed per-label and per-frame results.

Additional information on how to compute and use various metrics for dataset quality estimation is available here.

Reviewing GT conflicts

To see GT Conflicts in the CVAT interface, go to Review > Issues > Show ground truth annotations and conflicts.

Ground Truth annotations are displayed with a dotted-line border. The associated label and the (Ground Truth) marker are shown on hovering.

Upon hovering over an issue on the right-side panel with your mouse, the corresponding annotations are highlighted.

Use arrows in the Issue toolbar to move between GT conflicts.

To create an issue related to the conflict, right-click on the bounding box and from the menu select the type of issue you want to create.

Annotation quality & Honeypot video tutorial

This video demonstrates the process: