QA & Analytics

• 1: Quality control
• 2: Manual QA and Review
• 3: Automated QA, Review & Honeypots
• 4: Consensus-based annotation
• 5: Immediate job feedback
• 6: Analytics

1 - Quality control

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish (Immediate feedback)
• Review mode for problems found
• Quality analytics

In this section, we highlight only the key steps in quality estimation. Consult the detailed guide on quality estimation in CVAT in the Advanced section.

How to enable quality control

• In a new task
• In an existing task

For already existing tasks only the Ground Truth validation mode is available. If you want

How to enable immediate job feedback

1. Open the task Actions menu > Quality control > Settings

2. Set Max validations per job to above zero. 3 is a good starting number

   

3. Save the updated settings

4. Assign an annotator to an annotation job

5. Annotate the job

6. Mark the job finished using the corresponding button in the menu

7. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Learn more about this functionality in the Immediate Feedback section.

How to check task quality metrics

1. Open the task Actions menu > Quality control

2. (Optional) Request quality metrics computation, and wait for completion

3. Review summaries or detailed reports

   

Learn more about this functionality here.

How to review problems found

1. Open the task Actions menu > Quality control
2. Find an annotation job to be reviewed, it must have at least 1 validation frame
3. Select the job link
4. Switch to the Review mode
5. Enable display of Ground Truth annotations and conflicts

Learn more about this functionality here.

2 - Manual QA and Review

In the demanding process of annotation, ensuring accuracy is paramount.

CVAT introduces a specialized Review mode, designed to streamline the validation of annotations by pinpointing errors or discrepancies in annotation.

See:

• Review and report issues: review only mode
  • Assigning reviewer
  • Reporting issues
  • Quick issue
  • Assigning corrector
  • Correcting reported issues
• Review and report issues: review and correct mode
• Issues navigation and interface
  • Issues tab
  • Issues workspace
  • Issues comments
• Manual QA complete video tutorial

Review and report issues: review only mode

Review mode is a user interface (UI) setting where a specialized Issue tool is available. This tool allows you to identify and describe issues with objects or areas within the frame.

Review mode screen looks like the following:

Assigning reviewer

To assign a reviewer to the job, do the following:

1. Log in to the Owner or Maintainer account.

2. (Optional) If the person you wish to assign as a reviewer is not a member of Organization, you need to Invite this person to the Organization.

3. Click on the Assignee field and select the reviewer.

4. From the Stage drop-down list, select Validation.

   

Reporting issues

To report an issue, do the following:

1. Log in to the reviewer’s account.

2. On the Controls sidebar, click Open and issue ().

3. Click on the area of the frame where the issue is occurring, and the Issue report popup will appear.

   

4. In the text field of the Issue report popup, enter the issue description.

5. Click Submit.

Quick issue

The Quick issue function streamlines the review process. It allows reviewers to efficiently select from a list of previously created issues or add a new one, facilitating a faster and more organized review.

To create a Quick issue do the following:

1. Right-click on the area of the frame where the issue is occurring.

2. From the popup menu select one of the following:

   • Open an issue…: to create new issue.
   • Quick issue: incorrect position: to report incorrect position of the label.
   • Quick issue: incorrect attribute: to report incorrect attribute of the label.
   • Quick issue…: to open the list of issues that were reported by you before.

Assigning corrector

To assign a corrector to the job, do the following:

1. Log in to the Owner or Maintainer account.

2. (Optional) If the person you wish to assign as a corrector is not a member of Organization, you need to Invite this person to the Organization.

3. Click on the Assignee field and select the reviewer.

4. From the Stage drop-down list, select Annotation.

   

Correcting reported issues

To correct the reported issue, do the following:

1. Log in to the corrector account.

2. Go to the reviewed job and open it.

3. Click on the issue report, to see details of what needs to be corrected.

   

4. Correct annotation.

5. Add a comment to the issue report and click Resolve.

   

6. After all issues are fixed save work, go to the Menu select the Change the job state and change state to Complete.

   

Review and report issues: review and correct mode

The person, assigned as assigned as reviewer can switch to correction mode and correct all annotation issues.

To correct annotation issues as a reviewer, do the following:

1. Log in to the reviewer account.

2. Go to the assigned job and open it.

3. In the top right corner, from the drop-down list, select Standard.

   

Issues navigation and interface

This section describes navigation, interface and comments section.

Issues tab

The created issue will appear on the Objects sidebar, in the Issues tab.

It has has the following elements:

Issues workspace

In the workspace, you can click on the issue, and add a comment on the issue, remove (Remove) it, or resolve (Resolve) it.

To reopen the resolved issue, click Reopen.

You can easily access multiple issues created in one location by hovering over an issue and scrolling the mouse wheel.

Issues comments

You can add as many comments as needed to the issue.

In the Objects toolbar, only the first and last comments will be displayed

You can copy and paste comments text.

Manual QA complete video tutorial

This video demonstrates the process:

3 - Automated QA, Review & Honeypots

In CVAT, it’s possible to evaluate the quality of annotation through the creation of a validation subset of images. To estimate the task quality, CVAT compares all other jobs in the task against the established Ground truth job, and calculates annotation quality based on this comparison.

CVAT has the following features for automated quality control of annotations:

• Validation set configuration for a task
• Job validation on job finish (“Immediate feedback”)
• Review mode for problems found
• Quality analytics

Basics

There are several approaches to quality estimation used in the industry. In CVAT, we can use a method known as Ground Truth or Honeypots. The method assumes there are Ground Truth annotations for images in the dataset. This method is statistical, which means that we can use only a small portion of the whole dataset to estimate quality on the full dataset, so we don’t need to annotate the whole dataset twice. Here we assume that the images in the dataset are similar (represent the same task).

We will call the validation portion of the whole dataset (or a task in CVAT) a validation set. In practice, it is typically expected that annotations in the validation set are carefully validated and curated. It means that they are more expensive - creating them might require expert annotators or just several iterations of annotation and validation. It means that it’s desirable to keep the validation set small enough. At the same time, it must be representative enough to provide reliable estimations. To achieve this, it’s advised that the validation set images are sampled randomly and independently from the full dataset. That is, for the quality assurance to function correctly, the validation set must have some portion of the task frames, and the frames must be chosen randomly.

Depending on the dataset size, data variance, and task complexity, 5-15% of the data is typically good enough for quality estimation, while keeping extra annotation overhead for the Ground Truth acceptable.

For example, in a typical task with 2000 frames, selecting just 5%, which is 100 extra frames to annotate, is enough to estimate the annotation quality. If the task contains only 30 frames, it’s advisable to select 8-10 frames, which is about 30%. It is more than 15%, but in the case of smaller datasets, we need more samples to estimate quality reliably, as data variance is higher.

Ground truth jobs

A Ground Truth job (GT job) is a way to represent the validation set in a CVAT task. This job is similar to regular annotation jobs - you can edit the annotations manually, use auto-annotation features, and import annotations in this job. There can be no more than 1 Ground Truth job in a task.

To enable quality estimation in a task, you need to create a Ground truth job in the task, annotate it, switch the job stage to acceptance, and set the job state to completed. Once the Ground Truth job is configured, CVAT will start using this job for quality estimation.

Read more about Ground Truth management here.

Configuring quality estimation

There are 2 key components related to quality estimation configuration: Ground Truth jobs and quality requirements. Ground Truth jobs are configured at the Task level. In this section, we explain how to set up a Ground Truth job. Read more about quality settings here.

A configured Ground Truth job is required for all quality computations in CVAT.

• In a new task
• In an existing task

For already existing tasks only the Ground Truth validation mode is available. If you want

Validation modes

Currently, there are 2 validation modes available for tasks: Ground Truth and Honeypots. These names are often used interchangeably, but in CVAT they have some differences. Both modes rely on the use of Ground Truth annotations in a task, stored in a Ground Truth job, where they can be managed.

Ground Truth

In this mode some of the task frames are selected into the validation set, represented as a separate Ground Truth job. The regular annotation jobs in the task are not affected in any way.

Ground Truth jobs can be created at the task creation automatically or manually at any moment later. They can also be removed manually at any moment. This validation mode is available for any tasks and annotations.

This is a flexible mode that can be enabled or disabled at any moment without any disruptions to the annotation process.

Frame selection

This validation mode can use several frame selection methods.

Random

This is a simple method that selects frames into the validation set randomly, representing the basic approach, described above.

Parameters:

• frame count - the number or percent of the task frames to be used for validation. Can be specified as an absolute number in the Frame count field or a percent in the Quantity field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Random per job

This method selects frames into the validation set randomly from each annotation job in the task.

It solves one of the issues with the simple Random method that some of the jobs can get no validation frames, which makes it impossible to estimate quality in such jobs. Note that using this method can result in increased total size of the validation set.

Parameters:

• frame count per job - the percent of the job frames to be used for validation. This method uses segment size of the task to select the same number of validation frames in each job, if possible. Can be specified as an absolute number in the Frame count field or a percent in the Quantity per job field. If there are both fields on the page, they are linked, which means changing one of them will adjust the other one automatically.
• random seed - a number to be used to initialize the random number generator. Can be useful if you want to create a reproducible sequence of frames.

Honeypots

In this mode some random frames of the task are selected into the validation set. Then, validation frames are randomly mixed into regular annotation jobs. This mode can also be called “Ground Truth pool”, reflecting the way validation frames are used. This mode can only be used at task creation and cannot be changed later.

The mode has some limitations on the compatible tasks:

• It’s not possible to use it for an already existing task, the task has to be recreated.
• This mode assumes random frame ordering, so it is only available for image annotation tasks and not for ordered sequences like videos.
• Tracks are not supported in such tasks.

The validation set can be managed after the task is created - annotations can be edited, frames can be excluded and restored, and honeypot frames in the regular jobs can be changed. However, it’s not possible to select new validation frames after the task is created. The Ground truth job created for this validation mode cannot be deleted.

Parameters:

• frame count per job (%) - the percent of job frames (segment size) to be added into each annotation job from the validation set. Can be specified in the Overhead per job field.
• total frame count (%) - the percent of the task frames to be included into the validation set. This value must result in at least frame count per job * segment size frames. Can be specified in the Total honeypots field.

Mode summary

Here is a brief comparison of the validation modes:

Choosing the right mode

Here are some examples on how to choose between these options. The general advice is to use Ground Truth for better flexibility, but keep in mind that it can require more resources for validation set annotation. Honeypots, on the other hand, can be beneficial if you want to minimize the number of validation images required, but the downside here is that there are some limitations on where this mode can be used.

Example: a video annotation with tracks. In this case there is only 1 option - the Ground Truth mode, so just use it.

Example: an image dataset annotation, image order is not important. Here you can use both options. You can choose Ground Truth for better flexibility in validation. This way, you will have the full control of validation frames in the task, annotation options won’t be limited, and the regular jobs will not be affected in any way. However, if you have a limited budget for the validation (for instance, you have only a small number of validation frames) or you want to allow more scalability (with this approach the number of validation frames doesn’t depend on the number of regular annotation jobs), it makes sense to consider using Honeypots instead.

Quality management

If a task has a validation configured, there are several options to manage validation set images. With any of the validation modes, there will be a special Ground Truth (GT) job in the task.

Validation set management

Validation frames can be managed on the task Quality Management page. Here it’s possible to check the number of validation frames, current validation mode and review the frame details. For each frame you can see the number of uses in the task. When in the Ground Truth mode, this number will be 1 for all frames. With Honeypots, these numbers can be 0, 1 or more.

Frame changes

In both validation modes it’s possible to exclude some of the validation frames from being used for validation. This can be useful if you find that some of the validation frames are “bad”, extra, or if they have incorrect annotations, which you don’t want to fix. Once a frame is marked “excluded”, it will not be used for validation. There is also an option to restore a previously excluded frame if you decide so.

There is an option to exclude or restore frames in bulk mode. To use it, select the frames needed using checkboxes, and click one of the buttons next to the table header.

Ground Truth job management

In the Ground Truth validation mode, there will be an option to remove the Ground Truth job from the task. It can be useful if you want to change validation set frames completely, add more frames, or remove some of the frames for any reason. This is available in the job Actions menu.

In the Honeypots mode, it’s not possible to add or remove the GT job, so it’s not possible to add more validation frames.

Create

A Ground Truth job can be added manually in a task without a selected validation mode or in a task with the Ground Truth validation mode, after the existing Ground Truth job is deleted manually.

Delete

To delete the Ground Truth job, do the following:

1. Open the task and find the Ground Truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Delete.

Import annotations

If you want to import annotations into the Ground truth job, do the following:

1. Open the task and find the Ground truth job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Import annotations.
4. Select import format and select file.
5. Click OK.

Export annotations

To export annotations from the Ground Truth job, do the following:

1. Open the task and find a job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select Export annotations.

Annotation management

Annotations for validation frames can be displayed and edited in a special Ground Truth job in the task. You can edit the annotations manually, use auto-annotation features, import and export annotations in this job.

In the Ground Truth task validation mode, annotations of the ground Truth job do not affect other jobs in any way. The Ground Truth job is just a separate job, which can only be changed directly. Annotations from Ground truth jobs are not included in the dataset export, they also cannot be imported during task annotations import or with automatic annotation for the task.

In the Honeypots task validation mode, the annotations of the GT job also do not affect other jobs in any way. However, import and export of task annotations works differently. When importing task annotations, annotations for validation frames will be copied both into GT job frames and into corresponding honeypot frames in annotation jobs. When exporting task annotations, honeypot frames in annotation jobs will be ignored, and validation frames in the resulting dataset will get annotations from the GT job.

Import and export of Ground Truth job annotations works the same way in both modes.

Ground Truth jobs are included in task backups, so can be saved and restored this way.

Import, Export, and Delete options are available from the Ground Truth job Actions menu. Read more.

Annotation quality settings

Quality settings can be set up at the task or project level. If a task is not bound to a project, it uses its own settings. Tasks inside a project can use individual settings or inherit the project settings. Read more about project settings here.

To configure them, open a task or project Actions menu, select Quality control, and open the Settings tab.

The Settings tab has the following sections:

• General contains the Job selection filter. Only matching annotation and consensus-replica jobs are included in the next quality report.
• Job validation contains Max validations per job. A value above zero enables Immediate feedback for the task.
• Requirements configuration defines the criteria used to evaluate annotations.

Requirements configuration

Each enabled quality requirement has its own annotation type, target metric, threshold, filter, and comparison options. CVAT provides non-removable base requirements for supported annotation types. You can rename a base requirement, change its settings, or disable it.

The table shows the requirement hierarchy, target annotation type, metric, threshold, and enabled state. Use the arrow beside a row to expand its child requirements. The actions in each row let you create a child rule, edit the rule, and, for custom rules, copy or delete it.

Root requirements and custom rules

Root requirements are at the top of the hierarchy. CVAT creates one for every supported annotation type, and they cannot be deleted. They are useful as broad defaults, such as “all rectangles must reach 70% accuracy”. Disable a root requirement when that annotation type must not participate in the report or immediate feedback.

Use + button on a root or custom requirement to add a child. The parent and target annotation type are fixed for a child rule. Give the child a descriptive name and use its filter and comparison options to define a narrower check. For example, a root rectangle requirement can have a child rule for the vehicle label with a higher score threshold. Custom requirements can be copied when you need a similar rule and deleted when no longer needed.

Inheritance

Children inherit their target annotation type, target metric, threshold, comparison options, and attribute-comparison configuration from their parent. This avoids repeating the same comparison policy for every label or subset of annotations.

Change a value in a parent to update the effective value for descendants that have not overridden it. A child can override its metric, threshold, and applicable comparison options when it needs a more specific policy. Overridden fields show a revert control; use it to return to the parent value. A child requirement always keeps its parent and annotation type.

Annotation filters

The Filter control selects the annotations evaluated by a requirement. Use it to narrow a rule by label, annotation type, area, source, visibility, track properties, or label-specific attributes. Combine rules in the filter builder to describe the subset that the requirement must evaluate.

Parent filters are displayed alongside the current filter. They are always applied to child requirements, so a child can only narrow its parent’s scope. For example, a parent filtered to vehicle annotations can have a child filtered to vehicle annotations with a specific attribute value; the child cannot include annotations outside the parent’s filter.

Attribute rules

Use Attribute comparison when matching annotations must also verify attribute values. Enable Match unspecified attributes exactly to compare every attribute without a custom rule using the Exact comparator. This is useful when all attributes are part of the expected annotation result.

Add an Attribute rule when one attribute needs different handling. Select the attribute, enable or disable its comparison, then choose either Exact or Levenshtein. The Levenshtein comparator accepts similar text values and exposes a normalized similarity threshold from 0 to 1. Attribute rules, like the rest of the comparison configuration, can be inherited and overridden by child requirements.

Comparison options

When editing a requirement, set its Target metric and Target metric threshold to define its completion condition. The remaining form sections configure comparison behavior for shapes, groups, object visibility, points and skeletons, polylines, and attributes. Inherited controls can be overridden or reverted to the parent value. Hover the ? icon next to a control for a detailed description.

The available controls depend on the requirement target:

Save the settings after editing them. Changes apply when the next quality report is calculated; they do not alter existing reports.

Project quality settings

In CVAT, it is possible to group tasks into projects to share common configurations or establish a logical grouping for datasets. In this section, we explain options for quality management inside projects.

When tasks are inside a project, it can be convenient to reuse the same quality setup for all the project tasks. There is an option to use quality settings from the parent project for all or only for specific tasks inside a project. This is controlled by the corresponding toggle on the Quality control page of the task:

If some of the project tasks have individual settings, a notification is displayed on the project settings page. You can enforce project settings for all the project tasks by clicking the Force project settings button.

By default, new tasks inside a project inherit the quality settings of the project. You can freely switch between these 2 modes without losing the individual configuration for the task.

Updating project quality includes quality computation for all the nested tasks. If the task quality is updated in a specific task manually, project quality has to be recomputed to display relevant values.

Job filtering

Depending on the situation, you may need or don’t need to include specific jobs in the quality report. For example, your workflow may require quality checks only for completed jobs or maybe you want to exclude jobs from a specific task from a project quality report. There is an option to filter which jobs are included in quality computations. You can configure this by changing the Job selection filter in quality settings. Only jobs matching the filter criteria will be included in the quality results. If a filter is changed, quality must be recomputed for the filter to take effect.

Comparisons

Tags

The equality is used for matching.

Shapes

A pair of shapes is considered matching, if both their shapes and labels match. For each shape, spatial parameters are matched first, then labels are matched.

Each shape type can have their own spatial matching details. Specifically:

• bounding box - IoU (including rotation). For example, for a pair of bounding boxes it can be visualized this way:

  

  
  IoU = intersection area / union area.
  The green part is the intersection, and green, yellow and red ones together are the union.

• polygons, masks - IoU. Polygons and masks are considered interchangeable, which means a mask can be matched with a polygon and vice versa. Polygons and masks in groups are merged into a single object first. If Panoptic comparison is enabled, objects will be cut to only the visible (non-covered) parts only, which is determined by the shape z order.

• skeletons - The OKS metric from the COCO dataset is used. Briefly, each skeleton point gets a circular area around, determined by the object size (bounding box side) and Point size values, where this point can be matched with the specified probability. If a bounding box is grouped with the skeleton, it is used for object size computation, otherwise a bounding box of visible points of the skeleton is used.

  For example, consider a skeleton with 6 points and a square bounding box attached:

  

  In this example, the Point size parameter is 0.05 (5%) of the bbox side. Areas shown in the green color cover ~68.2% (1 sigma) of the points, corresponding to each GT point. A point on the boundary of such an area will have ~88% of probability to be correct. The blue-colored zone contains ~95% (2 sigma) of the correct points for the corresponding GT point. A point on the boundary of such an area will have ~60% of probability to be correct. These probabilities are then averaged over the visible points of the skeleton to calculate their similarity. The point-size value corresponds to one standard deviation in the normal distribution.

• points - The OKS metric is used for each point group annotation. Same as for skeletons, Point size determines relative point sizes. The Point size base setting allows configuring whether points in point groups should use the group bounding box or the image space. Using image space for object size can be useful if you want to treat each point as a separate annotation.

• polylines - A pair of lines is considered matching if all the points of one line lie within a “hull” of the other one. The “hull” is determined as the area around the polyline, such as if the line had some “thickness”. For example, the black polyline can have a hull shown in the green color:

  

  The line thickness can be configured via the Line thickness setting. The value is relative to the image side and determines a half of the hull width.

• ellipses - IoU, described in more detail above.

Tracks

Tracks are split into separate shapes and compared on the per-frame basis with other tracks and shapes.

Quality Analytics

Once quality estimation is enabled in a task and its Ground Truth job is configured, quality reports are available for the task and its jobs. Projects can also have an aggregated report for their tasks and jobs.

Open the task or project Actions menu > Quality control and select the Requirements tab. Select Calculate a new quality report. If no report has been calculated yet, the page shows a prompt instead of an empty table. The date of the most recent report and a download action appear after the report is available.

Analytics page contents

The Requirements tab shows every enabled requirement, its target metric, score, and whether it meets the threshold. A score of N/A with a warning icon means CVAT could not calculate that requirement, for example because there were no applicable annotations. Hover the icon to see the reason. A requirement that cannot be calculated is not treated as failed.

Use the confusion-matrix action next to a requirement to inspect its results, select another requirement, or download the selected matrix as CSV or JSON.

The Jobs tab lists evaluated jobs and their completion rate. It shows the visible requirement scores; use the table controls to reveal other requirement columns, filter, sort, or download the data. In a project, the Tasks tab provides the same requirement-oriented overview for tasks. Use the arrow action in a task or requirement row to open the related filtered list of jobs.

For tasks with a configured Ground Truth job, the Management tab controls validation frames and the Ground Truth job. The Settings tab is available for tasks and projects whether or not a report has been calculated.

Problem Reporting

All currently reported conflicts have error severity.

Quality Reports

Use Download in the report header to download the task or project quality report in JSON format. Each job row also provides a download action for its report. These files are useful for processing quality results in scripts.

Quality reports contain the requirement scores, metrics, and conflicts shown in the UI, along with detailed per-label and per-frame results.

Additional information on how to compute and use various metrics for dataset quality estimation is available here.

Reviewing GT conflicts

To see GT Conflicts in the CVAT interface, go to Review > Issues > Show ground truth annotations and conflicts.

Ground Truth annotations are displayed with a dotted-line border. The associated label and the (Ground Truth) marker are shown on hovering.

Upon hovering over an issue on the right-side panel with your mouse, the corresponding annotations are highlighted.

Use arrows in the Issue toolbar to move between GT conflicts.

To create an issue related to the conflict, right-click on the bounding box and from the menu select the type of issue you want to create.

Annotation quality & Honeypot video tutorial

This video demonstrates the process:

4 - Consensus-based annotation

With CVAT you can annotate the same data several times and then merge annotations automatically to obtain more reliable annotations.

CVAT has the following features related to consensus-based annotation:

• Creation of consensus replica jobs for regular annotation jobs in a task
• Automatic merging for annotations inside consensus replica jobs

Basics

If you want to improve the quality of your annotations, there are several widespread ways to achieve this. One of the methods is called consensus-based annotation or just consensus. In this method, the same data is annotated several times. Once there are several different annotations (“opinions”) for the same objects, they can be merged in order to obtain annotations of higher quality.

Let’s consider an explanatory example. Imagine there is a group of people and you want to learn whether something is true or not from them. To accomplish this, you decided to ask everyone and pick the most popular answer in the end. With such an idea, you can get many possible combinations of votes, including unanimous ones. This strategy is called majority voting - and it requires such a majority to exist. If there is an even number of votes for both options, you don’t have enough information to prefer one of the options to the other, so, in general, it’s desirable to have an odd number of people in the group. With larger groups the method becomes less sensitive to this requirement, as when the voters are independent and the question is meaningful, the answers are less likely to separate evenly between the possible options. This method can also be used if the question has more than 2 possible answers. In this case, there are more possible distributions of the votes, but the same logic can be applied.

Returning back to datasets, consensus annotation works very similar to the example above. Each image is annotated several times, typically by different persons, then the resulting annotations are compared between each other and merged, using majority voting or a different strategy. The key advantage of consensus annotation is that it helps to reduce personal annotator bias in annotation. This improves the quality of annotation by filtering out errors, noise (variance) and outliers in the annotation, leaving only the most representative ones.

Datasets, typically, have a large number of images and objects. This method of annotation requires several different annotations for the whole dataset, so it is expected to have several times of the annotation costs compared to the simple single-annotation approach. Depending on the annotation resources available, budget, and requirements, consensus annotation may or may not be feasible in a particular task.

One application for this approach that can be recommended is Ground Truth annotation. This type of annotation typically requires especially high quality annotations, because it is used to validate model or annotator answers. Ground Truth is typically limited only to a small portion of the whole dataset images, for example 3%. If a 3- or 5-fold consensus is applied to GT annotations, it is possible to obtain more reliable GT annotations for 10-15% of the full dataset annotation cost. Once there is such a reliable GT dataset, it can be used for annotator validation, on the remaining dataset ensuring the quality metrics are representative and objective.

Consensus replica jobs

A Consensus Replica job (replica) is the way to represent one of the annotator “opinions” in CVAT. Consensus replicas work similarly to regular annotation jobs - they can be assigned, annotated, imported and exported. When you decide to merge annotations from replicas, the results will be written to the parent annotation job.

Key properties of consensus replica jobs:

• Replicas are connected to annotation jobs. Each annotation job can have several related replicas.
• Only annotation jobs can have replicas. Ground Truth jobs cannot have replicas.
• Replicas use the same frame range as their parent annotation jobs.
• Annotations in replicas and parent jobs are independent from each other. Modifying a replica doesn’t affect the parent job or other replicas and vice versa. Removing annotations in a parent job doesn’t change annotations in its replicas.
• Replicas are not included in task annotation import or export. Per-job import and export still work for all job types, including replicas.

Read more about merging here.

Workflow

When annotating with consensus, the typical workflow looks this way:

1. Create a task with consensus enabled. Optionally, configure validation
2. Assign annotators to consensus replicas, wait until all the jobs are completed
3. Once all replicas in a parent job are completed, merge annotations
4. Review and resolve problems in the parent jobs

How to enable consensus in a task

Consensus annotation is configured at the Task level. It can only be specified at task creation. If you want to enable consensus for one of your existing tasks, you’ll need to recreate the task.

1. Go to the task creation page
2. Configure basic and advanced parameters according to your requirements, and attach a dataset to be annotated.
3. To enable consensus for the task, open the Advanced section and set Consensus Replicas to a value greater than 1.

1. Create the task and open the task page

If a task has consensus enabled, you’ll see the Consensus tag in the task summary. Existing Consensus replica jobs will be displayed in the job list under their parent annotation jobs.

Merging

For a given parent job with related annotated consensus jobs, merging will match annotations between the replicas and save merged annotations into the parent job.

There are 2 merging options available:

• merge replicas in all available parent jobs in a task
• merge replicas in a specific parent job

Merging is only available for a parent job if it is in the annotation stage and it has at least 1 replica not in the annotation - new stage and state. For simplicity, this can be read as “if there are any annotated replicas in the parent job”.

After merging, parent jobs are switched to the completed state automatically. If you prefer merging at the task level, it is recommended to switch merged parent jobs to the validation stage after they are merged to exclude them from the next merging and avoid losing the reviewed annotations.

How to merge all replicas in a task

1. Open the task Actions menu
2. Click Merge consensus jobs

1. Click Merge in the dialog window

The operation can take some time to be completed. Once it is completed, you will receive a status notification in the top right corner.

How to merge replicas in a specific parent job

1. Open the job Actions menu
2. Click Merge consensus job

1. Click Merge in the dialog window

The operation can take some time to be completed. Once it is completed, you will receive a status notification in the top right corner.

Review and resolve problems

After merging replicas, the parent job will contain merged annotations with consensus scores. These scores help you identify which annotations may need closer review.

Understanding consensus scores

Each merged annotation has two important properties:

• Score: A value from 0 to 1 that indicates the level of agreement between annotators. A score of 1.0 means all replicas agreed on this annotation, while lower scores indicate less agreement between annotators.
• Votes: An approximate number showing how many annotators agreed on this annotation. This is a UI-only value calculated as replicas × score. For example, with 5 replicas and a score of 0.8, the votes value would be approximately 4.

Annotations with lower scores may indicate:

• Ambiguous or difficult-to-annotate objects
• Inconsistent annotation guidelines
• Objects that need closer review

Filtering low-confidence annotations

Before starting manual review, you may want to filter out annotations with low scores to reduce the review workload. To do this:

1. Open the merged parent job in the annotation view
2. Use the filters panel to filter annotations by score
3. Review the filtered annotations and delete those that don’t meet your quality requirements
4. Save the annotations

Manual review process

Once filtering is complete (or if you skip filtering), proceed with manual review:

1. Open the merged parent job in the annotation view, enable review mode

2. Navigate through annotations using keyboard shortcuts:

   • Tab: Go to the next object
   • Shift+Tab: Go to the previous object

3. For each annotation, decide on one of the following actions:

   • Accept the annotation: If the annotation is correct, simply move to the next object
   • Adjust the annotation: If the annotation needs modifications:
     1. Press L to unlock the object
     2. Make necessary adjustments (resize, reposition, change attributes, etc.)
     3. Press L again to lock the object
     4. Move to the next object
   • Delete the annotation: If the annotation is completely wrong:
     1. Press L to unlock the object
     2. Press Delete or use the context menu to remove it
     3. Move to the next object

4. After reviewing all annotations, save your work using Ctrl+S (or Cmd+S on macOS)

Configuration

Merging settings

If you want to tweak some aspects of merging, you can do this on the Consensus Management page. It is available in the task Actions menu. Hover over the ? marks to understand what each field represents.

After you set values for the parameters, click the Save button. The updated settings will take effect on the next merging.

The following parameters are available:

5 - Immediate job feedback

Overview

The basic idea behind this feature is to provide annotators with quick feedback on their performance in a job. When an annotator finishes a job, a dialog is displayed showing the quality of their annotations. The annotator can either agree or disagree with the feedback. If they disagree, they have the option to re-annotate the job and request feedback again.

To ensure transparency with the annotator, the immediate feedback shows the computed score and the minimum required score. Information about the specific errors or frames that have errors is not available to annotators.

Feedback is only available a limited number of times for each assignment, to prevent Ground Truth revealing by annotators. This is controlled by a configurable parameter, so it can be adjusted to the requirements of each project.

How to configure

Immediate feedback settings, such as Target metric, Target metric threshold, Max validations per job and others, can be configured on the quality settings page.

This feature is considered enabled if the Max validations per job is above 0. You can change the parameters any time.

1. Open the task Actions menu > Quality control > Settings

2. Set the Target metric and Target metric threshold values to what is required in your project.
3. Set Max validations per job to above zero. 3 is a good starting number.
4. Save the updated settings

How to receive a feedback

1. Assign an annotator to an annotation job
2. Annotate the job
3. Mark the job finished using the corresponding button in the menu
4. Once the job is completed, you’ll see the job validation dialog

Each assignee gets no more than the specified number of validation attempts.

Available feedbacks

There are three types of feedbacks available for different cases:

• Accepted
• Rejected, with an option to fix mistakes
• Finally rejected when the number of attempts is exhausted

Additional details

6 - Analytics

CVAT provides analytics data for projects, tasks, and jobs to help you to track annotation progress and performance metrics at every level. Analytics support a wide range of use cases, including:

• Defining the working time a user spent on a job during a specific period.
• Tracking time spent in each job stage.
• Calculating the total number of ground truth objects in a project.
• Determining the number of ground truth images in a project.
• Checking interpolation rates to assess annotator efficiency.
• Identifying how many objects of a specific label were annotated in a resource.
• Calculating the average annotation speed of a user in a project or task.
• Analyzing how many objects or images were present in removed resources.

Analytics is a paid feature available in CVAT Online (paid tiers) and CVAT Enterprise.

In personal workspaces, analytics are available only to the workspace owner. In organizations, access depends on the user’s organizational role:

• Owners and Maintainers: Full access to all analytics.
• Supervisors: Access only to analytics for visible projects, tasks, and jobs.
• Workers: Access only to analytics for tasks and jobs assigned to them. Workers cannot update the analytics data.

Access

To open analytics:

• For a project
• For a task
• For a job

Analytics page

The Analytics page displays the data relevant to the specific resource (project, task, or job). Use the link in the page title to return to the corresponding project, task, or job.

Analytics data is not fetched automatically. When you first open the Analytics page, it will be empty. To fetch and display the analytical data, select the Request button.

Once the data is fetched and displayed on the page, you can check its relevance under the page title. A warning icon indicates that the resource was updated after the last analytics update.

To update the data, select button.

The Analytics page includes:

• Summary tab.
• Annotations tab.
• Events tab.
• Date filter.
• Export events button.

The Summary tab provides a statistics overview, while the Annotations and Events tabs contain the detailed data in table form.

To download a CSV file with all event data, select the Export events button.

Summary tab

The Summary tab displays the quantitative metrics:

• Objects diff: Difference between created and deleted objects in the selected time period. The value may be negative, if the number of the deleted objects exceeds the number of the created objects during the selected time period.
• Total working time: Total hours spent across all users, based on annotation-related events.
• Avg. annotation speed: Average number of objects annotated per hour. The value may be negative, if the number of the deleted objects exceeds the number of the created objects during the selected time period.

The Summary tab includes charts for object statistics, annotation speed, and diagrams for annotation distribution by labels and types. Hover over a chart or diagram to display tooltips.

Annotations tab

The Annotations tab shows annotation statistics for:

• Shape mode (the Detections tab).
• Track mode (the Tracking tab).

Both tabs always reflect the current state of the resource. Each tab includes a filterable, customizable table (learn how to work with tables).

You can search entries by the Label name column:

1. In the search box, enter the value or part of the value to find.
2. Select button or press Enter.

The Detection tab table contains the columns:

The Tracking tab table contains the columns:

Events tab

The Events tab displays the following metrics:

• Total objects: Total number of objects in the filtered jobs.
• Total images: Total number of images in the filtered jobs.
• Total working time: Total user time spent.
• Avg. annotation speed: : Average number of objects annotated per hour.

The Events tab table contains the aggregated events for the selected resource. Each event is defined by a unique status signature, which is a combination of the job’s assignee, stage, state, and the user who performed the action. As long as this status signature stays the same, all events are combined into one row. For example, if the same user creates two objects in the same job, the Events table will display one event that includes both actions.

However, if the job’s status signature changes (for example, due to an action performed by a different user) the analytics register a new event. As a result, actions that might otherwise be aggregated are instead recorded as separate events in the table.

You can filter the events by date range:

1. Select the date filter near the page title.
2. Select the first and last dates or enter them in YYYY-MM-DD format.

If the date filter is empty, the Events tab shows the metrics and events for the lifetime of the project, task, or job.

To reset the date range, select button in the date filter.

You can search the table entries by values in Task name, Assignee, Stage, State, User columns:

1. In the search box, enter the value or part of the value to find.
2. Select button or press Enter.

Other common operations with tables are described in the Working with tables paragraph.

The events table columns:

Working with tables

The tables in the Annotations and Events tabs support:

1. Exporting the data: select button.

   Note

   Visible columns do not affect the file with exported data. It always contains the complete table with all columns and rows.

2. Filtering entries by a custom rule: select Filter, and set filtering criteria. To learn more about how to set a filter, refer to the Filter article.

3. Clearing filters: select Clear filters.

4. Customizing columns:

   1. Select above the right side of the table.
   2. Toggle the checkboxes for the columns to display or hide them in the table.

5. Sorting entries: select the column name to apply sorting. The arrows near the column name indicate the applied sorting order. The arrow up indicates ascending order, the arrow down indicates descending order. You can sort the entries by one column only.

Large tables are split into pages. You can find the pagination controls under the table. You can also change the number of entries per page.